tl;dr
The narrative of verification and validation (V&V) is mired in process. The process is in the service of V&V as a means of assessment and credibility. This makes V&V as exciting as a trip to the DMV. The V&V community would be far better served to connect itself to science. Science by its very nature is inspirational serving as the engine of knowledge and discovery. As I noted before, V&V is simply the scientific method applied to computational science. This model serves far better than processes for engineering assessment.
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” ― Albert Einstein
The Symposium
A couple weeks ago I went to one of my favorite meetings of the year, the ASME VVUQ Symposium. It is an area of knowledge and personal expertise. I’ve committed and contributed a lot to it. The topics are important and interesting. The conference also seems to be slowly dying. The number of attendees and papers is dropping in number. This parallels the drop in interest in V&V as a whole. In published research, the V&V content has dropped off as practice is getting worse. Editorial standards are dropping. By the same token V&V content in research proposals is dropping off. Where it is expected, it is seen only as a duty and generally half-hearted.
The health of the conference can be measured by the length of it. The third day was canceled. Signs are bad.
In many places, it has been displaced by uncertainty quantification (UQ). UQ has the luxury of mostly producing results “in silico” totally artificially. You get results without having to go outside a code. Thus, objective reality can be avoided. The actual university because is a pain in the ass and a harsh mistress. UQ is full of complex mathematics and has inspired great work worldwide. It is an indispensable tool for V&V when used correctly. As computing has become the focus of funding, UQ has become the thing. Part of the reason is the huge appetite for computing UQ provides along with avoiding reality. The impact is the erosion of V&V.
There is a deeper perspective to consider. V&V has moved into practical use as part of applied engineering using simulation. V&V is a process to determine the quality of results. The key to that sentence is the word “process” and process sucks. Process is something to hate. Process is defined when practice is missing. Rather than an engine of progress and improvement, V&V has become an impediment to results. The movement to process is an implicit acknowledgment that practice doesn’t exist.
This is truly sad as V&V is equivalent to the scientific method. It absolutely should be standard practice. We should be working to change this.
“Societies in decline have no use for visionaries.” ― Anais Nin
Process is Death
“A process cannot be understood by stopping it. Understanding must move with the flow of the process, must join it and flow with it.” ― Frank Herbert
Process is something that is getting a lot of attention lately. The explosion of process is being recognized as an overhead and impediment to getting shit done. Process is a bureaucracy with its dullness and boredom. Rather than being interesting and exciting, it is dull and formal. The same thing is true in government. Process is in the way of everything. Things are checked and double-checked, then triple-checked. As Tim Trucano said, “V&V takes the fun out of computational simulation.” Science is fun and V&V is all engineering and pocket-protectors.
Process, regulation, and procedure are also a reaction to problems. When principles do not guide action regulations become necessary. The lack of principle is still there and resistance will occur. We see this all the time across society. We also see process and procedure taking the place of a professional practice. If the professional practice is in place, the process becomes natural. It can also adapt and bend to the circumstance instead of blindly following. There is also a suspension of judgment that right sizes effort. Sometimes the situation calls for more effort and others call for less. A professional practice guided by principles can do this. Regulation is not this.
Fortunately, the process overload has been recognized broadly. Recently Ezra Klein’s book, Abundance, has put this into the discourse. It provides examples and evidence of process overreach while suggesting a path to a better future. The example of California’s high-speed rail is compelling. There the regulations and process have led to little progress and huge costs. I’ve seen the same with nuclear power where process and regulation have removed progress. Everything costs much more and takes longer than it should. Rather than produce safe and sensible progress, process is a recipe for no progress. V&V feels like its the same thing to many practicing engineers and scientists.
It doesn’t need to be. It shouldn’t be.
V&V needs to be an engine of progress and excellence. The right way to orient V&V is as science. This should be easy because V&V is the scientific method for computational science. Here V&V can find its principles. Verification connects directly to modeling, and validation connects directly to experiments. Modeling that provides good results for experiments is the goal. V&V provides proof of the success (or failure). Going through the steps of V&V can be a practice where practitioners of modeling find confidence. It can be a means to produce evidence for doubters. I will not that the practitioner should be the first doubter and need to convince themselves.
Science is Inspiration
“We are what we pretend to be, so we must be careful about what we pretend to be.” ― Kurt Vonnegut,
Science is the place to look for a different path for V&V. The fact is that V&V should be essential for the conduct of science when computing is used. Not as a demand, but because it is simply how the scientific method works. Models are essential for science and drive prediction. These models are then tested and refined by experiments of observations. Models are mathematical and most often applied via numerical methods and codes. This points directly to the practice of verification. Validation is exactly the synthesis of the modeling with comparison to measurements. My proposition is that more V&V should be happening naturally. Science done properly should pave the way.
This is not happening.
If one looks to the scientific literature, V&V practice is receding. If we go back in time 30 years we saw a push for V&V in publishing. Editorial standards were introduced to enforce the push. Referees were haphazard and uneven with modest support by editors. The result was a temporary advance in the practice. As time has proceeded the advance has blunted and we’ve regressed to the former mean.
To some extent, this is an indictment of the literature. There is a gap in current practice and the proper scientific method. Unfortunately, progress starting 30 years ago was not sustained. Part of the issue is genuine animosity toward V&V from many quarters. I attribute much of this animosity to the dull process aspect of V&V. A worry about V&V as regulation contributed to this pushback. V&V as a process and requirement also challenged the role of editors and referees as the ultimate gatekeepers of science.
“In the long run, we shape our lives, and we shape ourselves. The process never ends until we die. And the choices we make are ultimately our own responsibility.” ― Eleanor Roosevelt
Science has always had an element of magic to it. The ability of models expressed in mathematics to describe the universe is incredible. It does feel almost magical when you first encounter it. Progress is a constant source of wonder. V&V is often a source of doubt. As such it challenges progress and is resisted by many. Instead, V&V should be a source of further focus and inspiration for science. It is an engine for better science and more solid progress. Science should also be the place for V&V to claim its place and legitimacy. V&V provides evidence of where science should focus on progress. Again, this challenges the gatekeepers.
If V&V continues to be a regulatory and bureaucratic process, it will die, It becomes part of our modern decline and descent into mediocrity. The path forward for V&V is to be an engine of knowledge and discovery. This focuses on action through principles and the adoption of practices that science depends upon. Good V&V is good science and could flourish as such.
“Whoever fights monsters should see to it that in the process he does not become a monster. And if you gaze long enough into an abyss, the abyss will gaze back into you.” ― Friedrich Nietzsche
The Regularized Singularity 