The narrative of verification and validation (V&V) is mired in process. The process is in the service of V&V as a means of assessment and credibility. This makes V&V as exciting as a trip to the DMV. The V&V community would be far better served to connect itself to science. Science by its very nature is inspirational serving as the engine of knowledge and discovery. As I noted before, V&V is simply the scientific method applied to computational science. This model serves far better than processes for engineering assessment.
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” ― Albert Einstein
The Symposium
A couple weeks ago I went to one of my favorite meetings of the year, the ASME VVUQ Symposium. It is an area of knowledge and personal expertise. I’ve committed and contributed a lot to it. The topics are important and interesting. The conference also seems to be slowly dying. The number of attendees and papers is dropping in number. This parallels the drop in interest in V&V as a whole. In published research, the V&V content has dropped off as practice is getting worse. Editorial standards are dropping. By the same token V&V content in research proposals is dropping off. Where it is expected, it is seen only as a duty and generally half-hearted.
The health of the conference can be measured by the length of it. The third day was canceled. Signs are bad.
In many places, it has been displaced by uncertainty quantification (UQ). UQ has the luxury of mostly producing results “in silico” totally artificially. You get results without having to go outside a code. Thus, objective reality can be avoided. The actual university because is a pain in the ass and a harsh mistress. UQ is full of complex mathematics and has inspired great work worldwide. It is an indispensable tool for V&V when used correctly. As computing has become the focus of funding, UQ has become the thing. Part of the reason is the huge appetite for computing UQ provides along with avoiding reality. The impact is the erosion of V&V.
There is a deeper perspective to consider. V&V has moved into practical use as part of applied engineering using simulation. V&V is a process to determine the quality of results. The key to that sentence is the word “process” and process sucks. Process is something to hate. Process is defined when practice is missing. Rather than an engine of progress and improvement, V&V has become an impediment to results. The movement to process is an implicit acknowledgment that practice doesn’t exist.
This is truly sad as V&V is equivalent to the scientific method. It absolutely should be standard practice. We should be working to change this.
“Societies in decline have no use for visionaries.” ― Anais Nin
Process is Death
“A process cannot be understood by stopping it. Understanding must move with the flow of the process, must join it and flow with it.” ― Frank Herbert
Process is something that is getting a lot of attention lately. The explosion of process is being recognized as an overhead and impediment to getting shit done. Process is a bureaucracy with its dullness and boredom. Rather than being interesting and exciting, it is dull and formal. The same thing is true in government. Process is in the way of everything. Things are checked and double-checked, then triple-checked. As Tim Trucano said, “V&V takes the fun out of computational simulation.” Science is fun and V&V is all engineering and pocket-protectors.
Process, regulation, and procedure are also a reaction to problems. When principles do not guide action regulations become necessary. The lack of principle is still there and resistance will occur. We see this all the time across society. We also see process and procedure taking the place of a professional practice. If the professional practice is in place, the process becomes natural. It can also adapt and bend to the circumstance instead of blindly following. There is also a suspension of judgment that right sizes effort. Sometimes the situation calls for more effort and others call for less. A professional practice guided by principles can do this. Regulation is not this.
Fortunately, the process overload has been recognized broadly. Recently Ezra Klein’s book, Abundance, has put this into the discourse. It provides examples and evidence of process overreach while suggesting a path to a better future. The example of California’s high-speed rail is compelling. There the regulations and process have led to little progress and huge costs. I’ve seen the same with nuclear power where process and regulation have removed progress. Everything costs much more and takes longer than it should. Rather than produce safe and sensible progress, process is a recipe for no progress. V&V feels like its the same thing to many practicing engineers and scientists.
It doesn’t need to be. It shouldn’t be.
V&V needs to be an engine of progress and excellence. The right way to orient V&V is as science. This should be easy because V&V is the scientific method for computational science. Here V&V can find its principles. Verification connects directly to modeling, and validation connects directly to experiments. Modeling that provides good results for experiments is the goal. V&V provides proof of the success (or failure). Going through the steps of V&V can be a practice where practitioners of modeling find confidence. It can be a means to produce evidence for doubters. I will not that the practitioner should be the first doubter and need to convince themselves.
Science is Inspiration
“We are what we pretend to be, so we must be careful about what we pretend to be.” ― Kurt Vonnegut,
Science is the place to look for a different path for V&V. The fact is that V&V should be essential for the conduct of science when computing is used. Not as a demand, but because it is simply how the scientific method works. Models are essential for science and drive prediction. These models are then tested and refined by experiments of observations. Models are mathematical and most often applied via numerical methods and codes. This points directly to the practice of verification. Validation is exactly the synthesis of the modeling with comparison to measurements. My proposition is that more V&V should be happening naturally. Science done properly should pave the way.
This is not happening.
If one looks to the scientific literature, V&V practice is receding. If we go back in time 30 years we saw a push for V&V in publishing. Editorial standards were introduced to enforce the push. Referees were haphazard and uneven with modest support by editors. The result was a temporary advance in the practice. As time has proceeded the advance has blunted and we’ve regressed to the former mean.
To some extent, this is an indictment of the literature. There is a gap in current practice and the proper scientific method. Unfortunately, progress starting 30 years ago was not sustained. Part of the issue is genuine animosity toward V&V from many quarters. I attribute much of this animosity to the dull process aspect of V&V. A worry about V&V as regulation contributed to this pushback. V&V as a process and requirement also challenged the role of editors and referees as the ultimate gatekeepers of science.
“In the long run, we shape our lives, and we shape ourselves. The process never ends until we die. And the choices we make are ultimately our own responsibility.” ― Eleanor Roosevelt
Science has always had an element of magic to it. The ability of models expressed in mathematics to describe the universe is incredible. It does feel almost magical when you first encounter it. Progress is a constant source of wonder. V&V is often a source of doubt. As such it challenges progress and is resisted by many. Instead, V&V should be a source of further focus and inspiration for science. It is an engine for better science and more solid progress. Science should also be the place for V&V to claim its place and legitimacy. V&V provides evidence of where science should focus on progress. Again, this challenges the gatekeepers.
If V&V continues to be a regulatory and bureaucratic process, it will die, It becomes part of our modern decline and descent into mediocrity. The path forward for V&V is to be an engine of knowledge and discovery. This focuses on action through principles and the adoption of practices that science depends upon. Good V&V is good science and could flourish as such.
“Whoever fights monsters should see to it that in the process he does not become a monster. And if you gaze long enough into an abyss, the abyss will gaze back into you.” ― Friedrich Nietzsche
My early adult life was marked by a vigorous pursuit of excellence. My time in Los Alamos provided the example, path, and environment for it. There I started to achieve it. At the same time, everything around me was decaying. Excellence is under siege in the USA. Eventually, the loss of excellence was too great and overtook the positive. The societal undertow has grown into a vortex of sprawling disappointment. Mediocrity lurks around every corner, and it’s swallowing excellence everywhere. Make no mistake, expanding mediocrity is the hallmark of our time. Excellence is in the past and receding. How deep into incompetence will we go?
“Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.” ― Joseph Heller
A Rant About Mediocrity
I’m going to start with a rant including a lot of cursing. So if you’re not down for that, stop reading now!. Frankly, if you can’t take some cursing, you’re not my people anyway. The situation we are in should make all of us very fucking angry.
I am really pissed off by this topic. I’m pissed off by the state of my country and the places I work. I’m following the mantra of writing what disturbs you, and this topic fills me with incandescent rage. In summary, we had greatness and excellence once upon a time. We have collectively managed to completely and totally fuck this up. While I will get to the reasons for the descent into incompetence, first and foremost I’m angry. Really deeply fucking angry at how we lost our edge. I’ll just note that I’ve spent a career developing expertise and accumulating knowledge professionally. What does this mean today? Fuck all the fuckwits they’ve put in charge.
I say this knowing many of the leaders at the Labs. Somehow we have a system that takes competent talented people and turns them into idiots. Great people are brought low instead of lifted up. Sometimes the result is pure incompetence or decisional paralysis. In other cases, they become unethical assholes, or the asshole becomes a monster. Societal forces seem to generate incoherence and destroy rationality. Our collective competence is far less than the sum of the parts. This is why I wrote the rest of this. Something about our system and society today is destroying everything good. It does seem to be the pathos of the current age that social stupidity is scheming to demolish any sense of excellence. Just look at our National leaders. Otherwise, talented smart, and successful people suck up ignorance and stupidity. They completely reject their own competence because the system can’t deal with it.
I see this directly at the labs every single day. You would think the importance of nuclear weapons might matter enough. It doesn’t.
What could possibly go wrong? Oh yeah, we aren’t testing these weapons and asserting they work via scientific prowess. Excellence at the Labs matters a lot, or it should. That fact is that it doesn’t really matter today. The approach is that Nuclear Weapons excellence can just be messaged. Except it can’t and I’m sure our adversaries in Beijing or Moscow can see through the bullshit. They know the truth. Our prowess has been in freefall for decades under the yoke of the same elements seen broadly in Washington today. These elements are the hegemonic power of money, lack of trust, and soul-crushing process. The entirety of politics and society bears responsibility. Politics on the left and the right have eroded excellence. One shouldn’t make the mistake of blaming Trump or Obama, Biden or Bush. The problem is all of us.
The path out of this is similarly society-wide. All of us need to find the way out.
“Mediocrity is contextual.” ― David Foster Wallace
The Pursuit of Excellence
When I step back and look at my personal history, I am so fucking lucky to be where I am. I had a solid middle-class upbringing and had a reasonable academic record prior to college. Frankly, I was skating by on my brains and putting little effort into academics. I did just enough so that my parents wouldn’t get wise to my habit of fucking off. I went to college and had an unremarkable record as an undergrad at a third-tier university (New Mexico). Granted, I got married early and worked full-time for most of that time, but my grades we just barely okay. I got my bachelor of science at a shit time to get a job in Nuclear Engineering. So I applied for jobs and didn’t even get a single interview.
“A life of mediocrity is a waste of a life.” ― Colleen Hoover
So, I defaulted into grad school at New Mexico. By the end of my first year, I managed to even disappoint myself. I saw my professional dreams dying due to my own self-imposed mediocrity. I made a pact with myself to get my shit together and start living up to my potential. I spent an entire summer relearning all my undergrad knowledge and skills. I entered the next year as a totally different student. From then on, I kicked ass as a student. I was the stud I could have always been. In short order, I realized my major professor was a complete asshole and I needed to escape from him. Note that I had a fully funded PhD project from NASA at that point that I was rejecting.
I broke from the professor in an epic meltdown. I thought of going to another school and found that between money and my grades; it was impossible. It was time to get a job. This was the best and luckiest decision of my professional life. I was looking for a job at the perfect time. I had everything needed to get a job: the right degree, an MS in Nuclear Engineering, USA citizenship, and a pulse. I had six interviews and six job offers. A couple of the jobs were horrible and non-starters (the interviews are entertaining and great stories though). Two were from local Beltway bandits (or Mesa bandits in New Mexico). They were okay. The last two were from National Labs including Los Alamos. The Los Alamos job was the best by a huge margin. After an urgent call to LANL, I got an offer and I took it.
Los Alamos was perfect; well close to perfect compared to elsewhere. For a student who had gotten their shit together, gaining huge ambition, it was a great environment. It was well beyond what a mediocre student from a third-rate university could hope to expect. I jumped in and immediately felt well out of my depth. I loved it and I was bathed in the excellence that defined Los Alamos. Better yet, the culture of Los Alamos was generous to a fault. I could tap into many people who were smarter than anyone I’d ever known. They would share their knowledge willingly and I grew. My work and colleagues were challenging and brilliant. I got better each and every day.
Los Alamos supported me in getting my PhD. The environment made me grow in ways I’d never anticipated. I finished my degree and continued to grow. Los Alamos was like the greatest grad school imaginable. Gradually, I started to feel that I was in my depth. I began to fit in. I began to meet my actual potential. Suddenly, the imposter syndrome that overcame me at Los Alamos disappeared. I was capable and I was an expert now. The excellence of Los Alamos had rubbed off on me. I had imbibed the culture of this magical place, and it transformed me. I had become a Los Alamos scientist and I belonged.
Little did I know that all of this was going to be destroyed by a tidal wave of idiocy and ignorance. The same idiocy and ignorance laying siege to all of us today. What I’ve come to realize the forces were already destroying Los Alamos and places like it for years before. The difference is that the storm was about to turn itself up to gale force. The terrifying fact is that the storm may be about to crank up to catastrophic hurricane force as you read this. Landfall is imminent if not already upon us.If we aren’t careful it will sweep everything good away. The danger is real. Mediocrity will be our legacy.
“The only sin is mediocrity.” ― Martha Graham
Money over Principles; Regulated to Death
“Ignore the critics… Only mediocrity is safe from ridicule. Dare to be different!” ― Dita Von Teese
How did we get to this point? We need to look back in history to the presidency of Ronald Reagan. The generally acknowledged wisdom at Los Alamos is that the Lab peaked in 1980. That was the year that Harold Agnew stepped down as Lab Director. Harold was a key person in the Manhattan Project and witness to major events in that age. Los Alamos went through forgettable leadership while government stewardship passed from the Atomic Energy Commission to the Department of Energy. This was a pure downgrade. The real corrosive influence was the attitudes of the government toward governance. Reagan represented a lack of trust and opposition to all things government. These forces unleashed by Reagan have grown and metastasized into a vile destructive force.
One of the major things coming from this period is a business principle. Milton Friedman’s approach to the business of maximizing shareholder value has become ever-present. It has become an engine of capitalism run amok. Businesses must always grow akin to a cancerous tumor. Sustainable business has gone out of fashion. The thing that matters to science and the Labs is the view that business principles became a one-size-fits-all all-cure to all things. By the mid-2000s this attitude would fully infect the Labs and reap destructive results for these paragons of science. We changed the social contract with the Labs from stewardship for the public good to corporate management. Somehow we thought a guiding principle to serve the Nation was bad. It needed to be replaced by a for-profit business. This change has only brought destruction.
The other force at work and in tandem is the regulation of every risk in sight. This regulation is dual in approach. On the one hand, it is an attempt to manage every single risk possible. It seeks to ensure that bsd things don’t happen. The other purpose is a general lack of trust for each other and institutions. Both of these desires are extremely expensive. Additionally, they are a bizarre way to provide accountability. Rather than leadership being accountable, the blame is projected onto everyone. Ultimately, the regulation ends up standing in the way of accomplishing things while driving up costs. It is inefficient and disempowering. It also speaks to a desire to control outcomes irrationally. The micromanagement of finances is driven by a lack of trust too. It amplifies all our leadership issues. Accomplishment becomes impossible.
The Nation only suffers and the benefits are illusions. The most corrosive influences of shareholder value are two-fold: money as a measure and short-term focus. The end of the Cold War brought the end of generous and necessary funding to the Labs. Congress now deemed it necessary to micromanage the Lab’s work and research. Over the preceding decades the micromanagement has grown to infect every detail of the Lab’s work Congress defines priorities rather than trust the experts. The overhead and intrusion have only powered a continuous lowering of standards and sapping of intellectual vigor. We now have little flexibility and massive oversight of all activities. The result has been continually lowering standards of work along with risk aversion. All of this is in service of controlling work and deflecting blame.
This influence has been modest in comparison to business-inspired management. The shareholder value-driven management philosophy is completely inappropriate for the Lab’s work. The real core of the problem is the lack of trust associated with how the Labs are managed. We have seen an explosion of oversight driven by suspicion and scandal avoidance. Technical work is graded by people who effectively have no independence. The management’s bonuses are dependent on good grades, and the reviewers know it. If you don’t give good grades you aren’t asked to review again. That paycheck is gone for the reviewer. In this way, duty fades away, and money corrupts the process. The Labs continue to be excellent, but it’s all smoke and mirrors. The truth on the ground is decline. Continuous, profound, and sclerotic decline over decades spreads like a cancer choking the Labs.
The problems with the shareholder value philosophy are becoming obvious at a societal level. In business, the approach can be applied to some significant benefit. With limits, this is where the approach has some virtue. It also has limits, such as supercharging inequality as an acute example. Its problems show up in producing sustainable businesses where growth isn’t an objective. This portends a conclusion that for managing science for National Security at the Labs, the idea is absolute lunacy. There is no profit to be had. The short-term focus that the stock market thrives on makes no sense. The result of the management is the destruction of any long-term health. Science at the Labs is withering under the yoke.
“The key to pursuing excellence is to embrace an organic, long-term learning process, and not to live in a shell of static, safe mediocrity. Usually, growth comes at the expense of previous comfort or safety.” ― Josh Waitzkin
Excellence is Hard; Mediocrity is Easy
“Caution is the path to mediocrity. Gliding, passionless mediocrity is all that most people think they can achieve.” ― Frank Herbert
There is little doubt that the Labs used to be great. The apex of this fulcrum is 1980. Los Alamos has faded; Livermore has faded; Sandia has faded too. The USA and the World are poorer for it. The excellence was supported and needed during the Cold War. The stakes of the work happening at the Lab demanded excellence and the Nation allowed it. Just as the USA grabbed victory in the Cold War, the support went. Part of the end of the Cold War was the hubris of “Star Wars”. The whole SDI idea was bullshit, but the excellence of the Labs sold the idea. The lie was huge and the Soviets believed it. The cost was destroying the trust of the Nation in the process. Relying on a bullshit idea like SDI played into the growing anti-government lack of trust.
Part of the issue is the use of financial incentives in management. For example, money can easily corrupt peer review. If it becomes clear that the reviewers are dependent on giving good grades to get a paycheck, the good grades come without the work. This is where the Labs are at. External reviews help determine the executive pay. The end result is that the reviews are always good. A bad review will lead to the reviewers not getting invited. It also projects onto the regulatory impulses where contracts falsely try to control management via regulated peer review. Leaders aren’t empowered and then held accountable simply. The cumulative result is a neutering of feedback. The reviews turn into admiration societies and increasingly have no value at all. For the Lab organizations, the alarm bell never rings, and quality simply degrades year after year.
The root of the issue then becomes the path of least resistance. Excellence is a hard thing to manage and requires focused attention. Mediocrity on the other hand is simple. Especially when all that really matters is the marketing of the work. It is easier to focus on perceptions of the work. Even this becomes simple as it is clear that standards are actually non-existent. You end up focusing on the work that is politically hot and leads to funding. You also focus on work that is flashy or presents well. More toxically, you simply know that the work is “world-class by definition” and the review has baked in results. All of this snowballs into a steady march toward mediocrity. There ends up being very little incentive for excellence to counteract these forces.
“It is difficult to get a man to understand something, when his salary depends on his not understanding it.” ― Upton Sinclair,
The Path Out
How do we get out of this?
While I can retire, I care about these topics deeply. The devolution of the Labs has been painful to watch and makes me seethe with anger. I am saddened for the Nation. These institutions are (or were) National treasures that have been squandered. The combination of mismanagement and lack of trust has wreaked havoc with the quality of the work. In many ways, the things that have hurt the Labs simply parallel the broader ills of society. The way out is similar to how the Country must heal from its downward spiral.
“Bullshit is truly the American soundtrack.” ― George Carlin
Appropriate and excellence-focused management is needed. We need to reorient the incentive structure and objectives for the Labs. The big issues are societal. There needs to be principles and values that transcend money. Excellence needs to have value for its own sake. There should be explicit empowerment to pursue excellence. Excellence needs to be recognized and bullshit needs to be called out. This will be painful. There is a lot of bullshit out there that managers think is great. We also need to take risks and allow failures. Without big risks and failures, excellence cannot grow. Risk and failure need to come from trying to achieve big things. This needs to be recognized for what it is and not punished or mislabeled as incompetence. This is a difficult thing to do. It is especially difficult in a time when bullshit is so regularly accepted.
The Labs need trust. We all need more trust. Trust is empowering. One of the key aspects of the environment that chokes excellence is an obsession with process. Most of this process is the result of mistrust. When there is mistrust there cannot be true excellence. The temptation or suspicion of bullshit is always present. Failure isn’t tolerated and punished. Rather than fail and learn, we fail and lie. Risks aren’t taken either because the downside is too extreme. We exist in an environment where every mistake is punished. The process is there to keep mistakes from happening. The result is no risk. Without risk, there is no progress or innovation. When it is all summed up, we can see that trust is a superpower.
At a deeper level, the difficulty of excellence can be seen to be one of lack of vulnerability. This is reflected in the humility needed for learning and the proper lessons from failures. Failures require trust and fuel the accumulation of expertise. Accepting all of these mishaps requires courage of vulnerability. We all of course see how today’s World chafes against this. Hubris, falsehoods as truths, and outright shameless bullshit are all expected. Vulnerability and failures are met with attacks, punishments, and reprisals. In these vile habits, excellence is snuffed out, and the tumble to mediocrity is catalyzed and becomes inevitable. When bullshit is as respected as truth, knowledge becomes negotiable. Then mediocrity cannot be separated from excellence. This is the state of things today.
“When an honest man speaks, he says only what he believes to be true; and for the liar, it is correspondingly indispensable that he considers his statements to be false. For the bullshitter, however, all these bets are off: he is neither on the side of the truth nor on the side of the false. His eye is not on the facts at all, as the eyes of the honest man and of the liar are, except insofar as they may be pertinent to his interest in getting away with what he says. He does not care whether the things he says describe reality correctly. He just picks them out, or makes them up, to suit his purpose.” ― Harry G. Frankfurt
My career is drawing to a close. In looking back it is obvious that V&V played a crucial role. This was never intended, but rather an outgrowth of other goals. The main driver was numerical methods research. V&V assisted my research and became a secondary focus. Along the way, I encountered remarkable resistance to V&V. This is because V&V challenges expert-based gatekeeping. It replaces their judgment with evidence and metrics. The response from V&V should be transformed into something supported. This is a connection to the classical scientific method applied to computations.
How did V&V intersect my Career?
“What’s measured improves” ― Peter Drucker
I am at a point personally where reflection on the past is quite natural. My professional time at revered institutions is drawing to its natural end. At the same time, my father is nearing death in a slow painful decline. My scientific career seems to be undergoing a parallel decline. It feels like it is crawling to the grave ushered by a lack of vision and strategy everywhere. Science and research options are under siege. Rather than being repaired, the decline is accelerating. Our science and engineering is in deep decline. Money is the ruling principle while quality is ignored. The result is an expanding mediocrity.
I have seen a host of significant events during my career that shaped and framed the World. The Cold War ended at the beginning marked by the Berlin Wall coming down in 1989. Working closely within the institutions that oversee nuclear weapons means that politics matter. World events are never far from shaping the work while providing emphasis to our responsibility. The technical work and its quality have always mattered. The stakes are huge. Events today may dwarf anything else from the span of my career. We shall see. I hope this is hyperbolic, but I fear not.
The quality of our work matters. It should matter more greatly if you are working on nuclear weapons. It is what I believe with all my heart. I’ve always embraced this as a primal responsibility. Verification and validation (V&V) is fundamentally about quality. This is why I got involved with it. The core aspect of V&V is measurement and evidence. It is a way of seeing the details of your work without appealing to expert judgment. It was a reaction to science that is ruled by expert gatekeepers.
Being an expert gatekeeper is a great gig. Usually that gatekeeper role is earned through accomplishments. Once the gatekeeper makes progress they often stand in the way of it. The gatekeepers then oppose anyone who disagrees with them. The gatekeepers are often journal editors and common reviewers. Too often they use this position to act as resistance to change and new ideas. These days the gatekeeper role is supercharged by how funding flows. In a day of science contracting, the money has even more power to strangle progress.
“If you thought that science was certain – well, that is just an error on your part.” ― Richard P. Feynman
How V&V Became a Thing for Me?
When I got started in science I wasn’t doing V&V. I was doing a little V&V, but didn’t know it. Like most of you I copied what I saw in the literature. I found ideas that I gravitated towards and then wrote papers like those scientists. Their papers were the roadmap for how I did my work. You adopt the accepted practices of others Eventually as you find success, you start to adapt. I was fortunate enough to get to work with some big names on a large research project. The tendency of youth is to listen with rapt attention to the experts. Over time, I grew tired of simply trusting experts; I wanted to see the receipts. I trusted and respected their work and judgment, but I also needed evidence.
Version 1.0.0
I started to see the cracks in their story. We were working with a couple of big names in computational physics and applied math. They were some of the scientists whose work I’d loved early on. Every couple of months they would travel to Los Alamos, or we’d travel to California for a project meeting. At these project meetings, we would be lectured on the “gospel” of the work. The issue was that the “gospel” changed a little bit each time. Eventually, I found that I needed to start doing everything myself. I needed to understand in detail of the “gospel”. I needed to see the evidence and verify what I heard.
“In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual.” ― Galileo Galilei
This process was my real transformation into a V&V person. I created an independent implementation of everything including testing. I would reproduce tests done by others and then create my own tests. During this time I documented everything and began to adopt my basic mantra of code testing. This mantra is “always know the limits of your code, and how to break it.” This meant I understood where the code worked well and where it fell apart. It tells you where you can safely use the code.
It also tells you where the code falls apart. This is where you should do work to make things better. This should set the research agenda. I have always seen V&V as a route to progress instead of simply measuring capability. V&V should provide evidence to support expanding capabilities. Today, the route to progress via V&V is weak to non-existent.
One of the lessons I learned was the separation between robustness and accuracy tests. Progress happens through transitioning robustness tests into accuracy tests. A robustness test is basically “Can the code survive this and give any answer?”. The accuracy test is “Can the code give an accurate answer?” This was useful then and continues to be a maxim today. We should always be pushing this boundary outward. This is a mechanism to raise capability and do better.
“We learn wisdom from failure much more than from success. We often discover what will do, by finding out what will not do; and probably he who never made a mistake never made a discovery.” ―Samuel Smiles
The Problem with V&V
“There’s nothing quite as frightening as someone who knows they are right.” ― Michael Faraday
In short order, I moved to the Weapons Physics Division in Los Alamos (the infamous X-Division). X-Division was ramping up development efforts to support Stockpile Stewardship. This was the ASCI program. The initial ASCI program was basically writing codes for brand-new supercomputers. The focus was on the computers first and foremost, but the codes were needed to connect to nuclear weapons. The progress was the desire to change from existing codes denoted as “legacy” to new codes. New codes were mostly needed because of the change in computers. This was not about writing better codes, but just using better computers.
The rub was that the legacy codes were trusted by the people who designed weapons. They were the simulation tools used to design weapons in the era when we tested these more fully. This trust was essential to the results of the codes. The new codes were not trusted. To replace the legacy codes this trust needed to be built. One of the mechanisms to build trust was defined as the processes known as V&V. The key part of the trust was validation. Validation is the comparison of simulations with experimental data. The problem with V&V is a certain emotionless approach to science. V&V is process-heavy and emotionless.
“The measure of intelligence is the ability to change.” ― Albert Einstein
Why is the process a problem?
The trust and utility of the legacy codes were mostly granted by experts. The people who designed weapons were the experts! “Designers”. They took an adversarial view of V&V and its process. This process is not expert-based, but rational and metric-based. What I have seen over and over in my career is tension between experts and process. V&V is rejected because of its non-expert rational approach. It was also rather dry and dull compared to the magic of modeling nature on computers. My original love of modeling on the computer was the embrace of its “magic.” It’s fair to say this same magic enchanted others.
I believe that the biggest problem for V&V is the dullness and process. V&V needs to capture more of the magic of modeling. The whole attraction of science is the ability of theory to explain reality. Computation is the way to solve complex models. This is part of the very essence of the scientific method.
“Any sufficiently advanced technology is indistinguishable from magic.” ― Arthur C. Clarke
Seeing V&V Clearly
The V&V program was added to the ASCI program in 1998. It tried to fill the gap of rational process in adopting the new codes. This rational process was supposed to build trust in these codes. Implicitly this put it into direct conflict with the power of experts. Nonetheless, V&V grew and adapted to the environment gaining adherents and mindshare. We can see V&V growing in other parts of the computational modeling world. In broad terms, V&V grew in importance through the period of 2000-2010. After this, it peaked and now has started to decay in interest and importance. IMHO the reason for this is how dull and process-oriented V&V tends to be.
“Magic’s just science that we don’t understand yet.” ― Arthur C. Clarke
A big part of this decay is the continued resistance by experts to the process aspects of V&V. I experienced it directly with my own work. I had the journal editor tell me to “get that shit out of the paper.” While the resistance to V&V at Los Alamos was driven by designer culture. Resistance to V&V was far less at Sandia, but still present. Engineers love processes, but physicists don’t. Still, V&V gets in the middle of processes engineering analysts like. For example, both designers and analysts like to calibrate results.
“The most serious mistakes are not being made as a result of wrong answers. The true dangerous thing is asking the wrong question.” ― Peter Drucker
They like to calibrate to data so that the simulations match experiments well. Worse yet they like to calibrate in ways that are not physically defensible. I’ve seen it over and over at Los Alamos (Livermore too) and Sandia. V&V stands in opposition to this. The common perspective is that V&V is accepted only so long as the results rubber stamp the designer-analyst views. If V&V is more critical, the V&V is attacked. The cumulative effect is for V&V to wane. We see V&V get hollowed out as a discipline.
“We may not yet know the right way to go, but we should at least stop going in the wrong direction.” ― Stefan Molyneux
Another program added to V&V’s waning influence. The exascale program spun up around 2015. In many respects, this program was a redux of the original ASCI with a pure focus on supercomputing. Moore’s law was dying and the USA doubled down on supercomputing research. This program was far more computer-focused than ASCI ever was. It also didn’t try to replace legacy code but rather focused on rewriting the legacy codes. This reduced resistance. It also reduced progress. At least the original ASCI program wrote new codes, which energized modernizing codes. The exascale program lacked this virtue almost entirely. Hand-in-hand with the lack of modernization was a lack of V&V. There was no V&V focus in the exascale. The exascale view was simply that legacy methods are great and just need faster computers. To say this was intellectually shallow is an understatement of extreme degree.
“Management is doing things right; leadership is doing the right things.” ― Peter Drucker
My own theory was that V&V needed to move past its focus on process. V&V needed to be seen differently. My observation was that V&V was really just the scientific method for computational modeling. Verification is a confirmation of solving the theory correctly. Validation is the comparison of theory with experiments (or observation). The real desire here is to connect V&V to the magic of modeling. I wanted to make V&V more smoothly part of the things I love about science and attracted me to this career in the first place.
What Can We Learn?
“Men of science have made abundant mistakes of every kind; their knowledge has improved only because of their gradual abandonment of ancient errors, poor approximations, and premature conclusions.” ―George Sarton
If I look back across my career a few things stick out. One is how the programs rhyme with each other. The original ASCI program was much like the Exascale program. We learned how to fund a focus on big hardware purchases, but not the science parts. In almost every respect the Exascale program was worse than ASCI. It was much less science and much more computers. The way this happened reflects greatly on the forces undermining science more broadly. Computers get interest from Congress, but science and ideas don’t. That interest creates the funding needed, and everything runs on money. Money has become the measure of value for everything today.
“People who don’t take risks generally make about two big mistakes a year. People who do take risks generally make about two big mistakes a year.” ― Peter F. Drucker
The biggest lesson is how irrational science is. Emotions matter a lot in how things play out. We would like to think science is rational, but it’s not. Experts are gatekeepers and they like their power. Rational thought and process are the expert’s enemy. V&V is unrelentingly rational and process-based. Thus the expert will fight V&V. Experts also tend to be supremely confident. This is the uphill climb for V&V and the basis of its decline. The other piece of this is money and its power. Money is not terribly rational, and very emotional. It is the opposite of principle and rationality. This combines to sap the support for V&V.
None of this changes the need for V&V. The thing needed more than anything is a devotion to progress. V&V is a tool for measuring progress and optimizing the targeting of progress. The narrative of V&V as the scientific method also connects better with emotion. In the long run a better narrative and devotion to progress will rule and V&V should play its role.
“The best way to predict your future is to create it” ― Peter Drucker
“The only way of discovering the limits of the possible is to venture a little way past them into the impossible.” ― Arthur C. Clarke
Rider, W. J. “Approximate projection methods for incompressible flow: implementation, variants and robustness.” LANL UNCLASSIFIED REPORT LA-UR-94-2000, LOS ALAMOS NATIONAL LABORATORY. (1995).
Puckett, Elbridge Gerry, Ann S. Almgren, John B. Bell, Daniel L. Marcus, and William J. Rider. “A high-order projection method for tracking fluid interfaces in variable density incompressible flows.” Journal of computational physics 130, no. 2 (1997): 269-282.
Drikakis, Dimitris, and William Rider. High-resolution methods for incompressible and low-speed flows. Springer Science & Business Media, 2005.
Rider, William J., and Douglas B. Kothe. “Reconstructing volume tracking.” Journal of computational physics 141, no. 2 (1998): 112-152.
Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281.
Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848.
The Regularized Singularity 