When a measure becomes a target, it ceases to be a good measure. Marilyn Strathern, formulating Goodhart’s Law

Let me be precise about that, because the imprecise version is wrong. Badly done V&V is no trouble at all for most managers. Well-done V&V is a nuisance for management. The reason is that well-done V&V finds problems and changes plans. Badly done V&V just lets plans and programs proceed unchanged.

The reason is simple. V&V finds problems and shortcomings in programmatic work. Those problems are anathema to a plan. They require extra effort to reach the level of success that managers have already described to someone above them. Management has over-promised, and V&V is the instrument that measures the shortfall. It puts a number on the distance between what was promised and what was delivered. These days, the funding is attached to the over-promise. Thus, V&V is a threat to funding. That threat needs to be neutralized.

The Checkbox

The result is that V&V becomes a desired checkbox for quality while the managers who want the checkbox decline to pay for the quality. That is a corrosive position to occupy. Interest in doing the work well steadily declines, and what survives is a stamp that certifies poorly done work as acceptable. V&V simply becomes a vehicle to justify the management’s BS.

The conclusion I have come to is that management needs approval for a great many things it has already done. It has committed to aggressive schedules across many projects. It has promised that modeling and simulation would return something large. On top of all that, it has spent a fuck-ton of money on big, flashy, expensive computers. All of it has to pay off, and it has to pay off visibly. Real V&V would show that the return is smaller than promised. Fake V&V supports their chosen narrative.

V&V done well questions every piece of that. What it would find is that the aggressive schedules, the enormous budgets, and the effort have gone into a variety of the wrong places. Managers would need to adjust and fix things. They would need to admit that the errors and predictions are not as good as expected. Fault would need to be admitted. These days, those are all career-limiting actions.

With four parameters I can fit an elephant, and with five I can make him wiggle his trunk. John von Neumann, as recounted by Freeman Dyson

This began to dawn on me when I realized that in the programs at Sandia, virtually no validation was actually being done. They would say there was validation, but this was an illusion. There was preliminary validation. Pre-shot simulations were compared to post-shot results. Then the model was modified and calibrated until it matched those results better. Usually the tests had no defensible error bars, so what we held was a set of point values from a single experiment, often not repeatable, set against a calculation.

That is not validation. That is curve fitting with a schedule attached. Since they had calibrated with the only test data available, validation becomes impossible. What we are left with is modeling in isolation, where the model form error cannot be determined. We spent an enormous sum on simulation technology, then systematically shot ourselves in the foot.

Burying the Evidence

Unless the individual truly responsible can be identified when something goes wrong, no one has really been responsible. Admiral Hyman G. Rickover

Then there is the example I have spent a great deal of ink on. Straightforward verification was performed on a code used for important problems, against analytical results that are highly relevant to those problems. The results were negative. They indicated real trouble with the code. Management’s response was to bury them, misclassifying a record to keep it away from prying eyes. This is corrupt. It is mismanagement. Worse, it guarantees that a problem in an important code will never be addressed, or even acknowledged. Instead, everyone is told to look away.

The same thing happens with validation results. Those are often properly classified at a high level and legitimately kept from public view, but the practice of burial is common and pernicious regardless. In substance it is the larger problem, because it leaves us with a very poor idea of how faithful our physical models are to the systems we use them to analyze and certify.

The human understanding when it has once adopted an opinion draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate. – Francis Bacon

The real victim of all this is quality, and science.

I have written repeatedly that properly practiced V&V is simply the scientific method applied to computation. When you meet this kind of obstinacy, this inability to deal with a problem once it has been named is the crime. What suffers is the quality of the results and any warranted confidence in them. The opportunity to mount a focused scientific effort to improve the models, the codes, and the methods is swept away with the inconvenient finding.

The purpose of V&V is to provide evidence that the work is good enough, and that the money bought something real. By massaging and burying results as they see fit, management short-circuits the mechanism entirely. They install a low-quality result as the standard rather than pursue excellence.

What these trends portend is institutional decay. Rather than engines of innovation and progress, the labs have become engines of the status quo: engines of “good enough.” Instead, they are engines of mistakes buried under (improperly) applied classification. In my view, the root cause is the preeminence of money as the measure of all things. Money has become the stand-in for quality and success. It is in the place of any real measure of technical merit or scientific progress. V&V is merely the place where the trend becomes most visible, and the place that suffers first. All the follow-on work that V&V should spearhead is abandoned with it, killed in the cradle.

Management simply wants to broadcast that everything it does is high quality and good enough right now, and that it is succeeding. There is an absolute inability on its part to mount any rational response to a genuine problem with the work.

Exascale: The Hero Calculations and No V&V

There is nothing so useless as doing efficiently that which should not be done at all. – Peter Drucker

A good share of the blame belongs to the Exascale computing program. One of its most pathological features was a general lack of support for V&V. There was no V&V element in the program at all. A small piece survived inside the NNSA portion, and that was a holdover from ASC. The reason is not mysterious. V&V would have delivered bad news about Exascale, and that should have been obvious from the outset.

Computing power was never the long pole for improving accuracy. The problem is, and was, the physical models themselves. Their limited accuracy, together with the intrinsic statistical variability of many phenomena that experiments generally fail to capture, is the actual source of the largest errors. V&V would have confirmed this and shown the level of investment in hardware to be foolhardy.

Anyone can build a fast CPU. The trick is to build a fast system. – Seymour Cray

Some studies could get you punished. I remember using the computers at Oak Ridge National Laboratory, where you would be penalized if anyone discovered you were running UQ studies on their big machines. UQ via sampling was trivial to scale up to the entire machine.

The reason was that the whole program had staked its claimed impact on the power of those machines to perform massive single-purpose hero calculations. That posture ignored how inefficient added resolution is at improving accuracy when the order of convergence is low, typically first order or considerably worse, and worse still once you reach phenomenology like turbulence, where all the statistical variability lives. No single hero calculation is science. It is a stunt. You can do them, but they have no error bars. The meta-belief was that the single calculation was “perfect”. This is like the DNS myth. The hero calculation is as good as an experiment.

A well-done V&V study would have made this plain and cast substantial doubt on the efficacy of the investment. Instead, politicians and managers were unified in their desire to promote success commensurate with the money, and to supply reasons why the money had been well spent. Flashy computing and unvalidated results were what they wanted to show. Under no circumstances should real science look under the hood and assess the quality of any of it. That could only produce bad news and demonstrate that the effort had been misdirected.

The Real Problem Is Balance

You can see the computer age everywhere but in the productivity statistics. – Robert Solow

The real problem is the lack of balance.

Investment in high-performance computing hardware, and in the software to use it, is money well spent. That is not the complaint. The problem with Exascale was that it was utterly unbalanced. Alongside the absence of V&V, there was no serious focus on methods or algorithms, and none on physical models or experiments.

Modeling and simulation is an integration across the whole of science. For it to flourish, every part of the scientific enterprise feeding it has to be healthy and pushing the state of the art forward. The program ignored the fact that methods and algorithms have delivered as much improvement in effective computing capability as the hardware has, and arguably more. Instead the focus on hardware became pathological.

We are now watching the same pathology repeat with artificial intelligence: enormous investment in data centers and computing hardware, with the same inattention to everything else. It is inflating an economic bubble that will very likely end in pain. That pain is almost entirely a product of the imbalance we saw in scientific computing, inherited wholesale by the work on AI. It will end up an economic own goal, completely unnecessary, and it will do immense damage.

The Canary

Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one. – Charles Mackay

V&V should be supplying detailed feedback and evidence to the scientific enterprise. AI too. It is the fulcrum of a balanced program that self-examines.

Instead it has become the canary in the coal mine.

Its neglect, and the generally poor quality of the practice that remains, is a harbinger of much larger problems: the absence of balance, and the inability to react flexibly to where the real bottlenecks in efficiency, efficacy, and quality actually sit in computational science.

It tells you something about how the AI effort has unfolded, too, and where the dangers in that activity are going to be found. We are probably too far in to avoid the collapse. One hopes we learn our lesson. Recent history seeds doubt.

“Science is a way of thinking much more than it is a body of knowledge.”– Carl Sagan

When someone shows you who they are, believe them the first time. — Maya Angelou

I’m going to start by doing something remarkably counter-cultural. I’m going to take responsibility for something.

In the run-up to my retirement, I trusted. I trusted my management even after I had accumulated a great deal of evidence that the trust was misplaced. I paid for that. It was my mistake, and I should have known better.

Trust is not the error. The world works better when people extend it. It is faster, cheaper, with less friction and less misery. The error is extending it to people who have already shown you, repeatedly, what they are. That isn’t generosity. That’s naivety, and mine was stunning.

Trust Broken

Trust and integrity are precious resources, easily squandered, hard to regain. They can thrive only on a foundation of respect for veracity. — Sissela Bok

To unravel my own naivety, go back six or seven years. Two things were happening then that bear on it.

The first was a large problem at the lab. It was expensive, it was embarrassing, and it was serious enough to draw a congressional investigation. The second thing I learned only later: that problem ultimately cost the lab director, Steven Younger, his job.

Around the same time, I fell into a vigorous discussion among senior staff about the research environment and its systemic failures. Forty or fifty of us worked up a white paper on how to fix it. A subset of us then agreed to meet with the lab director, and Vice President of Research to discuss the critique and suggest fixes. We met in the Lab director’s conference room, twelve senior technical staff and these two executives. We each spoke. We laid out the problems as we saw them. Then we waited for a response.

What we got was among the most unprofessional things I witnessed in my time at the labs. Dr. Younger began with a series of boasts about his own prowess as a researcher. He then turned on us for having the audacity to criticize the lab and its environment. He was belittling. He was angry. He was disrespectful to every person in that room. His arrogance was over the top and profound. Some of what the staff had raised was evidently more than he could take, and he was done pretending otherwise. No wonder he was sacked soon after.

The irony arrived later, when I understood that he was on his way out over the very problem we had come to talk about. His conduct in that room was not a departure from the culture that produced the failure. It was a demonstration of it.

Broken Institutional Trust

Wooden-headedness, the source of self-deception, is a factor that plays a remarkably large role in government. It consists in assessing a situation in terms of preconceived fixed notions while ignoring or rejecting any contrary signs. — Barbara Tuchman

The congressional investigation produced a report. The eagle-eyed reader will already have guessed where that report lives now: behind an official-use-only marking, out of view.

It contains a plethora of scapegoats and almost nothing identifying the actual source of the problem. The actual source was simple and it was obvious. The lab failed to put subject-matter experts into peer review early in the project. Those experts would have seen the problem and named it while it was still small. That was precisely why they were kept out, their review would have slowed things down and raised issues nobody wanted raised.

So the problem grew. It became far more expensive and far more embarrassing than early scrutiny would ever have been. The maxim of finding defects early was violated. The cost was massive.

The mechanism is the same one Dr. Younger displayed in that conference room: the reflex is to make the problem go away rather than to solve it. Hiding is cheaper than fixing. I had seen the identical reflex years earlier, in the same program, from Dr. Cilantro. He took exception to criticism of his staff not over the quality of their work, but over their systematic over-classification of it and the belligerence with which they met a review and any serious question. The same arrogance, in a different room, wearing a different face.

The other half of this unsavory sandwich is what happened to me more recently, with the report I flagged as problematic.

I am a subject-matter expert in the full scope of what that report covered. I was avoided. I was avoided in the earlier work, where the goal was to check the verification box and move on. When the more recent report identified real problems, there was no willingness to face them. There was only the old reflex again: bury it, and slap a classification on it to make the burial official.

Here is where my naivety deserves examination.

In every one of these episodes, I assumed the importance of the mission would win. I believed that national security — that nuclear weapons, of all things would finally force managers toward the right decision, whatever their inclinations. Surely the stakes were too high for anything else.

Sadly, that was foolish of me.

Reality is that which, when you stop believing in it, doesn’t go away. — Philip K. Dick

The stakes only work as a constraint if someone is still checking the answer against the world. As the distance from physical reality has grown, so has management’s confidence that reality is theirs to define. Nothing intercedes anymore. Nothing arrives to stop a stupid decision that no objective fact supports. You can simply declare the outcome and move on.

The Buck Stops Somewhere Else

The President — whoever he is — has to decide. He can’t pass the buck to anybody. No one else can do the deciding for him. That’s his job. — Harry S. Truman

Something more pernicious is happening in the wider society, and the labs are only a local instance of it. The examples set at the highest levels are of people — usually men — who take no responsibility for bad decisions or for the damage those decisions cause.

The debacle around the reflecting pool is as clear a case as you could ask for. Far too much money went to incompetent contractors who did poor work. Rather than own the decision to hire them, the president blamed other people. He invented people, and invented events, offered to explain away work that was simply bad.

Truman’s maxim has been revised. The buck stops somewhere else. I have witnessed firsthand, that revision reaches well beyond the Oval Office. It has permeated everything.

Trust Versus the Untrustworthy

Every bureaucracy seeks to increase the superiority of the professionally informed by keeping their knowledge and intentions secret. Bureaucratic administration always tends to be an administration of “secret sessions”: in so far as it can, it hides its knowledge and action from criticism. — Max Weber

You cannot work for people you cannot trust. Once trust is broken it is very hard to repair. So I left the people I could no longer trust.

More than that, I had stopped believing in the social contract between staff and management at Sandia. Some managers maintain a healthy, respectful relationship with their people, and those managers are a pleasure to work for. They may even be the majority. The problem is the substantial minority who operate on a different premise: that trust runs one direction only. They are in charge. Your job is to do what you are told. They do not worry about the truth; they decide what the truth will be.

Note the asymmetry. Staff are expected to trust that management is giving them the right direction. Management is expected to trust no one and explain nothing. Sandia too often operates without transparency even internally, and from the outside it operates without visibility at all.

The contrast with Los Alamos is instructive. Los Alamos is famous. It earned that fame with a starring role in the making of the atomic bomb, and Oppenheimer recently pushed it back into public view. Sandia has none of that notoriety, and revels in not having it. Invisibility is not an accident there. It is a preference.

I came to understand that this is exactly what the institution wants. Sandia does not want the spotlight, and its employees are encouraged to be invisible along with it. Management uses that darkness. It is what lets an organization cost the government an extra billion dollars, mislead the public about the cause, and drift quietly back into obscurity.

Measured against that, mislabeling a report as export-controlled is laughably small stakes. That is rather the point. Each episode that ends without consequence teaches management that the approach works. There is no penalty for unethical behavior. There is, in practice, a reward.

Coda

Show me the incentive and I will show you the outcome. — Charlie Munger

This is what an incentive structure tilted toward unethical behavior produces. It stops being a defect and becomes a feature. The absence of ethics turns into a vehicle for advancement, and competence becomes optional — worse than optional, because competence is expensive and difficult. Why pay for it? You can message success instead and skip the hard part entirely.

That is a recipe for institutional decline. No one is responsible for it.

Everyone is.

If you do not work on an important problem, it’s unlikely you’ll do important work.–Richard Hamming

An answer to an Asshole’s Question

I talked to my mentor, and he passed along a comment Cilantro had made about me:

“Why did we hire him in the first place?”

This was the prelude to them fucking me. Cilantro was my director, so I was fucked. They were going to come after me. I had no defense except surrender. Discipline. Shitty performance review. Cut off from anything good. It worked, and I was gone.

The timing was perfect. After Covid, and some modestly distressing personal health issues, my activation energy was incredibly low. My father was in hospice. I had watched him decline and suffer for over a year. Wasting my life working for incompetent, unethical assholes was an easy choice. I had checked my finances. I have two pensions too. I could retire and be comfortable.

It turns out Cilantro was asking a good question. Why was I there? Why had I come there in the first place?

The answer is twofold. My hire seemed a no-brainer back in 2007. The code I was working with, ALEGRA, was having problems. I had deep knowledge of hydrocodes from my years at Los Alamos. I knew how to write them, and I knew the details and foundations of hydrocode methods and applications. I was also a V&V expert. I had a great research record. I had been a line manager and a program manager. They were thrilled to have me.

All of those reasons were wishful thinking. It was a terrible fit. As difficult as it is to admit, Cilantro had a point. Why did they hire me? What use was I to that place?

I hired into the Computer Research Center, one of the most externally focused places at Sandia. It smells a little like Los Alamos. The difference is that this center could give two fucks about the Lab mission, and I was going into the most mission-focused department in it. My values were to care deeply about the Lab’s mission. Contributing to it was paramount. Thus, in the Computer Research Center that department was on the outside looking in. The rest of the Center was on the outside looking out.

If you gave a fuck about the mission, you worked in the Engineering Sciences Center. In fifteen years I would end up there. The problem is that Engineering Sciences is long on engineering and short on science. It bridges to the weapons program, where the knuckles drag. Engineers get answers and don’t ask questions. You aren’t there to think, just to do. The real problem is that the weapons program at Sandia is even worse. They are stuck in the past, engineering everything with decades-old approaches, unable to modernize.

The first draft of this essay was angry. Yes, I am angry about this. Even more, I am sad. The state of this great institution is pitiful and worth mourning. It reflects the same thing we see across society. Decline. Lack of ethics. Lack of competence. Abusive leadership. We deserve better and we are getting worse.

In a fully developed bureaucracy there is nobody left with whom one could argue, to whom one could present grievances, on whom the pressures of power could be exerted.– Hannah Arendt

Institutional Rot

Again and again, Sandia gave me reasons to question its choices about leadership. It seemed to choose people without the ethical core needed for leadership. Perhaps the leadership is somehow trained to abuse power. Whatever the reasons, we have the wrong people in positions of responsibility. It matters since the responsibility is for the USA’s nuclear weapons. Instead of seeking excellence in technical execution, they seek power and the easy way. This is not serving the nation.

It ends up being another example of a rotting institution. The motives and incentives of Sandia’s management are the opposite of what they should be. As a result, I don’t fit. My abilities and skills are not useful since they are guided by values. Those values of technical excellence serving national security are useless. Why find and fix problems when you can just wish them away? Why get the right answer for the right reasons, when you can just define success? This is the way Cilantro manages. Abuse power and squash anyone who gets in the way. Like me.

If this continues much longer, Sandia will cease to be fit for National Security. The Nation deserves better. I am sad to see this. I am sad to have spent any of my precious time on Earth serving this travesty. I am sad to see these people in positions of responsibility. They have authority and exercise it. Their exercise is for bad. Perhaps Sandia teaches and nudges them toward this behavior. This is a rot I want no part of. We see it everywhere in society. If Sandia is any guide, the USA is in real deep trouble. If a great institution acts like this, what other horrors are afoot? We are in danger. Our leaders are letting us down.

The greatest evil is not now done in those sordid “dens of crime” that Dickens loved to paint. It is not done even in concentration camps and labour camps. But it is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed, and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voice.– C. S. Lewis

Why I Quit

It is not that we have a short time to live, but that we waste a lot of it.– Seneca

Mortality was sinking in. I had to make some clear decisions about who I was, what I would do with the time left, and what values I would stand for. I had grown tired of cutting corners and making changes to fit in. Letting go of my most cherished values and ethics was one thing too far, one psychic blow I couldn’t take. I needed to be myself. I needed to stand up for the right thing.

What became very clear was that I stood in opposition to the values of this institution. Not its stated values, but its practiced values. In a sense, those values are most fully articulated in who the institution puts in positions of power, and as I’ve noted before, they put some genuinely awful people in those positions. Cilantro is an exemplar, and he is just like some of the other awful people they put in charge of things. When they hand these people authority, and those people exercise it through abuse of power, it speaks volumes about what Sandia actually is, as opposed to what it purports to be. You see people through what they do, not what they say. I was not prepared to sacrifice one more iota of my self-respect or my time supporting actions completely antithetical to my most cherished professional values.

The Report

For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.– Richard Feynman

The clearest evidence about the ruling that the report I’ve spent some time explaining is export-controlled can be found on OSTI itself. I received a letter demanding I take the report down after I downloaded it from the OSTI website. I got the document from a public website where things only get posted after a process. This needs some explanation. Only unclassified unlimited release documents go there. These documents have gone through a process where managers and other officials review them. This is not some decision the author makes. It is must be approved before it is issues a unclassified release number.

I don’t know what happened, but I can surmise. I had used the above process hundreds of times. It takes time, and other eyes see it. The report was in the bullseye of my professional knowledge and experience. I had worked on these test problems for decades with codes almost identical to the ones report. The results are predictable. Clearly, other members of management decided that it needed to be export-controlled instead after the fact. The reasons were political, not technical. There is NO valid technical reason for making it export-controlled. NONE! This is corruption and nakedly so. Fundamentally, the categorization of this report as “export controlled” is a lie. It is a lie focused on hiding bad results and, worse yet, shows no interest in solving the identified problems.

The real evidence of this is what you can find on the OSTI site. There are a number of documents related to CTH. Moreover, many of these are closer to the export-controlled line than the one in question. Just look at all the reports about high explosives. What is truly the smoking gun are two documents that are more extensive, but ostensibly the same thing. Verification reports including the test problems in the problematic report. The only difference is the nature of the results. These other reports were written trying to show the code is just fine, correct. They want to show that everything is okay. They do not identify any problems. This is for a good reason; they were written to deflect valid criticism.

There are two reports, from 2020 and 2021, on the verification and validation of the same code, CTH. Those reports were done with the objective of showing that the code is just fine. Checking the V&V box and moving on. This is verification done to blunt criticism, not partner with the science and engineering. That runs counter to the report I’ve discussed, which showed problems with the code. I encourage you to go to OSTI and download them. You only have to search under verification and CTH to find them. All of this just proves the corrupt intent of the managers, and the misuse of the export-controlled statute.

…a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it. – Richard Feynman

With regard to these earlier reports, the manager for CTH at that time knew damn well about my expertise in both shock physics and verification of shock physics codes. Knowing this, he never asked me to provide a peer review of those reports. I expect this is because he knew that I would find fault with them. They were not interested in doing excellent work. They were interested in deflecting criticisms.

What this also makes clear is that the decision is a pure abuse of power. They declared the report export-controlled because they can, not because any principle or technical fact required it. Acting without principle is simply a confirmation of the power they hold. Exercising that power is, first and foremost, how they manage. It is exemplified by the behavior of Dr. Cilantro, their boss. This is a key part of institutional rot across society. Sandia is just another example.

This provides absolute clarity about why the report is export-controlled. Management does not like the results. They are using export control to hide results they don’t like. It is unethical and corrupt, and it is emblematic of the way science is managed in the United States today. Corruption and the lack of ethics are everywhere.

It appears that there are enormous differences of opinion as to the probability of a failure with loss of vehicle and of human life. The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from the working engineers, and the very low figures from management.–Richard Feynman

…it would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy. – Richard Feynman

Verification as a Checkbox

The first principle is that you must not fool yourself — and you are the easiest person to fool.– Richard Feynman

There is one key difference with the verification reports that are available on OSTI. Those were done in order to provide a checkbox that verification had been performed. The main message is: we want everyone to shut the fuck up about verification, this code is just fine. The work was done by checking the box and declaring the code okay so that everyone could move on to more important things. Now we can move on to solving applications.

This is verification in its most egregious form, treated as a tedious and useless activity demanded by people who care about correctness and science. Those people are pests, and we do this work to shut them up. The last thing Sandia is really concerned about is high-quality technical work, which is why I was never asked to peer review these other reports. I found them by chance after they were published. Even if I had reviewed them, anything I said would have been ignored, because the purpose of the documents was to close the book on verification rather than harness it to make anything better.

This episode gets to the core of the question. You have reports that fall right into the bullseye of my professional and scientific expertise, and Sandia shows no interest in that expertise or in using it. They see verification as one of the checkboxes engineering requires, and they look to put it to bed as simply as possible so they can get on to the real work of engineering things and grinding out results. Getting things right is something that can be handled by calibration. Getting the fundamental hydrocode methods and solutions correct is something they have no real interest in. If it happens, it happens, but correctness is unnecessary, surplus to requirements. Engaging my expertise is a complication they can do without.

Nothing says “I am in the wrong place” more clearly than this

You could leave life right now. Let that determine what you do and say and think. – Marcus Aurelius

Coda

Sunlight is said to be the best of disinfectants; electric light the most efficient policeman. – Louis Brandeis

This was no place for somebody like me to work. I was wasting my time, my effort, and my life there, and I would encourage anyone with a desire to do creative, original work to stay away. It’s not for them. If you want a good paycheck and decent benefits, and you are prepared to follow orders without regard to the intelligence of those orders or the ethics involved, it’s the place for you. As long as you’re willing to follow the directives of managers who show some degree of incompetence and a general lack of ethics, you will do well there.

The truth is that I never belonged. It was broken before I arrived. Sandia is a place for small ideas, and it is about executing those small ideas with persistence and precision. It is not about accuracy or correctness. Its excellence is execution without regard for purpose or value. I am a scientist, not an engineer, and in the Sandia system the engineer executes whether the idea is a good one or not.

The real tragedy is the time that I wasted, when my skills and talents could have been used for something better. The other tragedy is the irresponsibility of those in positions of power at Sandia. They are overseeing aspects of the nation’s nuclear weapons and doing so without competence. That lack of competence is partially technical and partially ethical. In either case, the nation is being mis-served.

I really should stop writing about Sandia. I spent twenty years there, and I’ve come to realize it was a professional mistake. I never fit in, and while I was there I only grew to fit in less. Perhaps my frustrations boiled over and I self-immolated. I took a change of jobs into a place where I was even less suited. It was always a poor fit, and moving to Engineering Sciences amplified it. When Cilantro arrived as director, it was a reversion to form and my days were numbered. Fortunately, I had been getting my retirement shit together. I wasn’t going to put up with their bullshit, so I pulled the ripcord.

Drive out fear, so that everyone may work effectively for the company.– W. Edwards Deming

References

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, W. Rider, T. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LLNL-TR-411291. Lawrence Livermore National Laboratory (LLNL), Livermore, CA, 2008.

Duncan-Reynolds, Gabrielle Christiane, and Christopher T. Key. Improvements to the New CTH Code Verification & Validation Test Suite (FY2020). No. SAND–2020-13925. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2020.

Duncan-Reynolds, Gabrielle C., and Christopher T. Key. FY2021 Improvements to the New CTH Code Verification & Validation Test Suite. No. SAND2022-7834. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2022.

The early angry illustration.

“The most important figures that one needs for management are unknown or unknowable.” — W. Edwards Deming, Out of the Crisis (1986)

tl;dr

The miraculous nature of today’s world owes its existence to human progress. Some of that progress is serendipity. Some is targeted and guided. In science, measurement is the key both to inspiring progress and to recognizing it. For algorithms, methods, and computing, code verification is how we measure. The alternative to measurement is expert judgment. Both have value. But when you find a field that has stopped progressing, you will usually find too little measurement and too much expert judgment. If we want computational progress, measurement is our friend.

Progress Is Paramount

When I started graduate school, I worked on nuclear energy — specifically the more exotic aspects of it, in support of space exploration. As exciting as that might sound, I never developed any passion for it. It was a reaction to my professor, or perhaps just the topic.

When I started learning numerical methods for differential equations, though, I was full of excitement. The idea that you could simulate the real world on a computer fascinated me. I enjoyed the power and beauty of hyperbolic PDEs and their solution, especially the nonlinear methods that had recently come into vogue. I ate it up.

I learned numerical method after numerical method, gradually narrowing my focus to monotone shock-capturing schemes. FCT and TVD methods were in vogue then, and ENO methods were just being created. I was enchanted by the rapid progress of the previous decade and read paper after paper as the methods were developed and tested. I also grew fond of the associated mathematical literature, finding real beauty and elegance in how that mathematics supported both the methods and our understanding of them. As I devoured the literature, I began to study something else: how these methods were tested and demonstrated, and what the standard was for claiming one method was better than another.

What stands out in retrospect is that the prevailing view of “better” rested on a qualitative metric. The measure was the expert’s opinion about the vagaries and nuances of a visual plot. The judgment seemed defensible enough in one dimension. In two and three dimensions, acceptance of results was essentially an artistic verdict.

A few years later, after I had moved to Los Alamos, I noticed the working version of this standard: the gnarlier and swirlier the vorticity in a problem, the better the method was considered. And yet the progress made in these methods over a short period was astounding. As readers who know my general tendencies will guess, progress is something I care about deeply. Progress is what we should always be seeking.

“Working on the right things is what makes knowledge work effective. This is not capable of being measured by any of the yardsticks for manual work.” — Peter F. Drucker, The Effective Executive (1967)

Measurement Is Science

As this body of work became more familiar to me, the reliance on qualitative examination and expert opinion became more and more distasteful. I met a number of people who were genuine experts and who sat in judgment over methods and progress. Eventually I saw them for what they had become: gatekeepers. There was no objective measure of what was better. There was only expert opinion, and expert opinion favored those who already held power.

The problem with this status quo is that progress has screeched to a halt. Journal editors are thoroughly fed up with yet another paper developing and comparing limiters — and I am responsible for a couple of those myself. It became a cottage industry. Meanwhile, the standard of measurement for the problems that actually matter is nonexistent. This is the recipe for a stalled science. The key thing to recognize is the centrality of measurement to science. Unmeasured qualitative assessment is not science. The stagnation of progress is therefore no mystery — the two are correlated. Progress requires measurement.

This gets at the real tension over results and standards. The mathematics supporting this area is quite limited. Rigor comes with caveats and serious restrictions. Relatively few practical results carry any assurance from the theoretical foundations of the field. Into that vacuum steps qualitative gatekeeping, and the consequence is stagnation.

To overcome this, we need to measure results quantitatively. We also need a leap of faith: we must be willing to trust what we see empirically, in the places where theory cannot follow. We know qualitatively that modern methods were a quantum leap in capability. They let computations attempt problems that had been impossible, and they rapidly became standard. The problem is the threadbare theory. Consider the Lax equivalence theorem, which establishes that consistency plus stability is equivalent to convergence — but only for linear PDEs. All our experience says the theorem describes the utility of computing for nonlinear and far more complex problems. What we lack is the rigor to prove it. Stagnation has festered in exactly that gap.

“Absence of evidence is not evidence of absence.” — commonly associated with Carl Sagan

Methods, Algorithms, and Codes Need Metrics

As I matured as a scientist, I came to understand that measurement is feedback. Measurement is science, and method development needed it to flourish. This view generated a great deal of friction with the existing community — friction I found inside the Lab’s computational physics community as well as outside it. It came to a head in an exchange with an editor who demanded that I “just take that V&V shit out of the paper.” Gatekeeping at its finest.

That paper was the culmination of work at Los Alamos; it died on the vine at Sandia. In it I was looking for methods that could deliver quantitatively better solutions to discontinuous problems at an attractive cost — efficiency, in a word. Doing that requires unraveling which elements of method design actually contribute to better solutions, balanced against computational cost. The work grew out of an observation: WENO methods buy formal accuracy at high cost and fail to deliver practical accuracy. Jeff Greenough and I published that failure.

The next step was to turn that knowledge into a better method, which I believe we accomplished in the paper with Jeff and Jim Kamm. Code verification was central to demonstrating the progress, and that is precisely where we ran headlong into the gatekeeping. Results for problems with discontinuous initial conditions and shocks are evaluated qualitatively, full stop. Measurement and quantitative measures need not apply. We measure only for smooth problems, and only to confirm formal high-order convergence rates. That is the standard practice.

No wonder we are stagnating. No wonder there are thousands of papers tweaking limiters and WENO variants. Without measurement, progress is a random walk. I am still pushing this effort forward because I remain convinced it is necessary. Progress depends on aligning our approach with best practice; verification is part of that practice, and V&V must be quantitative. I got into V&V because it is how you do science correctly. The irony is hard to miss: engineers have embraced V&V while scientists and mathematicians treat it with genuine animosity. The problem is that engineers see V&V as a process and a check, not a fuel for progress.

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it… your knowledge is of a meagre and unsatisfactory kind.” — Lord Kelvin, Popular Lectures and Addresses (1883)

Code Verification Is the Way

“Measure what is measurable, and make measurable what is not so.” — attributed to Galileo

Once I understood modern methods, I started to see the cracks in practice, and working alongside some of the leaders in the field amplified the view rather than softening it. In everything I did, I looked to measure the connection between theory and observed results. Not only comparisons against analytical solutions, but all results.

Analytical comparisons carry the great power of objective truth. Without them, you can estimate error, but you cannot know it. Given how soft the theoretical results are, comparison against analytic solutions deserves a spotlight. The catch is that the problems with analytic solutions are the ones furthest from application.

I had discovered that V&V was essential. V&V provides feedback on your code and your model, and that feedback is actionable. Qualitative comparison is actionable too, but the variations it suggests come from expert judgment — and experts have bias. That bias usually takes the form of projecting one’s own philosophy of algorithm design onto the results. I preferred something less prone to it. Code verification became my instrument for measuring progress, and the arc of my work through the late 1990s and 2000s reflects that. Along the way, I became more knowledgeable about code verification and more engaged with it broadly.

What is worth acknowledging is that my position runs counter to both communities. The algorithms and methods people carry animosity toward quantitative verification. The verification community that came out of engineering has its own bias: it adopts relatively simple practices and tends to ignore most of the mathematical expectations. Its members are dismissive of the equivalence theorem while adhering to its precepts far beyond the range where it rigorously applies. So I stood out as an outsider. This is a role I have grown comfortable with, and even proud of. The quantitative comparison of results gave me constant feedback and guided my work.

“Nullius in verba” (“Take nobody’s word for it”) — motto of the Royal Society, 1660

Why Is Code Verification Despised?

The logic of quantifying error and using it as feedback is clear enough. So why is the practice resisted so hard? Why the animosity?

The answer is the power bound up in gatekeeping. Experts who hold power in their own right have that power amplified by serving as the judges of progress. They ensure their own work remains the cornerstone. Their judgments involve a great deal of handwaving: the reading of solution structure and oscillations is judgment, and judgment can be made to serve its holder’s ends. Quantitative verification is a different animal, because numbers cannot be waved away — they still require expert interpretation, but they constrain it.

The animosity, then, is largely gatekeepers defending their hold on a field. They decide who publishes and who gets credit, and they reject those they dislike or who fail to reinforce their standing. This behavior is hardly confined to computational fluid dynamics. The turbulence community is equally stagnant, if not more so, and for the same reason: persistent gatekeeping throttling progress.

I saw gatekeeping at Los Alamos as well, where the nuclear weapons designers held the role. They were quantitative about validation results, but they achieved those results through extensive calibration. In particular, they would quite cavalierly calibrate away numerical error using physics submodels. This is a common practice as weather and climate modeling does the same basic thing.

The hostility toward quantitative measurement of code performance is deep and profound, and together with the CFD gatekeepers, it has blunted progress for decades. The result is a hegemony of older methods still in use long after they should have been replaced. It limits what modern computing can contribute to genuinely important problems. This remains a force that saps some of the value we should be enjoying from computing. Over the history of the computational age, methods and algorithms have provided efficiency equal to or greater than computing itself.

The case for being quantitative about practical problems is, in some ways, simple. Practical problems have extremely low convergence rates — first order is typically the best you can achieve, because the dynamics of numerical methods change considerably in the presence of discontinuities. Under those conditions you should care far more about producing small errors than about high convergence rates. Highly accurate, high-convergence-rate methods simply don’t pay off, particularly when they are expensive.

The real question is how to deliberately produce methods that achieve small error, remain robust, and selectively preserve the solution structures that matter. That is a fundamentally different design problem than chasing formal order of accuracy. Pursuing high-order methods formally can produce more error on practical problems. They only show their benefit when the problem becomes complex enough to contain a great deal of intricate structure. This points toward greater adaptivity to tie the two uses together cleverly. Quantitative assessment is key to success.

“It is wrong to suppose that if you can’t measure it, you can’t manage it — a costly myth.” — W. Edwards Deming, The New Economics (1993)

References

Sod, Gary A. “A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws.” Journal of computational physics 27, no. 1 (1978): 1-31.

Boris, Jay P., and David L. Book. “Flux-corrected transport. I. SHASTA, a fluid transport algorithm that works.” Journal of computational physics 11, no. 1 (1973): 38-69.

Harten, Ami. “High resolution schemes for hyperbolic conservation laws.” Journal of computational physics 49, no. 3 (1983): 357-393.

Jiang, Guang-Shan, and Chi-Wang Shu. “Efficient implementation of weighted ENO schemes.” Journal of computational physics 126, no. 1 (1996): 202-228.

Rider, William J., and Douglas B. Kothe. “Reconstructing volume tracking.” Journal of computational physics 141, no. 2 (1998): 112-152.

Rider, William J. “Revisiting wall heating.” Journal of Computational Physics 162, no. 2 (2000): 395-410.

Rider, William J., and Len G. Margolin. “Simple modifications of monotonicity-preserving limiter.” Journal of Computational Physics 174, no. 1 (2001): 473-488.

Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281.

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848.

Banks, Jeffrey W., T. Aslam, and William J. Rider. “On sub-linear convergence for linearly degenerate waves in capturing schemes.” Journal of Computational Physics 227, no. 14 (2008): 6985-7002.

tl;dr

I worked with classified documents almost my whole career. This isn’t odd when you work at two nuclear weapons labs. Classification is important and essential for security. The law and practice of it is a fucking mess, none more so than export control. There are huge loopholes, and the powerful flout the rules with impunity. Corrupt managers abuse the law for their own ends. Those in power ignore the law regularly. Today it is simply another way power and inequality asserts itself.

For when everything is classified, then nothing is classified, and the system becomes one to be disregarded by the cynical or the careless, and to be manipulated by those intent on self-protection or self-promotion.”– Justice Potter Stewart

The fundamental issue

A couple of years ago we discovered that the former (and current) President had hundreds of classified documents in his residence. They were stored in ballrooms and bathrooms. He shared the information with various people, foreign and American, as he wished. Among these documents were some of the country’s most essential secrets, far beyond anything I ever saw or had access to. Had I done what he did, I would expect to be imprisoned for the rest of my life.

The President suffered no consequences for this extreme violation of trust. It was also an abdication of responsibility. I took it as a personal insult. It is also a pattern I had already seen over and over, and it inserted itself into my life again this week. Rather than protecting important information, classification is simply another way for the powerful to fuck with the rest of us. These violations are common and committed with utter impunity by those in power. The danger from this is real.

Identifying and managing classified information is very important. The protections and the secrets are essential. This information falls into broad categories. Secret Restricted Data (SRD) is the material associated with nuclear weapons. I dealt with this a lot; I even held a position of responsibility for identifying it. National Security Information (NSI) covers other timely information about things like spying and defense. These are the categories the President stole and misused. Finally, there is Controlled Unclassified Information (CUI), formerly Official Use Only (OUO). Export-controlled information falls under this umbrella. CUI is the most commonly and broadly abused category, and coverups and information hiding are rampant under it. I’ll get into this below.

“Secrecy is for losers. For people who do not know how important the information really is.” —Daniel Patrick Moynihan

This function of government is absolutely essential, and it must be wielded responsibly. Instead, the powerful often use it cynically and selectively for their own ends. The result is abuse of power. It is another way that the law applies only to the little guy. Another manifestation of the Epstein Class, who follow completely different rules than the rest of us. The result is a set of essential laws that simply become another way to tell the average citizen to fuck off.

First, let’s get to a few basics of the law and practice. Then I will elaborate on some of the abuse of power I witnessed over my time at the Labs. Some of this has parallels in the news (Trump, Clinton, Biden, John Deutch, …) worth mentioning. If you’re powerful, the law is just notional.

Classified material

When most of us think about classification, the thing that comes to mind is SRD, the category used to classify material related to nuclear weapons. It has been a major focus in my life.

When I worked at Los Alamos, I became an authorized derivative classifier. This job involved reviewing documents, presentations, and emails for the presence of classified information. There are many subtleties in all this, and I came to understand the area quite well, particularly in my specific expertise, which spanned various parts of computational science.

The real focus of all this classified work is always on the applications and how the work is reflected in technology. This is generally true for this type of classification, and it shows up again in export control. It is central to understanding the whole topic.

This form of classification is defined by statute. When you’re doing classification work, detailed technical guides define what is classified and provide a rubric to work with. Violations of this classification law carry penalties that can be quite severe. I had many interactions with violations and security issues, but generally speaking, my personal record was quite good.

When I transitioned to Sandia, it would have made sense for me to continue as a derivative classifier. My experience with the professionalism and competence of the Sandia classification office quickly convinced me this was a risk exposure I couldn’t tolerate. The classification office at Los Alamos was my partner and peer; the classification office at Sandia was adversarial and struck me as outright incompetent. The institution took a negative, adversarial approach toward employees and made me feel like I was the one at significant risk all the time. They also provided far less support for anything related to classified documents and pulled that support away from any sort of local control. In short, the overall attitude made me extremely uncomfortable having anything to do with classified work at that laboratory, so I declined the responsibility.

“The concept of the ‘official secret’ is the specific invention of bureaucracy.” — Max Weber

Export control law

The second place classification shows up is export control policy. Export control deals with information that is less sensitive and less damaging than nuclear weapons material. The law was written after the statutes governing nuclear weapons information, and ironically the statutory penalties are far worse. Compounding things, the law is much vaguer and written with a fear-based approach, with little to no technical guidance for determining whether something is export-controlled.

As I will elaborate, this makes the policy more arbitrary to adjudicate. It also makes it more prone to abuse and outright violations of the spirit and letter of the law. That led to my retirement, and to the episode this last week.

All of this matters a lot. When I look at the issue around my retirement, its sensitivity designation is central. I can say unequivocally that the document this all revolves around was not export-controlled. Yet, it was declared as such. It was not information sensitive in any way and should not have had any of these controls attached to it. The declaration that it was export-controlled was a pure abuse of power. Knowing the objections of the managers makes it corrupt on its face. It was declared export-controlled only because the contents of that report were bad for the laboratory and exposed problems. The designation was done purely to protect the outright incompetence and unwillingness of the managers to address the problems the report identified.

The applications of this work have nothing to do with the report or the material contained in it. These are solutions to hydrodynamic test problems with analytic solutions, used for code verification. The code verification results are clear and unambiguous: the code cannot solve these problems correctly. The reasons are simple and easy to explain. Unfortunately, they are not articulated in the report, and the institutional approach is to simply hide the report and assert that the results of these test problems are immaterial to the problems the code solves. Nothing could be further from the truth.

“It quickly becomes apparent to any person who has considerable experience with classified material that there is massive overclassification.” — Erwin Griswold

Abuse and violation are common

I wish I could say that abuse of power and violations of these laws were uncommon. The inverse is true. I mentioned the President’s hoarding of classified documents as a prime example. By the same token, Hillary Clinton used a private email system for classified work, which was a violation. John Deutch, the former CIA director, took classified material home. Former President Biden had a handful at home by mistake.

I saw two instances at Los Alamos where lab directors committed fairly serious security violations and were essentially let off the hook entirely. They suffered no consequences.

In one case, a lab director spelled out details of a security investigation in a public meeting. The classification level of a security investigation is the same as the material being investigated. He was simply let off the hook as they declared the material had suddenly become declassified.

Another lab director read aloud a passage written by the head of the NNSA that contained classified material. This was a subtle violation, but one common in the area where I worked. I immediately knew that what was read was classified. When I took it to the classification office, based on their previous experience, they felt it wasn’t worth touching. Again, because of the power of the individuals involved, they were off the hook. I don’t think they even had any idea they had committed a violation.

Given this experience, it comes as no surprise that managers believe they can get away with things. The CUI designation is not supposed to be used to hide embarrassments or problems, yet it is used that way over and over. Moreover, the institutions are making more and more information CUI, precisely because of the lax, undefined nature of the technical designation.

The case relevant to me now highlights this. They see something they don’t like, and rather than dealing with it and taking responsibility, they simply designate it as controlled information. Prying eyes can’t see it. This is done over and over, and because of the lax nature of the law, they act with impunity.

For genuinely classified information, there are supposed to be penalties and admonishments for using classification to hide information through overclassification. In export control law, just as there is no technical specificity, there is no such rule. In no case are these enforced actively. These managers get to do this with impunity. Nothing stops them. It’s simply something they’ve gotten used to doing. So we have a situation where the powerful hold a law they can abuse. They can hide their problems, hide their issues, hide anything embarrassing, and they just get away with it.

To put a point on this whole shitty situation: this harkens back to the same attitude we see with the Epstein class. Rich, powerful people flout the law, violate it with impunity, and let their power shield them. Granted, the things I’m talking about here aren’t nearly as horrible and disgusting as the crimes Epstein committed, but the abuse of power and privilege is exactly the same. They routinely use these laws to punish the little guy while violating them themselves. This is exactly what happened in my case, and I suspect it happens over and over across the entire federal establishment.

Institutional corrosion

“A popular Government, without popular information, or the means of acquiring it, is but a Prologue to a Farce or a Tragedy; or, perhaps both.” — James Madison

Moreover, this is a bipartisan issue. These actions are committed by everyone in a position of power, and they are among the most corrosive forces eroding trust in our leaders and our institutions. It needs to be fixed. Fixing it would be a major step toward re-establishing the trust and confidence we need in our leadership for a better future.

A position of leadership should be the opportunity to set the best example. Instead, in our current society, leaders regularly flout the rules and receive exceptional treatment. Rather than being examples for those they lead, they act with a sense of impunity and entitlement, and that entitlement amplifies the sense that the rules are different for them. The laws and regulations are merely suggestions; their position allows them to violate them without consequence. They do not have to abide by the same constraints, rules, and principles that all of us are required to follow. It is no small statement to say that this is a recipe for disaster, but this is where we are. I can see examples of this in action from the bottom of leadership to the top. Rather than setting an example to follow, they provide proof that power endows you with special privileges and little responsibility, while the serfs toil below.

The fact that these abuses are commonplace and practiced at every level of leadership is not an excuse. Rather, it is a test of the basic values and ethics those leaders possess. If a leader violates the rules or abuses their power, they have failed a fundamental test of character. In each of the examples above, those leaders failed that test. They have shown they are unfit for the positions they hold.

In their hands, the rules and regulations are simply tools of power, instruments for taking retribution against those they don’t like. Secrecy is important, but it is also a reliable engine of distrust, and that distrust is a threat to effective leadership. Our leaders amplify this threat by abusing people with the very rules and regulations they refuse to follow.

“The best weapon of a dictatorship is secrecy, but the best weapon of a democracy should be openness.” — Niels Bohr

“When someone shows you who they are, believe them the first time.” – Maya Angelou

I deleted a couple of posts today (the previous footnote). I had no choice. I received a threatening letter from Sandia that left me no recourse. I’m not going to relitigate the whole episode. It confirms that I worked for unethical and corrupt managers there. It simply amplifies the point. These people have absolutely no business managing anything, much less nuclear weapons. Their professional success indicts the institution.

I had downloaded an unclassified document from the OSTI website. It was marked as such, with a Sandia release number indicating it had gone through information release control (SAND2026-20700). These are designated “UUR,” meaning Unclassified Unlimited Release. I treated it as such. Moreover, the document was indeed completely unclassified. I’m an SME in every topic it contains. In fact, the whole reason I was hired by Sandia in the first place was this very expertise. The report was in the bullseye of my professional expertise.

It holds no sensitive information either — that is, unless you’re sensitive to bad results. Until I received the letter, I had no reason to believe otherwise. Any other designation of the document is complete bullshit. Yes, I was a derivative classifier in a previous life, and I take this shit seriously. Classification at Sandia is far more political than technical. The law has massive gaps in it that empower the management to systematically over-classify. They are using it to bury bad results. Rather than fix the problems, they hide them. In the long run, this will come to no good.

What I can share is an episode that defines the character of the person I believe is at the center of this. He’s only the third-worst person I met at Sandia, mind you. Let’s call him “Cilantro” because he leaves a bad taste in some people’s mouths. It’s a worthy nickname for this piece of shit, and feel free to use it if you have the misfortune of interacting with him.

A while back, Cilantro was a group leader overseeing some nuclear weapons work. I was asked to lead a peer review of an important body of work. Red flag number one appeared almost immediately: the review was held at a very high level of security inappropriate for the work. The extra security was immaterial to the work itself, but it reduced the pool of people available to see the work or judge it. My mentor, for example, was excluded as a result.

During the review, the material driving the designation was shown. It was completely immaterial to the presentation. At the end of the day, the work was reasonable in terms of quality and completeness. But the team presenting it was unnecessarily avoidant and belligerent toward the review team and our questions. They continually pointed to a huge stack of documents for answers, basically refusing to provide any detailed ones. I was genuinely perplexed. The work was good, so why the animosity?

What became clear was that they expected a rubber stamp instead of a review. They were arrogant, and it showed. Cilantro was their group leader. Ironically, the material that drove the designation wasn’t even classified — it was export-controlled. When I wrote the report, I had to classify it at the level of the presentation (SRD Sigma 14). I asked for the report to be declassified to the level appropriate for the information. The person responsible failed to do so despite my repeated prompting. I will note that there are rules against over-classification, and I was endeavoring to follow them.

Eventually, time ran out, and all I had was my first draft of the review. They passed, but the unprofessional and belligerent behavior was noted. I found it counterproductive and antagonistic. The memo went out, and I got it declassified at the last minute. It had not gone to the rest of the review team either.

Then Cilantro called me to his office. He didn’t like the memo. When I got there, unbeknownst to me, another manager was waiting. They ambushed and attacked me. How dare I criticize their people? I should have realized then that Cilantro doesn’t like critique unless it says he’s amazing. He refuses to see problems or do anything about them. He just wanted to berate me, and brought along someone else to make sure it was two on one.

So when Cilantro became a manager above me, I should have known to get the fuck out.

“He who decides one day that scientific statements do not call for any further test, and that they can be regarded as finally verified, retires from the game.”– Karl Popper

When we last spoke, I was reacting to reviews on a paper. It is a paper that touches on Peter Lax’s contributions to modern computational science. Our angle is the practice of code verification, with two main points: the equivalence theorem and shock tube calculations. There the focus is analysis and accuracy measurement. I’ve written previously about how important and valuable this is. We’ve started addressing the reviews and revising the manuscript accordingly. One point in the review prompted this essay. I’ll get to it in a moment.

https://bill-rider.com/2026/05/31/the-foundation-of-code-verification-the-lax-equivalence-theorem/

https://bill-rider.com/2025/03/23/verification-is-essential-verification-is-broken/

I had gotten myself into a good rhythm of writing. That has been derailed. The reviews are actually a minor part of this. My writing energy is going into the revision, a more formal and professional modality. The big thing that has gotten in the way is life. At home we had a water leak in the foundation. This is an emergency and really bad. I’m also in the midst of selling my parents’ home, with an inspection and a title company to deal with. All of this is on a timeline that doesn’t move, especially the leak, which is a “drop everything, fix it now” emergency.

Now back to the task at hand!

One comment in our reviews prompted a response and inspired this essay. In a nutshell, it said that quantifying error in shock tubes is commonly done. There is definitely truth in this. The question is how much truth, and more importantly, how to distinguish our work from the rest. Our work needs to be novel. I think it is, but we need to articulate that better. That is what I’m currently working to express with greater clarity. I wanted to put some meat on the bone for the comment.

Pointedly, what does the literature actually say about these problems? There are thousands of papers that use them. How is our paper, and our use of these problems, different from all of them? This is a rather daunting prospect. I enlisted Claude to survey the literature, sampling for practices with Sod’s shock tube in particular and shock tubes in general. It was a lark on my part. Claude performed beyond my wildest dreams. It was quite impressive. The rest of the essay will elaborate.

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”– Lord Kelvin

The study Claude did confirmed the reviewer’s comment, but only to a point. It took all of 20 minutes, five of which were me writing the prompt. It looked at 500 papers and selected 168 for further study. It surveyed the papers and documented the work. I have a spreadsheet of the papers with the results discussed, and it wrote me a report on the findings. I spot-checked it, but the same study would have taken me days by hand, tedious and time-consuming days. It confirmed my thesis to a large extent. I was actually surprised at how quantitatively stark the statistics were. Notably, astrophysics is the discipline where quantitative accuracy and convergence are actually practiced.

For shock tubes in general, about one paper in 20 computes the error. One in 50 computes convergence rates. Among those, there is little analysis aside from the observation that convergence is first order or less. There is no discussion of the practical implications of the accuracy. The study also pointed to the papers that come closest to what we are aiming to do. Beyond the findings themselves, it was the use of AI to address the question that struck me. I was blown away by how cool and easy this was. One of my coauthors expressed animosity toward using AI. That’s a topic for another day.

Another notable aspect is how many papers engaged in some form of quantitative analysis (about 55%). This was actually a very large number, but it was mainly focused on smooth problems, where the design order of accuracy can be achieved, often in all norms. In a sense, this applies the equivalence theorem very directly on (almost) linear problems, where all the nonlinearity in the methods fades away or diminishes to a few exceptional points. This is an essential part of code verification, but for the applications these codes are written to solve, it is utterly and completely inadequate.

The practice of code verification needs to be more than a check for correctness and a lack of bugs in the code. It needs to advance the methods in practical ways for the problems we write the codes to solve. Bugs and correctness are important, but they are far less than the practice could deliver. We are suggesting expanding the practice into a full partner in progress.

Prompt 1: “I would like a survey of practices in the literature. The question is how often solutions to the Sod Shock tube are used in papers. Then among those how often the error from the exact solution is computed. Then again how often are rates of convergence computed. Using the literature available (arXiv, etc.) can this survey be completed and an analysis given“

Prompt 2: “Rather than simply using Sod’s problem as the focus, extend the study to all shock tube problems having an exact solution. Ask whether the errors in the solution are computed. Ask whether convergence rates are computed. Then are the convergence rates compared with mathematical expectations.“

Code verification is a means of providing evidence and confidence in the proper functioning of our methods. Beyond providing that confidence, it can identify issues in the code and methods, and focus research efforts on addressing, mitigating, or improving them. It is not merely a checkbox. It is a means of producing actionable information from which a proactive research agenda may be defined. Our paper expands the role code verification can take to include computational efficiency.

The Lax-Wendroff theorem has seduced us into a form of false confidence. The vast majority of papers simply plot the solution against the exact solution. No errors are computed. There is an implicit view that the accuracy differences are immaterial. I’ve argued that they are material; the differences are subtle, but important. The lack of direct verification rests on the confidence the theorem provides us. This is a failing. The current literature does not treat the issue with the deference it deserves, nor is it consistent with the sort of confidence we should be endowing our methods with today.

Recognize the full scope of the theorem: it guarantees a weak solution, if it applies. A weak solution is desired, but it is not unique. The theorem does not guarantee a correct solution. That depends on an entropy condition, so the verification question is still alive, yet infrequently engaged. This is not an acceptable state of affairs.

When the Lax-Wendroff theorem is not in play, verification becomes even more necessary, since the guarantees are missing and must be demonstrated directly. You could converge to the correct solution, but assurances are missing. In either case, the differences between methods are reflected in achievable accuracy. That achievable accuracy is then reflected in meaningful differences in the efficiency with which it is obtained. These differences can only be studied quantitatively, and they become far more profound as dimensionality increases. Nonetheless, they are not given the attention they deserve. Our work is trying to move the needle toward better practice.

The other part of the story is how to use AI properly. This was an exceptionally cool idea, and AI did the work extremely fast. It felt like a huge boost in productivity. In my mind, it frees up my time and attention for other things. Today that means solving problems in my personal life. Tomorrow it could mean leisure. It can also free up time and energy for creativity and progress.

“Beware of bugs in the above code; I have only proved it correct, not tried it.”– Donald Knuth

“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding.” – Richard Horton, editor of The Lancet

Prelude

I awoke the other morning to find in my inbox something I had been anticipating and, to be honest, dreading for a while.

I looked at the subject line and muttered, “Oh Shit!”

These were the reviews for a manuscript I was an author on, mostly last year and earlier this year. The emotions this stirred up were quite intense, and I found myself pausing and putting off digesting the reviews for several hours. It got me to think about why I feel so much dread about this. Part of this is my lifelong experience with this process and various episodes with referees’ reports. Most referees are decent and constructive. A few are monsters. They make the entire process brutal and unpleasant. They ruin something wonderful. I will detail what has led to it being a fairly unpleasant experience to read most reviews.

Reviews are Essential

“Criticism may not be agreeable, but it is necessary. It fulfils the same function as pain in the human body. It calls attention to an unhealthy state of things.” — Winston Churchill

Before getting started on the reason for trauma, I would like to elaborate on the topic more broadly. Reviewing the work of peers is an essential part of science. This includes the process for journal articles. I truly believe in the peer review process and its centrality to the quality of the archival literature. It is also a process by which articles are both checked and improved, and it provides a great deal of valuable feedback to authors that is valuable. It also takes place behind the veil of anonymity. This also empowers nasty behavior. This is the source.

The other people who sit at the center of this particular process are the editors and associate editors for any given publication. I have served as an associate editor for a journal and found it a sometimes difficult but often rewarding process. My general observation is that the editors do nothing about reviewer behavior. Part of the issue is that the willingness of reviewers is tenuous. Moreover, some of the worst behavior comes from some editors and leaders in the field. One friend told me that I had chosen a particularly nasty subfield. Many of the leading lights of the field could be brutal and unprofessional. Other subfields were far nicer.

One of the things that occurs to me is the relative degree to which the editors don’t act as more effective gatekeepers in policing some of the more egregious behavior by reviewers. This could go a long way toward making things significantly better. One of the episodes I’ll discuss involved an associate editor directly and speaks to how underlying biases and general viewpoints of the editors can imprint on the overall process.

Done right, the process can lead to an enhancement of the literature. Done poorly, it can leave emotional scars and create baggage for the authors. It can also serve as a relative stagnation of a field. There is also gatekeeping that is counter to progress. In my opinion, there are a variety of biases and problems with the literature that the current process does not fully appreciate and does not do a good job of policing.

I can elaborate on some examples of the problems that I’ve seen in the literature and the reasons for engendering the sorts of emotions that they get. I will elaborate on a few of the more extreme examples in my time. I should note that the current reviews, in retrospect, will not likely stick out in this regard. I’m just noting my preconception and reaction. They’re fairly ordinary, but the emotions that they prompted got me to thinking about the whole dynamic.

As a professional, I do quite a few reviews myself. That number is probably dropping off, but in general, with retirement. The perspective I take in doing reviews is that I try to have empathy and compassion for the authors. This approach is prompted and governed by the experiences I detail below. I try to avoid some of the more egregious examples that I’ll discuss in this essay.

When I do a review, my intention is to provide something that is first and foremost focused on improving the quality of the article. I am acting as a quality filter as well. I am pointing out any lapses or issues with the narrative. That said, I certainly represent a specific viewpoint and opinion on what articles can contain (rather than what they should contain). This is something that I think is impossible to scrub. Nonetheless, I am mindful that my views are not perfect. I do wonder whether or not my own perspective and biases end up producing the effect that I speak out against regularly. God, I hope not!

“We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the—in any case erroneous—comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.” — Albert Einstein

Reviews are Traumatic

My very first papers were in an obscure subfield, space nuclear power. My co-author and professor was a big deal in the field. What I did not recognize was that this shielded me from issues. In this field, he was a criticism deflector. I would soon learn the sort of critiques one could receive without this protection. On the other hand, he was a monster to me. Writing a paper with him was torture. It was part of what drove me away and made me recoil from his influence. I stepped away and found a new path.

“Academic politics is the most vicious and bitter form of politics, because the stakes are so low.” — Sayre’s Law, attributed to Wallace Sayre

When I started on numerical methods for hyperbolic conservation laws, my education and work took a particular arc. The entry point, looking at their methods, was flux-corrected transport. These are methods that were devised by Jay Boris in collaboration with David Book and also include a rather spectacular contribution by Stephen Zalesak. These methods and papers were part of the start of a computational revolution. They sparked my imagination when I read them and tested out the methods. My goal was to understand FCT better.

In continuing my study, certain aspects of the physicist’s point of view from which flux-corrected transport (FCT) began did not sit well with me. I was drawn toward some order, mathematics. There were certain mathematical shortcomings that became obvious, and over time, I became far more attracted to the family of methods that were devised by Bram Van Leer and mathematically sorted out by Ami Harten. In the process, I wrote a paper trying to bridge between these two bodies of work. Unbeknownst to me, I had stepped into a minefield.

What I would find out later is that there was a fairly nasty war for credit between the FCT camp and the Godunov-type method camp. Represented by Jay Boris on one side and Bram Van Leer on the other. I have now heard tales of rather vociferous and nasty reviews of papers being passed back and forth between this set of authors and some depth of hard feelings that were generated as a result. When I wrote my paper on the bridge between the two, I had no such knowledge and walked into this as an innocent. These conflicts were completely out of my sight. I had no insiders to clue me in or warn me.

I had taken a certain perspective in looking at this, which was to embrace some of the mathematical rigor and structure that Harten, in particular, had introduced. I looked to see where that structure would embrace flux-corrected transport and the manner in which it would break down. It uncovered some mechanism for oscillations in results (non-monotone behavior). The way that I did the analysis was to project flux-corrected transport onto the perspective of total variation diminishing methods (TVD). I found that the limits of the method were that it would not produce TVD results. The TVD theory was a way of seeing FCT differently. A path to better understanding. A good idea that was stupid culturally.

“Criticism is prejudice made plausible.” — H. L. Mencken

I wrote the paper and sent it off to the Journal of Computational Physics. The upshot is the paper was never published in the Journal. It got cast into a sort of referee limbo, which also coincided with a tumultuous period with the Journal. The journal passed to new editors and directions. I also knew the identity of all three of the reviewers by the end of it, each a quite famous person in the field. One identified themself, another in a conversation, and the last by administrative error.

1. One of them, Ami Harten, liked the paper immensely and suggested that it be published immediately. This was the first review I received, and needless to say, I was over the moon. He sent me a letter and an analysis of the FCT Ami had published with NASA. Ami had done what I did. I found out why it had never been published. I only spoke with Ami on the phone, never met in person.
2. The second review I received was relatively neutral regarding the manuscript but offered a host of fairly pedantic corrections to grammar and various technical details of the paper. This was the Editor of the Journal, Phil Roe. I would meet Phil later in my career.
3. I got the third review, which was absolutely and unremittingly brutal. I should say that the brutality was matched by the technical skill of the referee, whose knowledge and technical edge were unmistakable. Nonetheless, the review was absolutely savage and, to some degree, has left a lasting mark on my soul. I did meet him once at a conference. He seemed quite nice and absolutely the opposite of the monster in the review.

“Man is least himself when he talks in his own person. Give him a mask, and he will tell you the truth—or his most vicious lies.” — Adapted from Oscar Wilde

I would have the bad fortune of having another paper reviewed by the same person, whose character in the next review was virtually identical. I could see the same approach and use of language. The giveaway was figures in the review that matched an article he wrote, and I had read. Technically well accomplished and absolutely and utterly savage in tone and attitude. The attitude, in fact, burgeoned into something completely unprofessional. There were ad hominem attacks. I suspect I was being treated as if there was a personal grudge. Some of the savagery was a vestige of the “Limiter wars”.

“Peer review is at the heart of the processes of not just medical journals but of all of science… Yet its defects are easier to identify than its attributes.” — Richard Smith

After five years and a change in Journal editorship, I was offered the opportunity to take up the effort to publish it. I declined as I had moved on. It was also right after my second child was born. I had just moved to X-Division as well. That third review, rather than making a better paper, simply killed the article. It left lasting trauma, too. In the meantime, Ami has tragically died far too young. The whole episode left me scarred and bitter. I think this was utterly counter-productive.

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.” — Max Planck

Reviews are Biased

I was talking about journal reviews with my friend and colleague Dana. Like me, Dana worked across multiple technical communities. As I complained about the reviews, Dana interjected, “Why are you working in shocks?” Continuing, “That bunch has the nastiest reviews; things are much different in numerical linear algebra.” This shone a light into a reality, the subcultures could be different. At this point, I can see that it’s harsh and sharp in shocks. In other areas, the culture is polite and constructive. It would be worth getting to the bottom of this. I won’t. Nonetheless, this is the start of bias in the process. These differences are subtle and matters of degree, not wholesale substance.

“Peer review is faith-based, slow, wasteful, ineffective, largely a lottery, easily abused, prone to bias…” — Richard Smith

One of my conclusions is that shock wave computing is highly expert-based and subjective. This leads to power for gatekeepers. Ultimately, this part of the bias exposes the nature of gatekeeping in each subfield. One rejects and removes critique. The second nudges and pushes work toward rigor. Not that it doesn’t happen in CFD; it happens less. It is easy to see which attitude moves the needle of progress, too. Conversely, there are rejections in numerical linear algebra. This is the tendency and the spectrum of responses. As in most cultures, these are taught and reinforce attitudes as acceptable. In a sense, the cultures are taught by the forefathers of a field. I wonder who set the tone originally?

My particular case stepped into the next bias. FCT, TVD, and Godunov-type methods are all great ideas. Each of them has distinct strengths and weaknesses. These small differences become the focus of all-out wars. I had inadvertently stepped into a battle. I took a side without knowing it and invited an ambush. None of this is explained to you until after the fact. I only learned the backstory later as my circle of professional friends “crossed the streams.” Eventually, you understand these things.

The insider who is a student of one of the leaders (or their students) of the field gets a different view. They also get a far more biased perspective. This is usually focused on the hero’s (or his advisor’s) role. It will thus inherit the bias of their advisor. The other side of the dispute is cast as the villain. One side is the other side’s villains. There is the occasional traitor or turncoat. All of this is unleashed onto the reviews. This almost completely explains the problems with my reviews.

Other behaviors are taught. The mode of peer review and critique is shaped. The tone and approach to giving a review are taught as well. Usually, you see this copied quite cleanly. Students often adopt the style of their advisor, rarely just in part. It is very much part of the abused becoming the abusers. In a sense, it can all come out as so much academic hazing. To some degree, I came to see most of the PhD process as having. There is genuine work and knowledge included in a PhD, but make no mistake, hazing. This hazing, if unquestioned, just fully bakes into the culture. We are all responsible for it and subject to it.

“We know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.” – Richard Horton, editor of The Lancet

Fixing the Process

As a starting point, some of these issues shouldn’t be solved. They are simply a reality of a human endeavor. People are biased. Those biases are present in all their actions. Some of these are good biases, such as a value of correctness and clarity. Thus, an article review is always biased. Nonetheless, we are poor at detecting the internal culture of a field from the inside. We should identify those reviewing constructive cultures, and adopt their practices. We should prize and promote progress. Of course, first a field needs to see its problems. Since the leaders are the ones teaching and promoting practices, it seems difficult at best.

Since I really did not have this sort of training. I developed my own protocols. I was more exposed to the ravages of reviews, too. I did not have a wise senior person to keep me from crossing a line. As such, I received the education the hard way. The truth is that the process might be impossible to fix. This may simply be part of a human endeavor. People will be people, and some of them will be assholes. We are tribal. Some of the assholes will only be assholes to people from the other tribes. Anonymity will empower this. Editors do not police shitty reviews. This is especially bad when the shitty review is technically excellent.

A few light rules would help a great deal to engender progress, and reduce the vile trauma”

1. Being anonymous tempts one to be a dick. Don’t be a dick. Maybe consider signing your review and giving that up.
2. Critiques should be focused on making progress and improving the paper. View the paper as salvagable.
3. Check your bias, and try to see the other side.

A big problem with gatekeepers isn’t that they are right, but rather they have power. They want to keep this power. They love the fact that their ideas are accepted and valuable. They do not want to give this up. Progress is a threat.

Maybe this problem just is and will be.

“When a true genius appears in the world, you may know him by this sign, that the dunces are all in confederacy against him.” — Jonathan Swift

References

Rider, William J., and Dennis R. Liles. “A Generalized Flux-Corrected Transport Algorithm I: A Finite-Difference Formulation.” arXiv preprint arXiv:2411.12627 (2024).

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848.

“I may be wrong and you may be right, and by an effort, we may get nearer to the truth.” — Karl Popper

Assessment and Analysis

There is a problem with how verification and validation (V&V) are presented. Too often today, V&V is simply an assessment, divorced from the numerical methods, theory, and physics it examines. The result is a hollowed-out version of V&V. There, an assessment proceeds without deep analysis. This isn’t any connection to a synthesis of expectations from theory or the gaps in that theory. Assessments conducted this way offer no path forward to improving anything. Thus, the assessment mentality is a threat to progress. It is also a threat to taking V&V seriously. It is just bad.

The question is whether the goal of V&V is assessment, plain and simple, or the definition of a path to a better product. In practice, the assessment becomes nothing more than a measure of what a given code or model does. It becomes value-neutral and passive. It typically doesn’t spell out what’s right or wrong. It just is. As such, it becomes entirely optional. Missing entirely are the partnership and the underlying purpose. To understand the proper use or limitations of a code or model. No longer finding the road to a better product: making the errors intrinsic to solving these difficult problems smaller. Gone is driving greater accuracy and fidelity in the code and its models.

This is a genuine obstacle to energizing V&V as a means of continuous quality improvement. Instead of a partner, V&V becomes an enemy. It is either a neutral rubber stamp for whatever is already being done, or a damning indictment of a code. The partnership is never evident, and neither is the connection to improvement. Moreover, the laboratories are at their best when they are multidisciplinary, and the assessment model reduces V&V to a one-dimensional activity. It loses being a multidisciplinary vehicle for excellent work.

The question to wrestle with is this: How did V&V fall into the assessment-only model and the trap that it became?

Part of the answer is the growth in depth of V&V work as it became a separate discipline. This separation was always a possibility as the field matured. It is also a trap because it sidelines V&V into a mode where it offers criticism without support. The pattern settles around a point of view where problems are found but never resourced in a way that allows them to be addressed and solved. Using V&V to improve codes and models becomes absent.

“It is not the critic who counts; not the man who points out how the strong man stumbles…” – Theodore Roosevelt

An Illustrative Example

This was a large part of the issue with the shock verification report that instigated my departure from Sandia. That report was a V&V assessment that only showed results. Any problems were merely weakly implied. The report offers no opinions and has no perspective. There was no route offered to answers, solutions, or improvements. Moreover, there was no charter to provide one; funding applied only to code maintenance and support of the user base. Improving the code in some cases is an unwanted nuisance. The entire notion of quality was presumed, and when the report spelled out that the quality wasn’t there while offering no path toward improvement, the experience turned decidedly negative.

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it… your knowledge is of a meagre and unsatisfactory kind.” – Lord Kelvin

The report that was the focal point of the conflict around my termination is a good example of the problem with assessment only. It was, simply put, a verification assessment of a set of codes used by Sandia for shock problems. The assessment was conducted on two classical analytic problems:

1. The Sedov-Taylor blast wave, where energy is deposited at a point, and a shock wave emanates from the deposition. This is an analytical solution for an explosion, an important use case.
2. The Noh implosion problem, which supplies the solution to an idealized implosion.

Both of these problems are standard at Weapon’s Labs across the World. They are part of a standard test suite for the NNSA Labs. There use goes back far into the past with the code development community at these labs.

The Sedov-Taylor blast wave has a rather complex analytical solution dependent on integral equations and delicate integration. The Noh problem has a simple algebraic form. Both problems are extremely difficult for hydro codes to compute. Their analytical solutions exist by virtue of the infinitely strong shocks they produce. In general, they are challenging for codes, and in particular for codes written in the classical ways of weapons labs across the world. As such, they are important verification problems to complete successfully, because success provides confidence in the code. A lack of success is a flashing warning sign.

The assessment itself was done well within standard practices for verification, with some important caveats. One of my chief criticisms of the report was the lack of deeper mathematical and numerical analysis of the results. In the mathematical sense, there are very distinct expectations that come with solving these problems. I have elaborated on these before in previous posts. These expectations are well defined, and a handful of essential theoretical papers in the literature establish what one should expect from a successful method.

By the same token, the numerical expectations are well defined, chiefly the expectation of obtaining a valid weak solution, as the Lax-Wendroff theorem establishes. In short, the codes tested do not all adhere to the precepts of the Lax-Wendroff theorem, and thus convergence to a weak solution cannot be guaranteed. In other words, they might give wrong solutions that are not valid weak solutions. Thus, the results, although produced classically and competently, are not tethered to any expectations. They simply lie there without context or predictions.

Furthermore, the results show the presence of the infamous carbuncle instability, which has plagued codes for decades. In the aerospace literature, the carbuncle is well understood. It is not so well understood in the context of the sort of codes tested here. Nonetheless, the way to cure the carbuncle is well established, though the cure would need to be adapted artfully to these particular codes. I have faith this is possible, but the report offers little or no discussion of the nature of the problem, let alone a route to its cure.

“When a measure becomes a target, it ceases to be a good measure.” – Marilyn Strathern

A second critique I raised was that the setup of the Sedov-Taylor blast wave was chosen improperly, in two regards:

1. The energy was initialized on the grid within a finite-size region. This introduces a length scale into the problem, and the absence of a length scale is the entire reason an analytical solution exists in the first place. Under mesh refinement, the calculation then converges toward the regularized problem rather than the Sedov-Taylor solution the assessment claims to test against. One cannot solve the problem in a manner that annihilates the conditions for solution. Yet, this is done.
2. The second effect is more pernicious and, in many ways, worse: it makes the problem easier to solve and less challenging for the codes. A choice was willfully made to lessen the challenge that a difficult problem poses, and this only partners with the stagnation of methods and codes, doing a disservice to the entire community. Quality demands that challenges be met, not shirked.

This points to what I believe would be a better way to execute these assessments: combine the assessment with the mathematical, physical, and numerical theories needed to improve results. At a minimum, place the results in the context of what would be technically expected from a correct code and model. It should define why a code or model can be trusted. Rather than a closed and bounded exercise, assessment should be a step. It should define a path for understanding current use and applicability. It should also define the trajectory for improvements.

Why We Need to Integrate V&V

“Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” – W. Edwards Deming

There is a useful analogy for how to view V&V if we see it as a measurement (assessment) methodology.

In medical care, we do not stop at measuring blood pressure unless it indicates good health. If the blood pressure is higher or lower than normal, it requires follow-up. That follow-up means more tests and potentially treatment. The measurement (assessment) is essential, but it defines the next steps. These steps follow a protocol based on the current understanding of medicine. If the measurement is bad enough, the treatment is immediate. In the case of the report, I focused on “a heart attack was imminent.” Rather than treat the problem, the patient decided to ignore the doctor. We all know how this approach works out. If the doctor fails to treat, it is malpractice. If the patient won’t listen, they are being stupid.

“Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product.” – W. Edwards Deming

This brings me to the primary objective of this essay. Rather than simply providing an assessment, the practice of V&V needs to connect results to the appropriate theories. In verification, these are mathematical and numerical results; in validation, physical theory relevant to the exercise is added to the mathematical and numerical foundation. All of this needs to be spelled out in detail, providing both context and meaning. It empowers the reader of the report to take the assessment and find a roadmap to improved results. V&V should not simply be an assessment but a partner in the progress and improvement of computations.

The assessment role for V&V may seem neutral, but it is not. It is a way of neutering V&V. It aids the stagnation of progress and provides cover for codes that refuse to change and improve over time. This does a disservice to the entire community. V&V not only serves the use of the code and model; it serves progress. V&V is used to document and provide evidence for progress. Codes and models should not be static, but should look to be constantly improving in quality. This spirit is on life support.

Verification asks “Am I building the product right?” while validation asks “Am I building the right product?” — Barry Boehm

V&V should be a way of measuring progress and a vehicle for establishing where progress is needed or possible. To identify this, V&V needs to include the relevant mathematical and numerical information for verification as a service to code development. Physical theory was added to establish the same conditions for improving modeling. Together, it becomes the lexicon for establishing capability assessment and identifying needs and opportunities.

The validation work links directly to application work and ultimately supports it in two ways: assessing the suitability of the modeling for a given physical application, and determining whether or not the code is sufficient. It is important to establish where improvements are needed in order to improve that sufficiency. In addition to narrowing the uncertainties inherent in any modeling exercise. This matters for any decision-making attached to the use of modeling and simulation in high-consequence decisions.

Today, this entire standard has become extremely tenuous, and it undermines the role of modeling and simulation. It almost invites unprincipled calibration of results and helps attack the legitimacy of modeling and simulation as a partner in high-consequence decision-making. Assessment only works to neuter V&V and enable stagnation. It only supports the status quo, and the status quo sucks.

Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.” – John Tukey

“One of the greatest pains to human nature is the pain of a new idea.” — Walter Bagehot

The Quest for Understanding

One of the animating ideas of my career in science is a quest to understand the origin, development, and acceptance of ideas. This has been a periodic challenge I took on throughout my career. You will find it in my dissertation, which was mostly self-driven and self-taught. It ultimately became a professional achievement, a more complete accounting of the early history of computational fluid dynamics (CFD). I have written an essay about this on this blog and given a presentation on the topic. It is perhaps the talk I have delivered most often. Beginning in gestational form at the JRV Symposium in 2013. There was a history of CFD around limiters after 1970 by Bram van Leer. It was titled “The History of CFD Part 2”.

I asked Bram, “Where is part 1?”

In response, Bram told me I needed to create part 1 of the history. He said, “That is for you to do!”

Afterward, expanding into a comprehensive presentation that I have given six or seven times over the years. It feels like this will be one of my understated achievements. Under the hood of this history is a desire to understand how we got where we are today. Not simply account for what happened, but why and how.

https://cfd.ku.edu/JRV.html

https://www.sciencedirect.com/science/article/abs/pii/S0021999124001980

Lessons from the History of CFD (Computational Fluid Dynamics)

I have posed the affirmative question before: why should a method be forced to satisfy only one requirement of quality when we know there are two? Those two requirements are the preservation of adiabatic solutions and the conservation form. Right now, the field simply picks one and walks away from the other. We choose rather than demand both. The choice then becomes calcified into doctrine. What I am pondering here is not the choice itself, but the reason we have never insisted on meeting both. The answer is that we refuse to hold ourselves to a higher standard. Progress is still needed and still possible, but it has stagnated to nothing because of an outright refusal to accept good ideas and merge them together. That refusal is not a technical limitation. It is a failure of will.

Conservation. Is it Optional?

Local Technical Cultures

An Encounter with the Great

One of the key formative moments in developing these ideas came in the year 2000. That summer, a special conference took place at Los Alamos to honor the 70th birthday of Burt Wendroff. Burt, except for a few years, worked his entire career at Los Alamos. He is best known for his PhD work under the advisement of Peter Lax. This produced the celebrated dual achievements of the Lax-Wendroff theorem and the Lax-Wendroff method. It was then documented in a paper in Communications in Pure and Applied Mathematics. Both achieved massive success over the years and have shaped modern computational fluid dynamics in profound ways. It has had profound success in compressible aerodynamics, defining much of what is currently done.

Classic Papers: Lax-Wendroff (1960)

Thoughts about Lax’s mathematical philosophy.

Peter Lax’s Philosophy About Mathematics

The occasion of this conference was my opportunity to actually meet Peter Lax. I have met many great scientists over the years. A few became acquaintances, and far fewer became friends. What I have learned from engaging with them is that these people are all human. They are all extremely smart. Some are extremely lucky, and their greatness comes from being smart enough while also being in the right time and place to create what they create. Of all the people I have met, Lax is perhaps the greatest, or nearly the greatest. That puts him in rather prominent company.

There was a photograph taken at the conference on the first day. I remember it quite well. I don’t actually remember who took the photograph. Burt and Peter came together warmly, and then the photographer noted that the ordering was wrong. He asked Peter and Burt to exchange places to get the ordering of the authorship for the Lax-Wendroff paper. Then the photo was taken. I was standing behind where the photo was taken, watching all this happen. It remains a very fond memory. Its great that the internet remembers the photo too.

Another source of great memory in that conference was my presentation there. I felt immensely lucky to do so. One of the more memorable moments came when Burt pulled me aside to let me know that Peter suffered from narcolepsy. I should not be alarmed if he fell asleep during my talk. Sure enough, when I gave my talk, Peter fell asleep halfway through. Bert’s warning saved me from being mortified, as Peter had already become quite the hero of mine by then.

My talk came from a period of my technical history when I was beginning to explore turbulence modeling. My interest was specifically in implicit large eddy simulation. I was trying to understand the role that things like limiters play in the ability of these methods to act as effective turbulence models. In addition, I was searching for the effective subgrid model implied by these methods. In particular, I was looking at the role of nonlinear dispersion.

Speaking in front of Peter was a particular honor, but also something that made me quite nervous. Peter had studied dispersion in calculations, inspired by the work of John von Neumann. In his original shock method, which was tried and successfully used during World War II, dispersion was rampant. Lax examined the solutions von Neumann’s method created, which produced a huge amount of ringing. This work was done in conjunction with another exceptional scientist, David Levermore. Presenting my work to Peter was both a source of pride and something that felt very dangerous. He was an exceptional genius after all, even by Los Alamos standards.

Nonetheless, looking back, this work was the beginning of perhaps my most successful research. It certainly grew into the best and most obviously successful project I ever worked on. One that resulted in many highly cited papers and a book. That book tied together many researchers in the area for the first time. It led to my engaging and meeting many other scientists working on similar things, including Jay Boris, one of the inventors of limiters. Paul Woodward was in the book too, but I already knew him.

Expanding My Understanding of Creation

“Novelty emerges only with difficulty, manifested by resistance, against a background provided by expectation.” – Thomas Kuhn

Over time, I came to understand how many of these ideas came into being. How they merged with other ideas. How some of them became distorted and lost their original intent. This is perhaps more evident in turbulence simulation than anywhere else. There, the Smagorinsky model is the original model used to represent subgrid turbulence in LES. Over the years, the basic identity of the Smagorinsky model has been lost. It was first and foremost a rearticulation of the artificial viscosity model developed by Richtmyer in conjunction with von Neumann. It has been repurposed to clean up calculations of geophysical flows. This repurposing came at the suggestion of Jule Charney, von Neumann’s collaborator in applying computing to geophysical fluid mechanics. Over the years, this connection to shock capturing was lost, if not outright ignored. This seems to be a combination of genuine and willful ignorance.

The greater animating purpose of this essay is the general lack of acceptance and use of Lax’s work at its place of origin, specifically Los Alamos. It includes the many labs that followed Los Alamos’ lead. This includes Livermore, Sandia, and overseas Labs. This requires a bit of history, involving how Lax came to these ideas. Then, some deeper pondering of why these key breakthroughs have generally gone unaccepted and unused at the place where they were born.

Lax’s ideas and achievements in the area of hyperbolic conservation laws are incredible and undeniably great. This was recognized when he received the Abel Prize in 2005. The fact remains that they hold very little sway and acceptance at places like Los Alamos or Livermore seems mysterious. They are revered in CFD around the rest of the World. This is the world shaped by the Manhattan Project. This ushered both computers and computational science into being. The original achievements of John von Neumann are more broadly accepted in the Los Alamos-connected work. Even though they lack the rigor Lax provides. Why?

“Most men can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.” – Tolstoy

Consider the Lax equivalence theorem as another example. The practice of verification that relies upon this theorem is carried out begrudgingly in places like Los Alamos. The same holds for the Lax-Wendroff theorem. The theorem with its demands, and its promised benefits, of conservation form. In a deep sense, it has greater rigor and applicability than the equivalence theorem. Conservation, too, is seldom used. The extensive mathematical theory of hyperbolic conservation laws, the crowning achievement of Lax’s work in the area, is also rarely leaned upon as a technical basis either.

The Foundation of Code Verification: The Lax Equivalence Theorem

Given the greatness of Lax’s work, one has to question the reasons. The question becomes even more pregnant when you realize that the origin of this work traces directly to Los Alamos. It is grounded in the achievements of von Neumann, which demonstrated the capacity of computers to solve this class of problems. This inspiration made Lax’s work timely and essential. It was clearly a motivation for greater understanding. Much great math was delivered, yet not used where the inspiration originated.

“One resists the invasion of armies; one does not resist the invasion of ideas.” — Victor Hugo

Lax was drafted into the U.S. Army in 1944. He was ultimately assigned to Los Alamos as part of the Manhattan Project, a recognition of his potential and burgeoning genius. There, he did not work on fluid dynamics, but rather on neutron diffusion. After leaving Los Alamos, he returned to New York, where he completed his PhD at the Courant Institute under Friedrichs. Upon receiving his PhD, he promptly went back to Los Alamos and spent a year there as a staff member. He worked closely with one of the famous Keller brothers, both great mathematician.

One of the most amazing things to discover was the write-up of Lax’s plan of attack on the mathematical theory of hyperbolic conservation laws. It was completed in conjunction with Keller during his time at Los Alamos. The outline in this report is complete and defines the next 25 years of research. One can trace all of Lax’s completed works in conservation laws from it. Ultimately, ending with the mathematical theory of hyperbolic conservation laws published in 1973. It is all laid out there in astonishing detail.

Lax, Peter D. Hyperbolic systems of conservation laws and the mathematical theory of shock waves. Society for Industrial and Applied Mathematics, 1973.

The advantage Peter had when he returned to Los Alamos was the knowledge that one could successfully compute solutions to hyperbolic conservation laws. The basic premise had been proven out in World War II by Feynman and Bethe with their method, based on Tony Skyrme’s work. By the time Lax returned to Los Alamos, Richtmyer had devised the artificial viscosity and made shock-capturing methods possible. Now computers and methods could be turned loose on shock waves. Thus, Lax worked with the knowledge that all of this could be done. It was now a matter of bringing order to it, along with some degree of mathematical rigor and knowledge to guide it. The question remains: why has his magnificent work had so little impact on its place of origin?

How Original Ideas Remain Dominant?

“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.” — John Kenneth Galbraith

If one looks at the codes developed and used at a place like Los Alamos, only one of them is written in what we call conservation form. This is a code I worked on, one that goes by the moniker xRage these days. My friend Rob and I could not ascertain the basic structure of the high-order Godunov code that xRage purported to be. Rob had trained under Phil Roe and Bram van Leer for his PhD. I am basically self-trained in the same area. The basic recipe, construction, and maxims that Rob and I had absorbed in our training could not be discovered in the way this code was written. We expected to find a viable first-order Godunov method under the hood. No such thing existed in the code. The second-order spatial differencing was novel as well. It worked, but not in a standard way. The same mentality applied to the Riemann solver and the multi-material treatment. I was part of adding interface treatments to the code. Without those, the material interfaces were too diffusive.

This code was originally written by a genius of a code developer, Mike Gittings. Mike was able to work magic with codes and get things to work by magic and code wizardry. The code worked and was astoundingly robust, loved by its users. In many ways not much different than the dynamic at Sandia with CTH (or its predecessor, CSQ). Yet the technical basis of the code was unrecognizable to those of us who had been formally trained. Here is the one example in the weapons complex where Lax’s framework might actually apply, and it is almost impossible to see what is actually there. To me, this is rich with irony, especially at Los Alamos.

“Nature, to be commanded, must be obeyed.” — Francis Bacon

One of the things that I believe is that the importance of hydro for multi-physics codes is not well understood. I was reminded recently in a LinkedIn comment by my friend Nathaniel Morgan from Los Alamos. The reason it is so important is that it provides the material map on which all the physics in a multi-physics code are computed. It also provides the state of that material thermodynamically and, more importantly, its position. Physicists naturally think about most of the physics in the Lagrangian frame of reference. Thus, this is comfortable for them. Thinking about it in a different frame of reference only muddies the picture and introduces new physical effects. The Lagrangian frame perspective is taken as primal.

“For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.” – Richard Feynman

That said, most of the physics that we are solving are extremely muddy and difficult. In almost every case where you’re dealing with a high-speed, high-energy flow, turbulence and mixing are present. As soon as the flow begins to twist and distort, the Lagrangian frame becomes untenable, and you lose the ability to clearly think about problems in that frame of reference. In a sense, this is the core of the problem with failing to progress away from the von Neumann view of how to compute this. Von Neumann’s method was originally composed in 1D with relatively limited computing. It is almost obvious that the simplicity it embodied is not appropriate for the type of computing and expressibility of ideas we have available to us today. Why is this approach still relied upon?

The reason why is the power of being the first mover. That first mover, the person whose reputation remains unsullied, is John von Neumann. Von Neumann is perhaps the greatest polymath of the 20th century, a genius of almost unparalleled magnitude, and perhaps one of the smartest people in the history of humanity, certainly on par with someone like da Vinci.

If one looks at the codes in use across the labs today, the model of computing is very much derivative of von Neumann’s method, paired with Richtmyer’s artificial viscosity. What cannot be said is that these codes bear any of Lax’s rigor in how they are used. Given the power and quality of Lax’s work, I have always found this mysterious. Moreover, much of what Lax did actually generates genuine animosity on the part of people at the labs. I find this difficult to understand. In a field with few results to lean upon, some of the best ideas are ignored. This entire phenomenon places an extreme amount of importance on the legitimacy conferred by being first and on the power of incumbency. The nature of things is that once something works at all, it is hard to displace, even when its shortcomings become increasingly obvious over time.

“He that will not apply new remedies must expect new evils; for time is the greatest innovator.” — Francis Bacon

Indeed, the main place where ideas related to Lax’s work found acceptance is through the work of van Leer principally, and limiters in general. Limiters provided a means to solve a problem that Lagrangian von Neumann-type methods ran into in multiple dimensions: the inevitability of mesh tangling. Mesh tangling meant that remesh-and-remap technology was necessary for these calculations to go to 3D and ultimately solve the problems they were designed to solve. Van Leer’s method made low dissipation and higher accuracy possible under these conditions, and it was accepted almost immediately across the entire lab complex.

What goes unacknowledged is that van Leer’s work was shaped by the work of Peter Lax. Lax provided the foundation, and everything van Leer did was built upon it. In a sense, Lax’s work did have its impact, but only in a derivative sense. Van Leer also engaged with Paul Woodward from Livermore. Paul’s collaboration with Bram helped bridge the methodology to the Labs. Van Leer’s methods were almost immediately adopted at Livermore, Los Alamos, Sandia, and AWE in England. It was rapid and pervasive.

These methods, once they leave the Lagrangian frame, have a problem: they do not conserve energy. This comes from the form of the energy equation used. There is a method to conserve energy devised originally by Roger DeBar of Livermore. Generally, that method is not robust enough to be used for practical problems. Scientists at Los Alamos have devised a much better version of it that may be good enough for practical use. Nonetheless, it is not in an obvious flux-conservation form; thus, Lax-Wendroff may not apply. The same thoughts are needed for conservation in the Lagrangian frame for staggered mesh methods. It can happen, but it depends on subtle, discrete details. For example, the first version to conserve relied upon a bizarre definition of kinetic energy (could be negative). Modern methods are strictly flux-conservative with some very specific choices. These include the time integration, where a particular predictor-corrector of second-order is needed. This also includes a corner mass invariant definition for momentum-kinetic energy.

This lack of energy conservation is exactly the underlying is that prompted my decision to retire. Even 40-plus years after van Leer remap became standard, energy conservation is uncommon. There are ways to make it work, but they are seldom used in practice. It is not in conservation form; thus, Lax-Wendroff is not applicable. I find this head-scratching and difficult to square with the importance of the work done with codes.

“The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already.” — Tolstoy

Here we are, 65 years after the Lax-Wendroff theorem was published. People remain completely unwilling to acknowledge this work or utilize its fundamental results in what they do. This rejection creates an absolute crisis of legitimacy, and it reflects decisions made over and over again. I have asked my friends at Livermore, and they report that the energy-conserving methodologies developed at Livermore, which do not use conservation form, are not utilized in their codes once they go into a mode using remap. Those codes have the same problems, albeit to a lesser degree than CTH. That is more a reflection of the higher quality of the methods, and of being modern codes. Nevertheless, the basic premise of this essay persists: Lax’s work is not accepted at its place of origin.

“It is difficult to get a man to understand something, when his salary depends upon his not understanding it.” — Upton Sinclair

I have written about this mentality before, in my essay on the shortcomings of current methods, where the field has chosen to honor the physical concept of computing adiabatic solutions rather than computing solutions in conservation form. The unwillingness to demand that methods meet both requirements means that the method that routinely and casually produces adiabatic results is chosen. Conservation is rejected as a preeminent requirement. Really, we should have both requirements met.

A Methods Challenge Worthy of Being Called World-Class

I will reiterate that I believe this is a fundamental mistake. The requirements and gifts of conservation exceed those of an adiabatic solution. Moreover, an adiabatic solution is a desired outcome, but it is utterly pathological, representing a rather profound resistance to the second law of thermodynamics, which manifests itself in mixing and turbulence most often in fluid dynamics. Adiabatic solutions are ephemeral and pathological. That means they shouldn’t be the foundational character of the method. They are wonderful if you can engineer them. Making them the premise upon which you design and accept new numerical methods is an act of faith, if not borderline lunacy.

Adiabatic solutions are largely an article of faith, mostly mythological. Shock waves are ubiquitous. You want to compute a weak solution, and you need to compute the physically relevant weak solution. I place this demand first, and the preservation of adiabatic solutions second, based on this analysis. Accepting what people have always done is easy. Change is hard. This alone explains most of what we see. The incumbent has an immense advantage. Without a devotion to progress and change, you simply do things the way you always have. You make excuses about why change is unnecessary, and you point to the successful track record of the past as the only proof you need to keep doing things the same way.

“The difficulty lies not in the new ideas, but in escaping from the old ones, which ramify… into every corner of our minds.” — John Maynard Keynes

The Essence of the Problem: Change is Hard

When I consider what John von Neumann would have thought of all this, I come to the following conclusion: he would have recognized the correctness and genius of Lax’s work. He was ever devoted to progress, and he would have seen the need to merge Lax’s ideas into his own. He would not be pleased that the ideas he pioneered continue to be used without the modifications necessary to make them more reliable and more generally useful for solving humanity’s most enduring and difficult problems. Perhaps we can turn over a new page, get to the place where we combine these ideas, and give each of them the air they deserve and the progress we all need.

Power from being the first mover, or from initial success in something, comes from the fact that it works. It’s very similar to the advantage conservatives have over progressives. The policies conservatives espouse have already been tried and have some functional basis in society. The same happens in science: why change something that works for something new? For this reason, very old solutions with significant flaws live on. Change is hard and uncertain. This is especially true in a world where trust itself is scarce and increasingly existential.

There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things.” — Machiavelli