“He who decides one day that scientific statements do not call for any further test, and that they can be regarded as finally verified, retires from the game.”– Karl Popper

When we last spoke, I was reacting to reviews on a paper. It is a paper that touches on Peter Lax’s contributions to modern computational science. Our angle is the practice of code verification, with two main points: the equivalence theorem and shock tube calculations. There the focus is analysis and accuracy measurement. I’ve written previously about how important and valuable this is. We’ve started addressing the reviews and revising the manuscript accordingly. One point in the review prompted this essay. I’ll get to it in a moment.

https://bill-rider.com/2026/05/31/the-foundation-of-code-verification-the-lax-equivalence-theorem/

https://bill-rider.com/2025/03/23/verification-is-essential-verification-is-broken/

I had gotten myself into a good rhythm of writing. That has been derailed. The reviews are actually a minor part of this. My writing energy is going into the revision, a more formal and professional modality. The big thing that has gotten in the way is life. At home we had a water leak in the foundation. This is an emergency and really bad. I’m also in the midst of selling my parents’ home, with an inspection and a title company to deal with. All of this is on a timeline that doesn’t move, especially the leak, which is a “drop everything, fix it now” emergency.

Now back to the task at hand!

One comment in our reviews prompted a response and inspired this essay. In a nutshell, it said that quantifying error in shock tubes is commonly done. There is definitely truth in this. The question is how much truth, and more importantly, how to distinguish our work from the rest. Our work needs to be novel. I think it is, but we need to articulate that better. That is what I’m currently working to express with greater clarity. I wanted to put some meat on the bone for the comment.

Pointedly, what does the literature actually say about these problems? There are thousands of papers that use them. How is our paper, and our use of these problems, different from all of them? This is a rather daunting prospect. I enlisted Claude to survey the literature, sampling for practices with Sod’s shock tube in particular and shock tubes in general. It was a lark on my part. Claude performed beyond my wildest dreams. It was quite impressive. The rest of the essay will elaborate.

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”– Lord Kelvin

The study Claude did confirmed the reviewer’s comment, but only to a point. It took all of 20 minutes, five of which were me writing the prompt. It looked at 500 papers and selected 168 for further study. It surveyed the papers and documented the work. I have a spreadsheet of the papers with the results discussed, and it wrote me a report on the findings. I spot-checked it, but the same study would have taken me days by hand, tedious and time-consuming days. It confirmed my thesis to a large extent. I was actually surprised at how quantitatively stark the statistics were. Notably, astrophysics is the discipline where quantitative accuracy and convergence are actually practiced.

For shock tubes in general, about one paper in 20 computes the error. One in 50 computes convergence rates. Among those, there is little analysis aside from the observation that convergence is first order or less. There is no discussion of the practical implications of the accuracy. The study also pointed to the papers that come closest to what we are aiming to do. Beyond the findings themselves, it was the use of AI to address the question that struck me. I was blown away by how cool and easy this was. One of my coauthors expressed animosity toward using AI. That’s a topic for another day.

Another notable aspect is how many papers engaged in some form of quantitative analysis (about 55%). This was actually a very large number, but it was mainly focused on smooth problems, where the design order of accuracy can be achieved, often in all norms. In a sense, this applies the equivalence theorem very directly on (almost) linear problems, where all the nonlinearity in the methods fades away or diminishes to a few exceptional points. This is an essential part of code verification, but for the applications these codes are written to solve, it is utterly and completely inadequate.

The practice of code verification needs to be more than a check for correctness and a lack of bugs in the code. It needs to advance the methods in practical ways for the problems we write the codes to solve. Bugs and correctness are important, but they are far less than the practice could deliver. We are suggesting expanding the practice into a full partner in progress.

Prompt 1: “I would like a survey of practices in the literature. The question is how often solutions to the Sod Shock tube are used in papers. Then among those how often the error from the exact solution is computed. Then again how often are rates of convergence computed. Using the literature available (arXiv, etc.) can this survey be completed and an analysis given“

Prompt 2: “Rather than simply using Sod’s problem as the focus, extend the study to all shock tube problems having an exact solution. Ask whether the errors in the solution are computed. Ask whether convergence rates are computed. Then are the convergence rates compared with mathematical expectations.“

Code verification is a means of providing evidence and confidence in the proper functioning of our methods. Beyond providing that confidence, it can identify issues in the code and methods, and focus research efforts on addressing, mitigating, or improving them. It is not merely a checkbox. It is a means of producing actionable information from which a proactive research agenda may be defined. Our paper expands the role code verification can take to include computational efficiency.

The Lax-Wendroff theorem has seduced us into a form of false confidence. The vast majority of papers simply plot the solution against the exact solution. No errors are computed. There is an implicit view that the accuracy differences are immaterial. I’ve argued that they are material; the differences are subtle, but important. The lack of direct verification rests on the confidence the theorem provides us. This is a failing. The current literature does not treat the issue with the deference it deserves, nor is it consistent with the sort of confidence we should be endowing our methods with today.

Recognize the full scope of the theorem: it guarantees a weak solution, if it applies. A weak solution is desired, but it is not unique. The theorem does not guarantee a correct solution. That depends on an entropy condition, so the verification question is still alive, yet infrequently engaged. This is not an acceptable state of affairs.

When the Lax-Wendroff theorem is not in play, verification becomes even more necessary, since the guarantees are missing and must be demonstrated directly. You could converge to the correct solution, but assurances are missing. In either case, the differences between methods are reflected in achievable accuracy. That achievable accuracy is then reflected in meaningful differences in the efficiency with which it is obtained. These differences can only be studied quantitatively, and they become far more profound as dimensionality increases. Nonetheless, they are not given the attention they deserve. Our work is trying to move the needle toward better practice.

The other part of the story is how to use AI properly. This was an exceptionally cool idea, and AI did the work extremely fast. It felt like a huge boost in productivity. In my mind, it frees up my time and attention for other things. Today that means solving problems in my personal life. Tomorrow it could mean leisure. It can also free up time and energy for creativity and progress.

“Beware of bugs in the above code; I have only proved it correct, not tried it.”– Donald Knuth