“Your life is not a simulation; it’s the real game. Play wisely.”― Richelle E. Goodrich

Direct numerical simulation (DNS) is one of the most powerful uses of our vast computing power. With that power comes great responsibility. That responsibility is currently not being met by the vast majority of practitioners. The common issue is a lack of attention to accuracy. This is basic quality control. Some of what gets called direct numerical simulation is nothing more than marketing for the extremely expensive, powerful computers. Marketing because we spend so much time and money on them.

Numerical simulation in general is not practiced with the care its promise deserves. That promise is access to vast quantities of precise data that rival experiments in their power to unveil the mysteries of the universe. Much of the problem comes down to verification and validation. These activities are essential for ensuring the quality of computations. As a rule, DNS does not lend itself to high-quality verification and validation (V&V). Instead, they rely on rules of thumb and expansive claims about accuracy. Many of the people who consume DNS results treat a DNS as equivalent to a declaration that the results are exact. This is a patently absurd notion that should be rejected reflexively.

I have written about this before, and I will reiterate some of the main points here. Over the past ten years, I have encountered these practices more frequently, engaged with some of the most prominent practitioners, and gained perspective. It is also worth mapping perspectives on DNS onto the claims now being made about AI. As it turns out, the two subjects are closely connected. The hubris and the sweeping claims surrounding DNS feel like a reflection of the hubris and the sweeping claims about AI.

“The simulation had now become indistinguishable from real life.”― Ernest Cline

Questions about the legitimacy and accuracy of DNS are best framed in two ways. First, whether the physical laws being solved to high accuracy actually describe the physical phenomena of interest. Next, does the accuracy of the numerical treatment meet requirements? Second, the numerical treatment itself. Numerical solutions to the equations of physics, typically partial differential equations, are intrinsically approximate, and those approximations carry errors. In general, both the physical model and the numerical method are assumed to be highly accurate. It is damning that the errors associated with them are rarely, if ever, estimated and reported as part of a DNS study.

A good place to start is the most common and well-known version of DNS: Navier–Stokes fluid turbulence. This is the practice that made DNS famous, and it is often the most well-developed approach. As a result, it also exhibits almost all of the common pathologies. Both the good practices and the pathological ones deserve discussion, because the latter probably require more care than they are usually given. The habits of research communities often run counter to better practice, and they can encourage some of the more egregious examples of overreach and missing quality control.

“The Navier-Stokes equation probably contains all of turbulence.” – Uriel Frisch

This form of DNS begins with the widely accepted contention that the incompressible Navier–Stokes equations contain all of turbulence. Uriel Frisch states this explicitly in his book Turbulence. I think the claim deserves more scrutiny than it gets. For one thing, all of these physical laws are to some degree approximations of continuum behavior, behavior that is itself non-continuum in nature. The deeper problem is that incompressible flows do not exist in nature. There is no such thing as an incompressible flow. This is easy to see: an incompressible flow implies an infinite sound speed, or, as a friend from Los Alamos used to quip, superluminal sound waves (sound traveling faster than light). What incompressibility really does is eject thermodynamics from the system of equations in any meaningful sense. Given that fluid turbulence remains a mystery, throwing thermodynamics out of the equations seems more than a little foolish.

“The law that entropy always increases holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations – then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation – well, these experimentalists do bungle things sometimes. But if your theory is found to be against the Second Law of Thermodynamics I can give you no hope; there is nothing for it to collapse in deepest humiliation.” ― Arthur Eddington

The folly runs deeper when you consider some of the best-known facts about turbulence. The first is the broad acceptance that turbulence has, in some form, a singularity associated with it. Proving the existence or non-existence of that singularity, whether smooth solutions exist for all time, is the essence of one of the Clay Millennium Prize problems. The singularity is seen most clearly in the famous Kolmogorov four-fifths law, which shows that as viscosity goes to zero, dissipation approaches a finite value. The compressibility that has been ejected from the equations is precisely a mechanism by which a singularity would naturally form; this is the same way one forms in standard compressible flow.

It would be a genuine irony if it turned out that turbulence has little or nothing to do with the incompressible flow equations. This would then mean that the Clay prize itself is meaningless. The solution would simply be an oddity of higher mathematics. The non-solution to the problem is probably telling us something! It would then be nothing more than the study of a challenging and oddly difficult class of equations that were believed to have physical significance, but in reality had little, other than as a useful approximation for a broad class of flows that does not include turbulence.

One key feature of compressible flows is the presence of a clear, phenomenological structure that leads to the formation of the singularities the four-fifths law points to. The same structure and dynamics appear in shock wave formation and propagation. The dissipation, or entropy creation, rates are functionally similar, being cubic in the difference in longitudinal velocity. The main difference is that compressible flow has a theory that only works in one dimension, whereas turbulence is a three-dimensional theory. What you have in turbulence is a field looking futilely at a horrendous physical system, incompressible Navier–Stokes, while pushing aside an obvious solution to the problem, compressible Navier–Stokes.

What we have is the pursuit of an essential physical theory using a system of equations that combines hyperbolic, parabolic, and elliptic forms, and that refuses to yield to the most powerful mathematical analyses available to mankind. We still do not have any constructive proof of a singularity. By removing the unphysical aspect of this system, the divergence-free velocity, we get singularities forming naturally. This is a well-posed system that matches the kinds of singularities and rate-of-production behavior we expect from theory and experiment. Frankly, it boggles my mind that we continue to pursue this theory down the incompressible rat hole.

Incompressibility removes sound waves from the equations, and it also removes thermodynamics. The key point is that sound waves are the precise physical mechanism in compressible flow that produces singularities. That is the other essential nonlinearity that the incompressible flow equations make completely degenerate. Frankly, it is no wonder we have failed to make real progress in nearly a century. This is the first and perhaps most important objection to current DNS practice.

The second concerns the numerical methods and the integration of the equations. The prevailing standards rest on rules of thumb established in the foundational channel-flow simulations of the early-to-mid 1980s, with resolution set relative to the Kolmogorov length. These give rough accuracy bounds; a stated error on the order of five percent is commonly used to set resolution. This is best defined in Moin and Mahesh’s review paper of 1998. It deserves more scrutiny. The current rules of thumb produce flows that look reasonably well resolved, but there is no well-established sense of the error. Usually, there is no real knowledge of the numerical errors incurred in integrating a DNS. To put it bluntly, error bars do not exist for these calculations. Where error bars do appear, they almost always reflect the statistical convergence of computed quantities, not the numerical error of the solution.

“This defines the minimum scale, the size of the smallest feature in the flow.” – Henshaw, Kreiss & Reyna

The Kolmogorov length is an energy-norm scale that marks where dissipation occurs in a turbulent flow (L2 norm). To say the least, it yields a fairly optimistic view of how computational effort scales with Reynolds number. Others have taken even more pessimistic views, most notably Kreiss, who worked from an L∞-norm length scale. The question is what is the necessary scale to resolve? That estimate puts detailed simulation of turbulence completely out of reach for any meaningful Reynolds number. This may well be the right view, if singularities are the heart of turbulence and the proper focus of any DNS. If turbulent flows are weak solutions perhaps a L1 norm view would be appropriate. My fear is that it is true: that the resolution of singularities in turbulent flow is exactly the secret we are missing, and the breakthrough we so badly want.

Now consider the cultural side of DNS practice. The published literature, and the credit for contributing to our knowledge of turbulence, is driven by computing DNS at the highest Reynolds number possible. That pursuit leads to corner-cutting and less care, which works directly against the questions raised above and against the error estimation and quality assurance the field so badly needs.

The field needs high quality because DNS is so often used to replace or augment experimental data. When computation stands in for experiment, it should be held to the same standards as experiment, the same rigorous procedures. Actually arguably to higher standards, since this is a man-made source of data. In almost every respect the opposite is true. DNS is simply assumed to be like experimental data, only more copious and easier to obtain, at least once you have the high-performance computing needed to produce it.

The same trends appear in other fields that use “first-principles” calculations to do DNS. In molecular dynamics, for example, potentials are used to describe the behavior of molecules. These potentials are highly accurate, but still approximate and imperfect, compact descriptions of the physical behavior rather than the behavior itself. The same mindset prevails: the prize goes to the biggest, most expansive, largest-scale simulation one can achieve. All of it works against the pursuit of quality. V&V is largely absent and surplus to requirements.

“It takes less time to do a thing right than to explain why you did it wrong.”― Longfellow

Finally, you reach the ragged edge of what gets called DNS. These are the simulations that are largely marketing exercises on the part of institutions looking to promote themselves. Here a DNS is simply a very large-scale calculation.

I have seen a great deal of this at the national labs, where you will find a code solving the Euler equations together with some other combination of physics to produce a very expensive, very detailed model of some system. It gets promoted as a DNS purely on the strength of the computing resources consumed. The calculation is enormous, and it is called a DNS by virtue of being massive.

This is not to say such exercises are useless. Calling them DNS does a disservice to every other DNS, and lends them an air of legitimacy and truth they have not earned. They are best understood as exploratory attempts to explain complex phenomena, a worthwhile and valuable use of computing, but not direct numerical simulation. What they really are is marketing, for the very expensive computers and the very expensive programs these laboratories are engaged in. That institutions get away with it calls into question the nature of peer review and the quality of the broader scientific enterprise they are part of.

All of this brings us full circle, back to ideas related to AI. The current push for computing at a massive scale is focused on AI, and you have the same claims that massive quantities of data and computing lead to some sort of magical access to the truth. Fortunately, we have already seen through this, in the much-noted discussion of how often large language models hallucinate and tell falsehoods. That is largely a positive thing to consider going forward. We see the problem; now we need to solve it.

A deeper issue to consider is whether the hallucinations we see in AI are also present in DNS. Do DNS results hallucinate as well, and if so, how do we find them? In both cases, identifying and eliminating these hallucinations is a key technological advance worth pursuing. Given the economic, political, and national security consequences of AI, that pursuit moves over into something much closer to a life-and-death struggle.

There is a clear path forward for both DNS and AI. This is V&V and lots of it. The approach to making progress is straightforward, even if the work is detailed, technically demanding, and requires handling uncertainty and the fidelity of calculations. That is why we fund research in the first place, right?

In both areas, the first priority should be genuine measurement and testing of accuracy, with a clear understanding of the error uncertainty and the computational cost. This applies whether we are dealing with DNS of turbulence or a LLM. Developing this accuracy is essential, because it is not simple to measure and has multiple layers.

“A brand that feels human earns something no algorithm can replicate: trust” ― Warren Kornblum

Efficiency is the second key factor. In other words, how much computing cost is needed to achieve a certain level of accuracy? To know the efficiency, knowledge of accuracy is essential. In the AI case, for example, the cost per token has become a problem. The recent approach to tokenmaxxing has led some companies to withdraw support for AI and reassess its utility and value. This is positive, if we focus on improving efficiency and avoid trying to solve problems by throwing more and more computing power at them. How efficiently and effectively we use the computing power we have matters greatly, and this has been a problem across computational science.

“There is nothing so useless as doing efficiently that which should not be done at all.” ― Peter F. Drucker

Ignorance of accuracy and efficiency has led to stagnation in methods and methodology. This comes with an attitude that says, “Methods are done, there is nothing to do here.” Nothing could be further from the truth. This stagnation is antithetical to progress. We see it in the quest for high-resolution methods, where an obsession formal high-order accuracy has killed the ability to develop more efficient, more effective methods. There we never measure accuracy on practical problems. In either case, the joint focus should be on the accuracy and fidelity achieved on practical, real-world problems, using idealized problems only to guide us. Only using idealized problems where accuracy there can be directly tied to accuracy in the real world.

Brute-force computing is an amazing thing to have. What has become clear is the vast cost of that computing. It is becoming a huge technical and political issue. We should feel duty-bound to use it as effectively and efficiently as possible. This pursuit of efficiency should be a unifying principle across the world of computational science, driving important, real-world impacts.

We should also recognize that the pathologies of high-performance computing that have consumed computational science for the past decade or more are now being inherited by AI. The whole notion of data centers is the sharp end of the spear here. The AI world needs to be more mindful about using that computing power efficiently and effectively. The unfortunate thing is that there is an obsession with raw computing power, without regard for the efficiency or the accuracy that results from its use. This has been a plague on the field, and it needs a correction sooner rather than later.

“People don’t buy what you do; they buy why you do it. And what you do simply proves what you believe”― Simon Sinek

References

Bethe, H. A. “The Theory of Shock Waves for an Arbitrary Equation of State.” Office of Scientific Research and Development, Report No. 545, 1942.

Frisch, Uriel. Turbulence: The Legacy of A. N. Kolmogorov. Cambridge: Cambridge University Press, 1995.

Henshaw, William D., Heinz-Otto Kreiss, and Luis G. Reyna. “Smallest Scale Estimates for the Navier–Stokes Equations for Incompressible Fluids.” Archive for Rational Mechanics and Analysis 112, no. 1 (1990): 21–44.

Kolmogorov, Andrey Nikolaevich. “The Local Structure of Turbulence in Incompressible Viscous Fluid for Very Large Reynolds Numbers.” Comptes Rendus (Doklady) de l’Académie des Sciences de l’URSS 30 (1941): 301–305.

Menikoff, Ralph, and Bradley J. Plohr. “The Riemann Problem for Fluid Flow of Real Materials.” Reviews of Modern Physics 61, no. 1 (1989): 75–130.

Moin, Parviz, and Krishnan Mahesh. “Direct Numerical Simulation: A Tool in Turbulence Research.” Annual Review of Fluid Mechanics 30 (1998): 539–578.

“The scientific paper in its orthodox form does embody a totally mistaken conception, even a travesty, of the nature of scientific thought.” – Peter Medawar

A limiter is an intervention that steps into a numerical method, examines the data, and prevents the method from doing something unphysical, bad, or dangerous. The determination is based on focusing on the monotonicity of data to produce “safe” solutions. The first limiters were ad hoc, but they gained rigor over time. This is clearest with the TVD theory by Harten. A limiter acts when the solution the method would otherwise choose is going to cause harm, i.e., oscillations. In that moment it adaptively selects a less accurate method in exchange for a safe, robust solution.

Limiters are a major change in how numerics work. We had always solved nonlinear equations, but with limiters, we suddenly had nonlinear methods, and those methods were powerful. Before limiters, you had a choice between two poor options: accurate solutions that oscillated, or crude ways of imposing a physical principle. One such crude tool is artificial viscosity. Another is the safety of low-order dissipative methods like upwinding. Limiters are tightly tied to the entropy conditions that accompany both. An entropy condition is simply a way to select the physically realizable solution and separate it from the unphysical one. Artificial viscosity can achieve this as well. The power of a limiter is that it applies dissipation only where dissipation is necessary, and “limits” it everywhere else.

I asked five different AI engines, Claude, ChatGPT, Gemini, Llama, and Grok, about the history of limiters. The quality varied, but the answers were remarkably consistent. They were also consistently wrong: each returned the same incorrect, flattened version of reality, repeated five times over. That uniform wrongness is what makes this essay necessary. I have noticed an interesting side effect with AI. When I ask a question and the response is badly wrong, I feel like I should write a blog post about it. The information feels like it should be added to the corpus of knowledge. Right now, the corpus is missing key parts of this topic.

“No scientific discovery is named after its original discoverer.” – Stephen M. Stigler

What really stands out to me is what I should make of the other questions I ask these models. Especially the ones of equal or greater complexity than this one. If I already know the models are unreliable on expert nuance, how unreliable are they on everything else? How fast does reliability fall off as a topic becomes more complex? The decline seems significant. It is a thought, and a question, that frankly has been haunting me of late, and it is a warning that should be heeded broadly across most serious work. It also seems to reflect on the validity of the broader success claimed for “foundation” models.

When I queried these models, what I mostly learned was how wide and how deep the gaps in their knowledge are. They get the basic sketch right. The details, the truth of what actually happened, are missing. All they can hand back is what was processed through the published literature. This is a problem I have come to appreciate over a career: mathematicians, scientists generally, and engineers are terrible historians. The real story is not in the regular literature. Most people never see the details, so the details never get written down.

That is exactly why this is worth doing. I want the history of how things actually happened, as opposed to how they were later cleaned up. The real story has a great deal to teach us. We should know the truth about important discoveries like limiters. In learning how they were really made, we learn how discovery itself works: what the recurring patterns are, and how a rough, half-formed idea turns into state-of-the-art technology. How ideas are exchanged and bred for new ideas. That path is never as clean or as simple as the textbooks make it look. To sanitize it this soon after it happened is a real loss.

What I want to do here is put back the personal touches and the subtle narratives that get sanded off. The formal literature has little tolerance for these essential details, so we are left with a sanitized, formalized history that loses much of its humanity. These stories matter. A great deal has already been lost in the feedback loop between AI and Wikipedia: the depth and genuine understanding that come only from deeper research, and from having lived through some of it. This happens over and over in science, and limiters are another example.

“Everything of importance has been said before by somebody who did not discover it.” – Alfred North Whitehead

Now, back to the story of limiters.

The history of limiters is one of the most important developments in numerical methods. They were built specifically for hyperbolic PDEs, equations that are central to a vast range of applications. The methods we built to solve those equations are a large part of what sets modern computational technology apart. The limiter story is, at its core, the victory of constraints over simplicity. It is also a case of the same idea appearing independently and simultaneously. The conditions were right for inventing them. Both qualities make it worth understanding in depth.

If we step back to 1970, several things are happening at once. The previous twenty-five years had produced a set of competing methods for solving hyperbolic PDEs, and all of them were bad in some way. You had either oscillatory methods with ad hoc stabilization, or dissipative methods with terrible accuracy. Godunov had proved a barrier theorem that seemed to sentence us to one or the other unacceptable option: it stated that non-oscillatory linear methods are necessarily first-order accurate. At the same time, computing power was beginning to grow and to expand beyond the government and defense labs, and computational science was moving from a niche pursuit toward the center of science. As more varied people waded into computational science, ideas flowed. Things that had been accepted needed to be improved.

In this moment, two men set out to overcome the prohibition of Godunov’s theorem. One, Jay Boris, was at the Naval Research Laboratory in the United States. The other, Bram van Leer, was a student in the Netherlands studying astrophysics. At the time they did not know each other, yet they would devise strikingly similar ways of solving the same problem. It is clear that Boris knew of Godunov’s theorem at least in part, and that van Leer knew it as well. Boris understood its limitations. Van Leer did as well and also looked to modernize Godunov’s method.

It is also worth noting what else Boris was working on at the time: the technique that has become known as the Boris fix for Alfvén waves. In magnetics, one recurring difficulty is the appearance of extremely fast Alfvén waves as their speed approaches the speed of light, which severely constrains the time-step size in some MHD codes. Boris devised a method that artificially changes the speed of light to allow more efficient computation. Some of the basic concepts behind that fix are suspiciously similar to what you get with limiters.

In the Netherlands, Bram Van Leer was trying to figure out how to combine the high accuracy of a method like Lax-Wendroff’s with the guaranteed monotonicity of Godunov. I remember being at a conference in 2004 and asking Bram how he discovered limiters. He told me that he could just see that it could be done. He could overlay the solutions from the two methods and see that there was a way to blend them. He just needed to find the recipe. The key was that he knew it could be done. The key was the inspiration. After that it was just a matter of details.

This recipe was what he discovered over the course of a series of papers and has come to be known as limiters. The most common of these are the Van Leer limiters, along with a host of numerical methods for the advection equation and ultimately a higher-order extension of the Godunov method for the Euler equations. His limiters were similar to what Boris created but different in approach. The success of Van Leer’s methodology is attributable to two things: his knowledge, inspiration, and brilliance, and some fortuitous circumstances. I’ll get into this later, but Bram had help in extending his methods to a broader community. He also made better choices for outlets.

The target audience for Van Leer’s work was astrophysics and aerodynamics. Both were essential to the success of the methodology, particularly aerodynamics, where the mathematical theories of Lax and the influence of the Courant Institute were substantial. Astrophysics was important for personal connections and for tying into the national laboratories. Jay Boris worked within the plasma physics community, which was not as fertile a place for expanding his ideas to a broader set of humans. To some extent, this is simply the luck of the draw.

What both men created would become known as limiters, and it produced the Limiter War. They waged pitched battles over credit for years, and the fact is that they both deserve it. They both did the work, and the invention was nearly simultaneous because the conditions were right. Those conditions were the ones laid out at the start of the 1970s: the expansion of computational science and the powerful computing that let it be pursued. What is less well known is that there were two further inventions of limiters, again independent, in nearly the same period.

“In science the credit goes to the man who convinces the world, not to the man to whom the idea first occurs.” – Francis Darwin

The first came from an aerodynamics engineer named Vladimir Kolgan, who worked in the Soviet Union. He developed a numerical scheme that looks very much like what we would today call a second-order ENO scheme. It used a limiter that returns the slope with the smallest magnitude. Kolgan was also producing what we would now call the first high-order Godunov method. He took the scheme Godunov introduced in the very paper that contained the barrier theorem and extended it to second order. The development and popularization of the high-order Godunov method is most often credited to Bram van Leer as one of his chief contributions, yet Kolgan got there first.

“Kolgan succumbed to lung cancer in 1978, at the age of 37; at that time the final papers by Boris’ group and by Van Leer had yet to appear.”– Bram van Leer,

Kolgan’s limiter function is quite similar to the minmod function, which is now used ubiquitously in the design of limiters. Jay Boris invented minmod, and it was so well designed that it has been reused over and over across the field. For the uninitiated, minmod is short for “minimum modulus.” It returns the argument with the smallest magnitude, provided all the arguments share the same sign. If they differ in sign, it returns zero.

The fourth inventor of limiters is Ami Harten, a student of Peter Lax. Harten developed a class of methods known as the artificial compression method. It looks at the magnitude of gradients and removes dissipation from low-order methods such as first-order schemes. The effect is to blend a second-order method with a first-order one, removing the dissipation inherent in the first-order method. Harten is best known for his papers on TVD methods, a mathematically rigorous version of high-order methods. TVD was a formalization of van Leer’s ideas, and it supplied a host of theoretical properties. It rests on four very simple linear test equations, and it carries significant limitations on the achievable order of accuracy. Nonetheless, it provided a mathematical rigor that powered the acceptance of limiters broadly.

“Discovery consists of seeing what everybody has seen and thinking what nobody has thought.” – Albert Szent-Györgyi,

As I argued in the case of the Lax equivalence theorem, the limited but rigorous theory behind TVD methods provides a firm foundation for them. The theory is narrow, but it is exactly that rigor that drives acceptance, development, and the eventual expansion of the methods to a broad range of applications. Peter Lax put much of this into order in his less well-known work on high-resolution methods, where he described how combining accuracy with nonlinear limiters yields what we now call high-resolution methods. That is the most accurate single description of the broad class of methods discussed here.

These applications are strongest in aerodynamics, but they also include a major shift in how the nuclear weapons labs chose to solve their problems. That leads to a couple of closing threads to pull.

In the late 1970s, van Leer was visited by a young scientist from Lawrence Livermore who was on sabbatical. His name was Paul Woodward. A genuine synthesis of ideas followed. Van Leer’s 1979 paper on the MUSCL method was substantially influenced by Woodward’s visit. It included a version of Godunov’s method that used a number of implementations and tricks common in the Lawrence Livermore codes, among them building an Eulerian method through a Lagrangian remap, along with the time-integration ideas Woodward had promoted.

Woodward, in turn, took van Leer’s ideas and worked them into a fine art. In collaboration with Phil Colella, he developed the piecewise parabolic method (PPM), in a sense an extension of van Leer’s Scheme V from his 1979 paper. PPM removed some of the extra variables that earlier scheme required and reconstructed them from finite-volume values. To this day, PPM remains one of the most powerful methods for solving hyperbolic PDEs.

The methods adopted most directly by the weapons labs were the ones van Leer championed, and they took hold most completely in numerical advection and remap algorithms. Van Leer’s high-order methods with limiters for the advection equation revolutionized remap and changed the codes forever. Over the span of a few years every code that did advection or remap adopted Van Leer’s ideas. His methods were so much better that adoption was immediate.

They also had an enormous impact on astrophysics, enabling a phase change in the calculations that could be attempted. This was seen most acutely in accretion disks, which cannot form naturally if you simulate with first-order methods. You need the combination of accuracy with mononticity to compute the phenomena. Once limiters were introduced, you could suddenly compute them. The degree of improvement in capability was quantum in nature.

The explanation is relatively simple. A first-order method effectively adds a linear viscosity, and at almost any reasonable resolution the flow you compute is effectively laminar. You never get the transition to high-Reynolds-number flow, to turbulence, or to other fine structures. With limiters, you remove this laminarization and recover a second-order flow that can behave like a true high-Reynolds-number turbulent flow. This effect is something Len Margolin and I wrote about in one of our more cited papers.

“We present a rationale for the success of nonoscillatory finite volume difference schemes in modelling turbulent flows without need of subgrid scale models … certain truncation terms … have physical justification, representing the modifications to the governing equations that arise when one considers the motion of finite volumes of fluid over finite intervals of time.” – Len G. Margolin and William J. Rider,

The way I explain it is this: you need a second-order solver to get these structures, but you also need to keep that solver well behaved. Limiters are the means to do both, and that is why they produced a phase transition in the kinds of methods we use to solve hyperbolic PDEs. Now today there are a host of limiters and related methods available. For the most part, all of them are capable of doing amazing things compared to classical methods. We await the next revolution in methods. What this might be is a subject of debate. Many feel like the field is fully cooked. Nothing more is needed to improve. To me this is a patently absurd point-of-view.

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.” – Max Planck

References

Boris, Jay P. “A fluid transport algorithm that works(Development of explicit, Eulerian finite-difference algorithm for solving continuity equation).” (1971).

Boris, Jay P. A physically motivated solution of the Alfvén problem. No. NRLMR2167. 1970.

Boris, Jay P., and David L. Book. “Flux-Corrected Transport. I. SHASTA, A Fluid Transport Algorithm That Works.” Journal of Computational Physics 11, no. 1 (1973): 38–69.

Boris, Jay P., and David L. Book. “Flux-Corrected Transport. III. Minimal-Error FCT Algorithms.” Journal of Computational Physics 20, no. 4 (1976): 397–431.

Colella, Phillip, and Paul R. Woodward. “The Piecewise Parabolic Method (PPM) for Gas-Dynamical Simulations.” Journal of Computational Physics 54, no. 1 (1984): 174–201.

Godunov, Sergei K. “A Difference Scheme for Numerical Computation of Discontinuous Solutions of the Equations of Fluid Dynamics.” Matematicheskii Sbornik 47 (89), no. 3 (1959): 271–306.

Harten, Ami. “The Artificial Compression Method for Computation of Shocks and Contact Discontinuities. I. Single Conservation Laws.” Communications on Pure and Applied Mathematics 30, no. 5 (1977): 611–638.

Harten, Ami. “High Resolution Schemes for Hyperbolic Conservation Laws.” Journal of Computational Physics 49, no. 3 (1983): 357–393.

Kolgan, Vladimir P. “Application of the Principle of Minimizing the Derivative to the Construction of Finite-Difference Schemes for Computing Discontinuous Solutions of Gas Dynamics.” Uchenye Zapiski TsAGI 3, no. 6 (1972): 68–77. Translated and reprinted in Journal of Computational Physics 230, no. 7 (2011): 2384–2390.

Lax, Peter D. “Weak Solutions of Nonlinear Hyperbolic Equations and Their Numerical Computation.” Communications on Pure and Applied Mathematics 7, no. 1 (1954): 159–193.

Lax, Peter D., and Burton Wendroff. “Systems of Conservation Laws.” Communications on Pure and Applied Mathematics 13, no. 2 (1960): 217–237.

Lax, Peter D. “Accuracy and resolution in the computation of solutions of linear and nonlinear equations.” In Recent advances in numerical analysis, pp. 107-117. Academic Press, 1978.

Margolin, Len G., and William J. Rider. “A Rationale for Implicit Turbulence Modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821–841.

Van Leer, Bram. “Towards the Ultimate Conservative Difference Scheme. I. The Quest of Monotonicity.” In Proceedings of the Third International Conference on Numerical Methods in Fluid Mechanics, edited by Henri Cabannes and Roger Temam, 163–168. Lecture Notes in Physics 18. Berlin: Springer-Verlag, 1973.

Van Leer, Bram. “Towards the Ultimate Conservative Difference Scheme. II. Monotonicity and Conservation Combined in a Second-Order Scheme.” Journal of Computational Physics 14, no. 4 (1974): 361–370.

Van Leer, Bram. “Towards the Ultimate Conservative Difference Scheme. IV. A New Approach to Numerical Convection.” Journal of Computational Physics 23, no. 3 (1977): 276–299.

Van Leer, Bram. “Towards the Ultimate Conservative Difference Scheme. V. A Second-Order Sequel to Godunov’s Method.” Journal of Computational Physics 32, no. 1 (1979): 101–136.

Van Leer, Bram. “Upwind and High-Resolution Methods for Compressible Flow: From Donor Cell to Residual-Distribution Schemes.” Communications in Computational Physics 1, no. 2 (2006): 192–206.

Van Leer, Bram. “A Historical Oversight: Vladimir P. Kolgan and His High-Resolution Scheme.” Journal of Computational Physics 230, no. 7 (2011): 2378–2383.

Woodward, Paul, and Phillip Colella. “The Numerical Simulation of Two-Dimensional Fluid Flow with Strong Shocks.” Journal of Computational Physics 54, no. 1 (1984): 115–173.

Woodward, Paul R. “Trade-offs in designing explicit hydrodynamical schemes for vector computers.” In Parallel computations, pp. 153-171. Academic Press, 1982.

Woodward, Paul R. “Piecewise-parabolic methods for astrophysical fluid dynamics.” In Astrophysical Radiation Hydrodynamics, pp. 245-326. Dordrecht: Springer Netherlands, 1986.

“For his groundbreaking contributions to the theory and application of partial differential equations and to the computation of their solutions.” — Citation for the 2005 Abel Prize awarded to Peter D. Lax.

This post focuses on one of Peter Lax’s pivotal contributions. Lax was one of the greatest mathematicians of the 20th century, and his work shaped many fields. None, more deeply than computational science. Winner of the Abel Prize in 2005. He died about a year ago at 99.

His career was shaped by the defining event of that century. He emigrated from Hungary with his family in late 1941. He was drafted into the U.S. Army at 18, and was assigned in 1945–46 to the Manhattan Project at Los Alamos. He was a brilliant high school graduate, but known to top scientists already. He returned to Los Alamos with a PhD as a staff member in 1949-50 and spent most summers there through the 1960s. There he witnessed the beginnings of computational science, and its power. Then he contributed to it mightily.

For my interests, Lax’s contributions sit primarily in hyperbolic conservation laws, where he developed much of the essential theory. These underpin both the mathematical and the numerical solution of these equations. My focus here is the equivalence theorem (Lax and Richtmyer 1956). It is sometimes called the Fundamental Theorem of Numerical Analysis (by Gil Strang). It states that for a well-posed linear initial-value problem, a consistent finite-difference method is convergent if and only if it is stable. The theorem applies rigorously to linear PDEs. This linearity restriction has long been used as the excuse for not placing it in a more central role in the practice and justification of code verification. I believe that excuse is wrong-headed and short-sighted, and I will make the case in what follows.

I write this with two things in mind. The first is the beauty and importance of mathematical foundations. This has utility in giving us confidence in what we do as computational scientists. The second is the recognition that the physical laws to which we apply mathematics are themselves only approximate too. Mathematics is the way we drag the physical world into order so that it can be understood and, with luck, mastered. We keep in mind that mathematical descriptions always fall short of the real thing. This falling-short does not diminish their importance. They are precisely what we lash ourselves to: the proverbial mast in the storm. Diminishing the power of this theorem only lessons our ability to withstand the waves.

“It was the experience of being part of a scientific team — not just of mathematicians, but people with different outlooks — with the aim being not a theorem, but a product. One cannot learn that from books, one must be a participant… It was there — that was in the 1950s — that I became imbued with the utter importance of computing for science and mathematics.” — Peter D. Lax

The Case for the Theorem

I believe strongly that Lax equivalence theorem is the foundation of verification. I will make the case for why. I will start by why the standard objection to it misses the point.

If one consults the canonical V&V references, Roache (1998), or Oberkampf and Roy (2010) you find a curiously dismissive attitude toward the theorem. The reason is always the same: strictly and rigorously, it applies only to linear partial differential equations. We all know that almost everything interesting in science is governed by nonlinear equations. In both books the theorem is mentioned exactly once, and in both it is dismissed almost as fast as it is raised.

I think that is short-sighted.

The theorem captures is the essence of what we ask of any computation: more computational effort will produce a more accuracy. It ties together the two properties that make this possible. These are consistency and numerical stability. It states that, for a well-posed linear problem, the two together are equivalent to convergence. That is the whole game. We design consistency and stability into a method. This carries the expectation of convergence. Verification is the discipline of checking whether the method actually delivers what the theory promises.

“Physics is like sex: sure, it may give some practical results, but that’s not why we do it.”― Richard P. Feynman

Yes, the rigorous statement is restricted to linear equations. But the objection loses the forest for the trees. Most of the nonlinear equations we actually care about are contractive. A contractive equation is close to a linear one in terms of what we expect from it. This is what is witnessed in our daily work. The genuine danger the linear caveat points at is real. Nonlinear equations can produce ill-behaved structures and solutions that do not converge the way linear ones do. Even linear equations do as witnessed by the fractional convergence of linear advection. That danger is the exception we should watch for, not the rule that should make us throw the framework away. The theorem tells us what to do. It defines the practice we must embrace.

The forest is this: Lax’s theorem expresses a fundamental, almost axiomatic belief about computing. It encapsulates precisely why verification is worth doing at all. The theorem is useful ammunition in a practice that is resisted commonly.

“The mathematical formulation of the physicist’s often crude experience leads in an uncanny number of cases to an amazingly accurate description of a large class of phenomena.” — Eugene P. Wigner

A Practitioner’s View

“There are many hypotheses in science that are wrong. That’s perfectly alright; it’s the aperture to finding out what’s right. Science is a self-correcting process. To be accepted, new ideas must survive the most rigorous standards of evidence and scrutiny.” ― Carl Sagan

Let me put it personally. Over a career spent designing methods, the two conditions I come back to every time are consistency, and stability. Consistency is where accuracy lives and gets defined. Stability is essential and must always be checked at design. The accuracy or stability of a technique matters more than the technique itself. I check the code, I look at how it converges on simplie problems. Then I go to more real problems. With observed convergence I confirm that the overall recipe I designed is a good. Evidence that I actually implemented what I intended to design. I have done this many times, and every single time, the result still lines up with the conditions of the equivalence theorem. It is disturbing to me that the community does not more widely recognize this simple pair of conditions as the foundational contribution it is. It is the cornerstone of the practice of the computational scientist.

There is more foundation to build here, not less. Extending it to specific classes of nonlinear equations would be valuable work. The Lax–Wendroff theorem already shows part of the way. For a consistent, conservative scheme, a convergent solution is guaranteed. A weak solution of the conservation law if you have conservation form. This has a technicality that you still need an entropy condition to pick out the physically correct weak solution.

So does the equivalence theorem hold for nonlinear equations all the time? No. Does it hold most of the time? Empirically, yes. There is the heart of my complaint: why reject something that holds most of the time? We should of course be careful, precisely because we cannot claim it always holds. Rejecting it outright is short-sighted. This is especially true given the power of the underlying concept. It has driven the growth of computational power in the decades since the Second World War. Its precepts hold for the vast majority of important work. When it does not hold, we have a crisis.

I would rather be tethered to something that does not rigorously apply than to nothing at all. This theorem supplies most of the practice we rely on during verification. That alone is reason to embrace it. Pure mathematics may not regard it as ironclad for the nonlinear problems we actually run. That is an important caveat to make clear. Still it contains very nearly the entirety of our well-founded beliefs about what these computations are doing, and it holds up empirically. That is exactly what makes it the right thing to stand on.

“I heartily recommend that all young mathematicians try their skill in some branch of applied mathematics. It is a gold mine of deep problems whose solutions await conceptual as well as technical breakthroughs.” — Peter D. Lax

References

Lax, P. D., and R. D. Richtmyer (1956). “Survey of the Stability of Linear Finite Difference Equations.” Communications on Pure and Applied Mathematics 9 (2): 267–293. DOI: 10.1002/cpa.3160090206.

Oberkampf, W. L., and C. J. Roy (2010). Verification and Validation in Scientific Computing. Cambridge University Press, Cambridge. ISBN 978-0-521-11360-1.

Roache, P. J. (1998). Verification and Validation in Computational Science and Engineering. Hermosa Publishers, Albuquerque, NM. ISBN 978-0-913478-08-0.

I’ve taken a brief hiatus from posting, and I wanted to share why.

About three weeks ago, my father passed away after a long and extremely painful illness. It caused him a lot of pain, and it was also very hard on his loved ones. Last week, my brother visited me, and we spent a lot of time getting my dad’s affairs in order. We lost my mom six years ago, so we had to handle, for the final time, the remains of my parents, along with a lot of other business.

We have possessions to sort through, a few things to keep, an estate sale, and ultimately the sale of the family home. All of this has been going on, and as you might expect, it has gotten in the way of writing.

I’ve drafted a couple of other posts, and I’ll start working on them in the next couple of days. I hope to be back to my normal pace of writing things that are hopefully interesting and thought-provoking for all of you.

“AI won’t replace humans, but those who use AI will replace those who don’t.” – Garry Kasparov

One of the most important questions that needs to be answered today is: How do you use AI? How do you use it properly and effectively to do your work, help run your life, and ultimately make things better? It has an amazing capacity to do this, but only if it’s used well. AI is also potentially completely and utterly destructive. It is destructive if it merely replaces the thinking person with unthinking AI slop. The direction today is moving towards destructive, and change is needed for it to be good for society.

As I wrapped up my professional life, AI suddenly landed on my radar in 2022. I immediately saw it as something huge and amazing, a moment that had happened a few times before. It felt bigger than either the advent of the Google search algorithm or the smartphone. AI is a massively transformative technology. We need to tackle the task of making the transformation positive.

“Abundance … is the state in which there is enough of what we need to create lives better than what we have had.” –Ezra Klein and Derek Thompson

My concern right now is that the oligarchs and their endless greed and appetite for money will just look for more. Look at how social media played out. In addition, we have government incompetence and plain mental laziness. Corporate interests are simply poisoning our national “strategy.” Mostly, we don’t have a strategy other than buying a shitload of computers (data centers). Alongside this is destroying scientific research. The result is we are going to completely botch the rollout of AI and fail to take advantage of what it can truly do.

The bottom line is that dealing with AI properly is twofold:

1. There’s a technique and a mindset one needs to adopt. This mindset is verification and validation of AI’s work.
2. The goals for our organizations and institutions need to adapt to get the most out of them. AI can allow people to work more and better, not simply eliminate people.

This is a mentality of abundance, not scarcity. Right now, both of those things are decidedly not in place. The correct mindset for engaging with AI is the scientific method. In practice, that means using verification and validation. These are mindsets and techniques that help unleash AI safely and productively.

The core mindset is that AI should augment people, not replace them. It should help each person do more and better work, whether at work or at home. A foundational principle is to improve human life. Clearly, this idict is absent today as a backlash is growing day by day. Right now, no one (corporately or institutionally) is actively engaged in this mindset. If we do not change, AI could ultimately doom itself as a technology. The United States will lose its edge in the battle for dominance through the backlash. To succeed with AI, we need to adopt an abundance mindset rather than a scarcity mindset.

“The confidence people have in their beliefs is not a measure of the quality of evidence but of the coherence of the story.” – Daniel Kahneman

I’ll start with a non-technical example. My wife and I recently bought a tin raven sculpture for our front yard. We love ravens in the mountains here in New Mexico and wanted to display this. I found the perfect pedestal: a rock in our front yard where the raven could stand and be visible from the driveway, the street, and my kitchen window. The problem was that the raven kept blowing over in the wind. So I asked ChatGPT, “How can I secure the bird to the rock so this doesn’t happen all the time?” It suggested buying clear epoxy. It seemed a reasonable logical solution. To validate that approach, we checked at the store where we normally shop and found a product that matched the recommendation.

The validation was straightforward: the product was available. Moreover, the solution seemed like it would work both short-term and long-term. So far, it has been a great solution we hadn’t thought of initially. We shall see how it weathers through our seasons and persists.

This example illustrates a basic methodology in simple form. It also shows the danger of AI. The danger is that AI becomes like social media: it sells you a specific product (a brand), lets you click to buy it on Amazon, and has it delivered. One could easily see that happening, and it would start a downward spiral for AI as a tool to transform society. It would be monetized just like social media and become an engine of greed. It would sell us crap and rapidly become enshitified like all those companies.

I’ve argued that V&V is the scientific method. I think the terms in V&V are especially useful for constructively engaging AI. The first piece is verification. It can apply to confidence that a tool is theoretically correct. For modeling and simulation, the definition is straightforward. For AI, the definition is slightly different: it’s about whether the tool can provide basic information reliably and correctly.

“Progress is more about implementation than it is about invention.” –Ezra Klein and Derek Thompson

When I start querying a large language model on a topic, I always begin by asking axiomatic questions to determine whether the basic information is present in the LLM’s responses. More importantly, are there gaps or mistakes in that knowledge that need to be accounted for before I try to probe into the unknown? This is a verification exercise and a way for me to gain confidence. Conversely, I might find that the LLM is faulty. I can proceed through other avenues. Through verification, I can find if the LLM is well-suited for the pursuit of the question that I am thinking about. This is the initial step.

The real work comes in validating the LLM’s responses to deeper, more unknown questions. A key is to approach this with a healthy dose of doubt and take everything the LLM produces with a grain of salt. Addressing and calming these doubts is V&V thinking. One way to validate the results is to research the LLM’s responses and check whether they are factually correct. Another approach is to test the results in the field and see whether they hold up. We did this with the tin raven.

If you are writing code or running a literature search for your work, you should also validate. See that the references the LLM finds are actually real. Once you have validated the results, you can use them with confidence. You get an acceleration of your work, but you still have to do the legwork to confirm whether it is correct. One can do far more validation by using multiple LLMs to flag variations in response.

There are a few ways to query an LLM to help you assess the reliability of its responses:

1. Ask the LLM to provide references and links so you can track where the information came from and evaluate whether it is reliable.
2. Ask the LLM to provide multiple responses to the same question. Make the LLM score each result to get a sense of its relative confidence. This can help you probe the broader uncertainty in the results. For higher-stakes questions, draw more samples and pay attention to where the score drops off. You can see if LLM is providing responses it believes are less likely. Validation would examine the veracity of its assessment.
3. This uncertainty needs to be probed and validated. I have seen cases where the lower-probability response was actually the better one. This rightly calls its results into question.

This gets at the power of AI, which is very good at breadth and at incorporating a broad spectrum of views. That breadth is also the danger, since there is no truth embedded. One needs to be mindful of the breadth it is producing. It is dangerous. Some of the responses LLMs provide are garbage (hallucinations or bullshit).

Humans provide depth. Humans provide thought and checking. The combination is powerful. If you take the breadth and consider it carefully, the LLM results can broaden your perspectives. This can, in turn, encourage deeper thought. That deep thought needs to be encouraged across the board.

This gets to the worry I would have when the leadership at work engages with using AI. When I was working, the push to use AI was superficial and clumsy. I’ve heard from friends at several institutions that the leadership’s approach to AI has been mindless cheerleading. For example, getting people to use it without any sense of responsibility or technique. For example, they might say, “Let AI write your performance review or performance plan.” No effort was put into showing how to engage with it properly. The entire engagement lacked any skill or depth.

“We have a startling abundance of the goods that fill a house and a shortage of what’s needed to build a good life.”–Ezra Klein and Derek Thompson

This kind of mindless work calls into question the performance review itself. It is not befitting leadership and malpractice. Leadership should encourage people to think and do the hard work of using AI properly. Use AI to augment their work (not replace it). The goal should be to make work better, deeper, and of higher quality. Ultimately, the aim is to use it as a tool to encourage broader, more open-minded thinking. That thinking needs to be applied in a verified and validated way so it can be used for things that matter. Given the mission of these institutions, any other approach is reprehensible.

Let’s get to the real enemy of success and AI: a short-term view of what constitutes success. For corporations, that means money. For institutions, it means money too, and increasingly so. Success means taking the long-term view of success. Using AI properly is a long journey. It requires deep engagement with the development of detailed processes analogous to V&V used in modeling and simulation. We all need to put in the work.

“The ability to discipline yourself to delay gratification in the short term in order to enjoy greater rewards in the long term, is the indispensable prerequisite for success.” — Brian Tracy

If you adopt a scarcity mindset, you use AI to replace workers and reduce the workforce size. You invite the backlash we are starting to see. In the short term, this works like a charm. We have institutions and corporations that show no fealty to the nation as a whole and will eagerly make decisions that are negative for society. We’ve seen this with social media, and if we see it with AI, the damage will be profound. The opportunity cost is even higher. AI could do miraculous things. That opportunity is being squandered.

This dynamic between humans and AI plays out most profoundly in education. If we adopt a scarcity mindset and fail to adapt, AI could completely upend our current educational system. We can choose to make it better or worse. Right now, we are moving towards worse by resisting this technology. Our educational system already needs an overhaul. Do we grab the opportunity to improve it, and train the next generation with this technology? It could be a catalyst for a positive transformation.

My attitude is that we should assume every student is using AI and design a system that is impervious to cheating. That assumption is the key. The essence is teaching students how to use AI properly. It is what I described above. They need to have the necessary techniques, knowledge, and drive to augment and accelerate learning. These lessons can be meaningful with or without AI. The hard part is putting some burden on the teachers. Education cannot be static or underestimate this technology. The value of a liberal arts education will suddenly skyrocket. The skills and knowledge are going to be in greater demand. True human individuality and authenticity are also needed to stand out with AI. AI flattens and makes work anonymous. In the future, the touch of inspiration from an individual will be a hunger.

“The first lesson of economics is scarcity: There is never enough of anything to satisfy all those who want it. The first lesson of politics is to disregard the first lesson of economics.” – Thomas Sowell

My stance is this: if you submit work that is essentially regurgitated AI “slop,” you get no credit. That is a zero, a foundation. If you produce something wonderful and good without AI, more power to you. The question is, if you are using AI, are you producing a better product? Have you done the due diligence and the work required to make something better than you could produce on your own? We should adopt disclosure about AI use across the board. We should always talk about how AI was used in the production of the work, and we should do this at work and at school, every time.

For example, I use Claude to edit and do a comprehension and grammar pass on my writing. I also use Claude to generate potential social media posts for each blog post. I do this as a matter of course as part of my education on LLMs. I make sure the writing itself is all from me.

Ultimately, this attitude should apply in education, in work, and in life. Placing the scientific method and verification/validation at the center of how our new world works is long overdue. The quest to use AI productively in society can power this shift. It would change AI from a force of unbridled destruction into one of creation and quality.

The abundance mindset, which says we keep all the people but make them produce more over the long run, is the path to wealth and success for society as a whole. It requires patience, investment, and a strategy that our current leadership seems incapable of producing. Ultimately, this is our greatest battle: getting our leadership to choose a long-term path to success over short-term profit-taking. The signs today do not look good, since short-term profit and wealth are dominating everything, whether public or private.

“We had better be quite sure that the purpose put into the machine is the purpose which we really desire.” – Norbert Wiener

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” — Richard Feynman,

Yes, it is entirely optional. The question is whether it should be.

My view is that it should not be optional; conserving should be foundational. Optionality is a real problem across computational physics. There is a tolerance for this practice that reflects deeper cultural issues and the history of various technical fields. Like many things, it is an accepted practice that should be unacceptable.

I started exploring this question by poking around with AI. This is the way things are done these days. One can study a topic with a large language model. Since I’m currently paying for Claude at a modest level, this was the first choice. I asked it first, and like most large language models, the initial response included a lot of ass-kissing and dick-sucking that I didn’t ask for. It does that almost reflexively. When you get answers like that, push back and reject them.

I should note that Claude has been editing and digesting most of what I write lately, so it knows my views very well. This is not what I wanted. I wanted an honest broker and a scan of genuinely different ideas. I had to call Claude out to get those.

As a best practice, I also asked the same question of ChatGPT and Gemini. I got much the same answers, and as you’d expect with the free versions of ChatGPT and Gemini, they were a bit disappointing. There were some common threads worth hanging onto and considering.

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman,

One of my questions was: which conservation law is the most fundamental and primal?

“The law that entropy always increases holds, I think, the supreme position among the laws of Nature… if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.” — Arthur Eddington

I got a lot of navel-gazing nonsense that tried to sound thoughtful, but was not remotely useful. One answer was that energy is the most fundamental, mainly because of how energy and mass interact in special relativity. For relativistic flows, that is the correct answer. For most flows, this was idiocy. The other common response cited entropy as a conservation law, which it is not. It should not be listed as one of them. Entropy is not conserved. It is, given a sign convention, a quantity that matches an inequality. The responses failed to talk about the nature of that inequality dynamically, which is incredibly important. For large-scale flows, the entropy observes some well-known asymptotic limits.

I also asked about the numerical aspects of conservation. The responses highlighted the importance of the Lax-Wendroff theorem and its implications. That was a high point, especially for Claude, since it has digested all my writing. The discussion also mentioned Godunov’s theorem, which is important but completely unrelated to this particular question. Lax-Wendroff states that conservation form is needed numerically to compute weak solutions to these equations. These weak solutions are appropriate for singular (shocked) flows. Weak solutions are also not unique. To get the unique and correct physical solutions one needs an entropy condition. This is a solution that is properly limiting to solutions containing vanishing dissipation.

From the LLM questions, my conclusion is that conservation is important across the board. All the equations are esential. I would identify mass as the most fundamental conservation equation, since most other equations follow from it. This perspective starts with mass as the foundation for the other conservation equations. It is included in momentum, energy, and charge conservation. It is not in magnetic conservation or the solenoidal condition.

“The opposite of a correct statement is a false statement. But the opposite of a profound truth may well be another profound truth.” — Niels Bohr

For fluid equations, the mass equation is the first moment when you derive the conservation laws from first principles using the Boltzmann equation. I will stand by this premise. I should note, however, that viewing energy as primary is an indictment of the labs. By and large, do almost every calculation without conserving energy. There are exceptions to this statement, but non-energy conserving is well-accepted. In fact, more accepted than energy conserving methods.

Let’s get to the point: I believe conservation should not be optional.

I’ve written about the obsession with preserving adiabatic conditions, and how that leads to a use of non-conservative methods. The non-conservation comes from the choice of an internal energy equation. It is an evolution equation, not a conservation law. Actually, The focus is the energy equation causing the issues. I do think energy should be conserved as a constraint, ideally by construction. The preservation of adiabatic conditions should be what you compromise on, and build into these schemes. Right now, the opposite happens: conservation is the thing that gets sacrificed.

“In mathematics you don’t understand things. You just get used to them.” — John von Neumann

My reasons are straightforward. In the labs they care about flows that are highly energetic. Those flows have weak solutions, and if you’re interested in weak solutions to these equations, the Lax-Wendroff theorem applies. It applies to finite volume schemes, finite element schemes, and every scheme you can imagine. It is not limited to one specific method. It applies to a class of equations that can be solved. The theory is simply ignored by them. For people working in solid mechanics, the same principle applies. They are also bound by conservation laws and the Lax-Wendroff theorem.

The “element death” or “element deletion” approach to violating mass conservation is one of the most appalling things I can imagine. There is a key difference between this method and the problems with the energy equation. The discarding of mass is not based on a differential equation. It is done without an equation. Thus, it destroys the entire legitimacy of the solution. It is physically inconsistent. It is as if a Star Trek transporter beamed the mass out and put it on the Enterprise. The method is complete bullshit and simply incompatible with science.

I dealt with this in the mechanics culture at Sandia, and it is something they are committed to as an act of intellectual hubris and laziness that is completely objectionable and indefensible. The fact that many extremely important codes rely on this should not be tolerated. More than tolerated, it is even promoted as the right thing to do. In the end, it produced only a complete lack of respect for this community. It is a method grounded in complete ignorance. It is witchcraft and wizardry, and not something that anyone should depend upon.

“Every act of conscious learning requires the willingness to suffer an injury to one’s self-esteem.” — Thomas Szasz

All of these communities would be well served by treating conservation as a fundamental principle. The systems they study conserve as nature does. By the same token, entropy should not be conserved. Instead, it should be treated as an inequality. Entropy should be conserved locally only when the conditions exactly match those that would produce this. Otherwise, the inequality should be applied. In important conditions of shock waves and classical turbulence the rate of entropy production is well known. This is tied to the large scale structure of the flow. This is described by the jump conditions in a shock. It turbulence it is the large scale variation in the longitudinal velocity.

This inability to adhere to a bulletproof physical constraint and concept is a threat to progress. The cultural factors that lead to loyalty to these poor practices are the root of the issue. The methods we use to solve key problems are far less capable as a result, and we should not tolerate that. Our scientists have a deep responsibility to responsibly solve problems of huge national interest. The failure to apply the principle of conservation is an attack on the legitimacy of these equations. This threat is different depending on the root of the violation. The mass violations are far worse because they are not differential. We should know better, we should do better, and we should demand that conservation be treated as an ironclad law. The other principles as processes we try to optimize under the constraint of conservation. Today’s methods do the opposite.

“When life itself seems lunatic, who knows where madness lies? … and maddest of all, to see life as it is and not as it should be.” — From Man of La Mancha, – Dale Wasserman

We should demand better, but after what I’ve seen, we should expect less. The labs are in a race to the bottom, and I wouldn’t expect anyone to do anything bold or good in this environment. I’ll keep up my pointless assault on windmills.

“Hope is not the conviction that something will turn out well, but the certainty that something makes sense, regardless of how it turns out.” — Václav Havel

tl;dr

A key element in personal and professional success is the culture of the place you work, down to a very local level. What I discovered is that these local cultures are essential to how places operate. They are founded on practices that appear to work given the constraints of the job. They are shaped by the legends of people who have excelled there in the past. Together, these comprise a local culture that your work had better resonate with. If you don’t, you will find resistance and ultimately difficulty in achieving professionally. You either adopt the culture by adapting your beliefs to it, or you fight it and lose.

“Culture eats strategy for breakfast.” — Peter Drucker

Every Culture Holds Experience as Proof with Practices and Legends

As I described in my last post, when I took the job at Sandia in 2007, I had a highly refined set of skills. I had refined the craft of producing robust, powerful numerical methods for a wide class of problems of interest to the labs. The people who hired me at Sandia saw this as well. The thing that neither of us understood well enough was how difficult the craft I possessed would be for the culture I was going into. The culture and computation at Sandia have several strongly idiosyncratic aspects:

• It is fundamentally a computational mechanics community, not a computational physics community.
• Massively parallel computing was an epic achievement at Sandia, and the momentum from that achievement still holds sway today in the minds of those who lead it.
• The combination of computational mechanics and massively parallel computing is a legendary success. Everything else at Sandia is in the shadow of those.

“Men do not change, they unmask themselves.” — Madame de Staël

Coming from Los Alamos, I thought of things more in terms of computational physics. I was also well versed in modern computational fluid dynamics and generally believed in what came with it: high-resolution schemes, Riemann solvers, and the absolute necessity of conservation form. Virtually none of this was accepted or even deemed important at Sandia. This led to an almost immediate culture clash between my firmly held beliefs and the beliefs of those I was working with. This was true even when working on the ALEGRA code, which was an outlier within the core computational mechanics community at Sandia.

Take, for example, the concept of mass conservation. In my mind, conservation of mass is a sacrosanct physical law. At Sandia, in the computational mechanics community, conservation of mass is merely a suggestion. There is a willingness to sacrifice it in the name of robustness and convenience. The willingness to sacrifice mass, when it became inconvenient, almost immediately came to a head when I arrived at Sandia. The people would do it blithely, almost without a thought. I can contrast this with Los Alamos, where such a practice ended a storied and very expensive code project virtually overnight.

“It is impossible for a man to learn what he thinks he already knows.” — Epictetus

The episode at Los Alamos is interesting because the adherence to conservation of mass, and the consequences of not doing so came from an avenue at the lab that was hardly the most principled in terms of how computations were pursued. I’ll get into that a little bit later.

At Sandia, the computational mechanics community will remove elements from problems as soon as they become distorted. The elements are viewed as corrupted and completely unable to be fixed or retained. They’re just deleted. To me, this looked like a practice that was tailor-made to destroy the legitimacy of any calculation that was done.

“An expert is a person who has made all the mistakes that can be made in a very narrow field.” — Niels Bohr

Much to my revulsion, this practice was copied in the Eulerian hydrodynamics arena with the code CTH. There, the discard feature was used to get rid of problems, dominantly with the equation of state. This was whenever a material entered some modestly unphysical state that would cause issues with things like sound speeds, creating very small time steps. ALEGRA had a similar feature. They called it Cell Doctor, a way of doctoring the code in the same fashion as discard. The procedure was simple and pervasively used, without seeming understanding of the negative consequences of doing so.

“The graveyards are full of indispensable men.” — Charles de Gaulle

My reaction to this practice was immediate and strong. I found the entire idea completely reprehensible and a violation of principles so important that there had to be a different solution. I started by making fun of Cell Doctor, giving it other names like Cell Undertaker. Nothing could change the fact that I was pushing against a practice that was widely accepted and, to some extent, celebrated within the community I was now working in. Let me be perfectly clear: there is nothing that has ever convinced me that discarding the conservation of mass is a good idea.

I look back now, still feeling the same way, and realize that what I was chafing against was an extreme cultural norm that the computational mechanics community had accepted as almost second nature. My resistance to it made me an outsider and a heretic. I was so sure of the correctness of my perspective that I didn’t step back to examine the nature of this disagreement and its sources. I was rejecting a technique that was storied and accepted within the Sandia community, one that had been championed by several key people who were heroes and legends of the past. Therefore, my repudiation of it was also a repudiation of those legends.

“It is no measure of health to be well adjusted to a profoundly sick society.” — Jiddu Krishnamurti

In retrospect, I see my error. Not that I was wrong technically, but that I was wrong culturally. I was choosing to battle something that had been accepted by this community, and it was also something I could not replace with other means. I had a general set of principles and practices that pushed against all of these ideas. In the end, I should have simply withdrawn, because it was a fight I was never going to win. My retirement was my final surrender, see the light of the futility of fighting culture no matter how wrong it is.

To some extent, I witnessed similar trends at Los Alamos. For example, code developers in the Los Alamos ecosystem were treated as second-class citizens. I have described before how this came to pass. Being a code developer in the 1950s was seen as a way to take a break from the front-line grind of developing nuclear weapons, especially with Pacific testing. This mentality persisted through the 1960s, 1970s, and 1980s. Code developers were always considered less than the people who designed and analyzed nuclear weapons. This lack of professional standing damned them to second-rank status. It damned the resources given to code developmetn too. It really damned the codes they developed.

There was also ongoing tension between the Theoretical Division and the Applied Theoretical Division, where the weapons work predominantly took place. Those wars, and the attitudes that were imprinted in the culture. This meant that very few Los Alamos codes were ever used to do the work in the Applied Theoretical Division at Los Alamos. Almost all the code work was done using codes developed at Lawrence Livermore National Laboratory, and even at AWE in England. Conversely, it’s no small statement to say that code developers in computational physics had a much higher standing at Lawrence Livermore. The difference is huge. Livermore developers are some of their most storied employees (e.g., George Zimmerman). This alone accounts for much of that laboratory’s success in developing codes. Many of these codes were ultimately used by Los Alamos in their work in place of homegrown codes.

“It is the mark of an educated mind to be able to entertain a thought without accepting it.” — Aristotle

I had already waded into some pretty deep waters in terms of the culture. My expertise in modern methods (which came more from CFD) chafed against the methods used by Los Alamos and Livermore. The methods most in use at those labs were based on the work of John von Neumann and Robert Richtmyer. The codes developed in two and three dimensions were ultimately derivatives of that basic methodology. They had been used for decades and their basic use was axiomatic.

The codes at Los Alamos and Livermore would never sacrifice mass conservation under any circumstances. It was viewed as sacrosanct. This gave rise to the use of various remap and remesh methods. These allowed the Lagrangian approach they took to be relaxed, but solving many problems as they became more complex. Ultimately, instabilities in mixing took over, rendering a Lagrangian calculation impossible. The Labs used methods that allowed one to slowly back away from Lagrangian.

“Science advances one funeral at a time.” — Max Planck

My own work was on modern Eulerian codes, which completely sidestep this problem. They are also quite unpopular at the labs. They had great success elsewhere, primarily in the aerospace community. They had also achieved greatly in astrophysics, which is similar to work at the Labs. None of this mattered at all. These methods were counter-culture heresy. One of the key differences is energy conservation. Modern Eulerian codes conserve total energy. The Lab codes do not and they have reasons for this preference. High contrast adiabatic compression is the reason.

Outside the labs, these codes have matured greatly and have a great deal of energy and utility. The intrusion of this technology into the labs’ work has been slow and fraught with problems. The main issue is that these methodologies are quite different and come in as a counterculture example. The culture fights and resists it because it’s foreign, not because it’s not a good idea. It simply wasn’t used by the heroes of the past, and it is something external that they don’t trust. It matters little that the methods had their foundational origin at Los Alamos (e.g., Peter Lax). These methods had not pulled their weight in solving nuclear weapons problems.

“All truth passes through three stages. First, it is ridiculed. Second, it is violently opposed. Third, it is accepted as being self-evident.” — Arthur Schopenhauer

tl;dr

I built a craft over four decades. I found deep knowledge of the literature, numerical methods, tools, public speaking, and the discipline of writing. Most of it went unused in the second half of my career. Los Alamos in the 1990s was generous, curious, and open enough to hone that craft. Senior staff would give you their expertise if you showed up with intellect and judgment. Sandia was an engineering culture that valued maintaining the status quo over advancing science. The edge I had developed was treated as a liability rather than an asset. My craft stagnated. Much of my writing now is an attempt to understand what happened and why.

“I had to learn quickly, for the work was hard and the demands real, but no one could have asked for a better apprenticeship.” — Bertrand Russell

My Hard-Won Craft, Its Use, and Its Ultimate Lack of Utility

The things you know how to do make a huge difference in work. They define what you can technically achieve and provide to where you work. Success, it turns out, depends on things like personality, culture, and interpersonal skills. Usually, one thinks about education as happening at school. I think this is short-sighted and too narrow. I went to a second- or third-rate university. The skills I left school with were modest and ordinary. These basic skills did provide me with a foundation that was ready for something extraordinary.

“Life can only be understood backwards; but it must be lived forwards.” — Søren Kierkegaard

I would say that I have extensive knowledge of the literature in my field. I read widely and absorbed an immense amount of knowledge. In addition, I have learned much about the history of my field, and this is hard-won because it is hidden. Scientists are not terribly good historians.

At school, I did a Master’s thesis. It was pretty much shit and mostly a waste of time. I did learn a bunch of things not to do. I learned who not to work with, and what attitudes and relationships at work are unacceptable. In a deep sense, I found the same thing at the end of my career at Sandia. What was unacceptable when I was 24 was even worse at 62. When it happened again, I retired and left. Ironically, I did the same thing at school. I lucked out and got a job at the best possible place. So many other jobs would have been horrible, and my craft would have been frozen in place.

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

I had the great luck of working at Los Alamos in a period when you could learn a great deal. The staff at Los Alamos were generous and full of curiosity and love of science. If you showed a decent intellect and judgment, you could tap into their expertise. This is exactly what I did. If I look at the structure of this, I see a drama happening in three acts at Los Alamos.

1. My initial job provided modest education, but support for education and my PhD. This was mostly individual self-study. The main thing the job did was connect me to the rest of the Lab and set the stage for the next stage.
2. I worked on a real research project and started to connect with the rest of the Lab. There was a collaboration with Doug Kothe. Also, what effectively became a Habilitation-style thesis with that Project. The project also connected me to high-level researchers like Phil Colella and John Bell.
3. I moved to the Applied Theoretical Physics Division (X) and started to apply my skills to the Mission. There was a well-funded program (ASC) with lots of energy. I also executed a very successful research project on turbulence. The mission work motivated me to do a bunch of really hard things (like turbulence). This really matured me. I wrote a couple of books and a shitload (or is that a fuckton) of papers.

“What we have to learn to do, we learn by doing.” — Aristotle

In addition to my extensive knowledge of the literature, I had a bunch of practical skills. Early on, I started using symbolic manipulation software (Mathematica, Maple, Macsyma). I learned how to do everything by hand, but found complex analysis could be automated. Using software like this is informative about how to think and use AI. This included many forms of stability analysis, derivation of methods, and other forms. This automation allowed me to attack a variety of methods and explore things efficiently. I also began to catalog problems and pathologies to combat. I explored ways to mitigate them.

I remember a quip Phil Colella threw at me: “You’re a really good method engineer.” I think it was meant as an insult, but eventually I took it to be true and a modest compliment. Yes, I became a “numerical methods engineer.” I had a great set of skills and knowledge to tackle all kinds of problems. I learned a great deal about hyperbolic conservation laws, multiphysics, numerical linear algebra, and turbulence. I combined these, mixing and matching to great effect.

“We do not remember days, we remember moments.” — Cesare Pavese

I remember when I gave my first talk on hyperbolic PDE solvers at the AIAA CFD conference. Phil was in the audience. I gave a really shitty talk. Too many equations and too much nervous energy all put together into a blur. At the end, I knew that I had fucked that up. After that, I worked diligently on becoming a better public speaker. Today, I feel like I am a very good speaker. Along the way, I gave a class on “Eulerian Hydrodynamics” at Los Alamos with 40 lectures and 1200 slides. They asked me to give the class again. In a sense, this was the capstone of my career there. I was giving back to the Los Alamos staff what I had been given by others before. I was giving my experience and knowledge as a reflection of my growth there.

My time at Sandia was a way for me to apply my skills, but the work I was assigned did not require the skills I developed later, after 2000. Everything I learned beyond that was surplus to requirements. I kept learning while I was there, but the environment was much more closed off and conservative, and it lacked generosity. It is hard to say whether that was a feature of Sandia culture or just the passage of time, but it was probably a combination of both.

“It is difficult to get a man to understand something when his salary depends upon his not understanding it.” — Upton Sinclair

I still learned a lot, especially through statistics and how to apply them, and the job was difficult. Much of what I did at Sandia involved adapting well-established technologies to the limitations and constraints we faced there. It also meant accommodating the relatively backward practices and less-than-optimal problem-solving approaches that seemed standard there.

In the end, my craft stagnated and did not grow during my time at Sandia. I feel some regret and a sense of loss about that. A lot of my recent writing has been an effort to understand and explain what happened. I think the simplest explanation is that Los Alamos was special and different from most places. Sandia was ordinary and lacked that special something: curiosity and open-mindedness. One reason for this is culture; another is the particular pace of our modernity. Sandia is fundamentally an engineering culture, with interesting local cultures surrounding it (next post). Those cultures tend to be more focused on maintaining the status quo than on advancing the state of science.

The unexamined life is not worth living.” — Socrates

That is not to say Los Alamos has all of this figured out. When I got there in 1996, they were still using code written in the 1950s for some of their most important analysis. The same attitude is present at Sandia with their famous code, CTH. It is even older now than that LANL code in a comparative sense. Both cultures tolerated unacceptably old technology for essential work. Why do places like these hold onto technologies that are obviously past their sell-by date? It seems to be a matter of resistance to change. The active choice of very conservative communities of practice. They lack the imagination and bold resolve to do anything different.

I’ll close by saying that I’m sure Los Alamos is a shadow of what it was in the 1990s, when it generously honed my craft and gave me a set of tools I’m proud of. That makes me wonder whether Sandia also declined during that era. In my experience, there isn’t much reflection of that decline at the lab itself. I see the deep-seated issues in the United States as a whole.

“Mastery is not a function of genius or talent. It is a function of time and intense focus applied to a particular field of knowledge.” — Robert Greene

tl;dr

My greatest professional accomplishment was founded on a failure. It shaped the rest of my career, but those heights were never returned to. The work was key to how my craft as a computational physicist grew. It made me who I am. The culture of Los Alamos in the early 1990s allowed it to happen. That culture is dead today. The culture at Sandia would have never allowed it.

“The most important questions in life can never be answered by anyone except oneself.” — John Fowles

An Inspired Lunchtime Question

A good friend took me to lunch recently to celebrate my retirement. He’s a few years out from his own retirement. Lunch was enjoyable as expected, but one moment stands out. He asked me about my greatest professional accomplishment. In your career, what achievement do you look back on with the most pride? What a spectacular question! I asked him as well. Our answers said a lot about us.

My answer was easy to come up with.

Back in 1994, I had moved into the computational science part of the Lab. I was continuing my examination of interface tracking while collaborating with Doug Kothe. I had come to understand the importance of these methods to the Lab and its core mission. The method of David Youngs, with piecewise linear (or planar) interfaces, was the state of the art. To understand it, I implemented it and used the manner of implementation I had seen in code. The time integration of the method used operator splitting. It worked pretty well.

I have a firm belief that to really understand a method, you improve it. An obvious way to improve this method is to remove operator splitting and create an “unsplit” method. This was far more complex and logic-intensive. When I tested it using the standard translation of simple shape tests, it worked.

I was also working on better tests for these methods. The standard tests were lame as fuck. The whole reason you need interface tracking is the phenomenon of shear. Without shear (and vorticity), Lagrangian methods are just fine. Vorticity in a flow is what makes Lagrangian methods fail. I introduced problems from the literature for advection with vorticity. These problems were introduced by Randy LeVeque and Piotr Smolarkiewicz. Randy also introduced a cool time reversal term where a flow could be rewound to the initial state. These problems were more realistic and difficult. They immediately broke my “improved” method. I could not debug it either. As I learned, the new method had too much cyclomatic complexity (too many nested if-thens).

An important point about this is failure. I tried something, and it failed badly. This failure would power accomplishment. Without the failure, all the good things that followed would not have happened. The other key point is that no one paid for this work. There was no project, no funding, no management involved. I was working on another project using C++ to code up methods. In those days, the compile-link cycle with C++ took forever (15 minutes a pop). The code I wrote, along with everything below, happened during the compile-link time. I was using Fortran, and it compiled and linked very fast. The Fortran on the UNICOS Cray Y-MPs was awesome.

“The scientist is not a person who gives the right answers, he’s one who asks the right questions.” — Claude Levi-Strauss

I went back to the drawing board. I ingested techniques from computational geometry. I realized the entire methodology for Youngs’ method could be implemented using it. I created a set of primitives from computational geometry and created Youngs’ method anew. It worked great. The code had very low cyclomatic complexity. I then did the unsplit version, and it worked too. It passed the tests and was easy to debug. Doug and I wrote up the work and submitted “Reconstructing Volume Tracking” for publication. It was published in the Journal of Computational Physics, including the history of these methods. The paper has over 2500 citations now.

Something else is the clincher for the greatest accomplishment. A few years later, I was involved in adding interface tracking to an important Eulerian code at Los Alamos. It turned out that any Eulerian code needs this, and AMR is not enough for interfaces. I was working with Ed Dendy, who was hands-on in the code. I was mapping out and deriving methods. We designed a method using artificial compression that blended with the existing code seamlessly. It is the standard method used today. Ed also added the version of Youngs’ method I created to the code. Recently, I confirmed that this code is still used today.

The achievements here were multiple. I created a new way to implement a method that is a genuine improvement. I created a better version of the existing method. We wrote a comprehensive paper on the methodology that was well-received. Finally, I created better test problems for this class of methods. These tests have become far more standard and helped drive improvements across the board in interface tracking. Improved level set methods are a key example. Finally and critically, the methods and code I worked on are used for important problems at Los Alamos. It is a substantial and meaningful contribution to our national security.

All of this produces a great sense of pride and accomplishment.

The lessons from this chapter of my career are deep and bountiful. Almost everything about it is counter to the current scientific environment at the Labs. Today, no manager trusts their staff enough to allow this unless you do the work off the clock.

“To be yourself in a world that is constantly trying to make you something else is the greatest accomplishment.” — Ralph Waldo Emerson

Postscript

This work is a source of pride and achievement, but it also fills me with dismay. I left Los Alamos 20 years ago, and this work is 30 years old, with the implementation in the code dating back 25 years. I expect the technology to have advanced over that time. The work I did so long ago could be a foundation, but it should have been replaced by something better. The fact that something better is not on the agenda at Los Alamos leaves me with a sense of despair. The part of me that feels responsibility and duty would rather have seen that work replaced by something newer, better, and more capable. It’s been a quarter century after all. My accomplishment was shaped by curiosity, responsibility, and duty.

I was driven by a desire to support the Lab’s mission and national security by addressing something that seemed very important. Those are things to celebrate with hopes young people find the same inspirations. My greatest worry is that the combination of motivations today undermines this. The basic incentive structure for the institutions and people is different today. Another key element is trust, which is lower in both directions. Proper incentives and trust were key to my accomplishment.

“If what you have done yesterday still looks big to you, you haven’t done much today.” — Mikhail S. Gorbachev

I’m going to write shorter, more frequent posts for a bit. See how that works. I would love any feedback or reactions.

References

Rider, William J., and Douglas B. Kothe. “Reconstructing volume tracking.” Journal of Computational Physics 141, no. 2 (1998): 112-152.

Rider, William, and Douglas Kothe. “Stretching and tearing interface tracking methods.” In 12th Computational Fluid Dynamics Conference, p. 1717. 1995.

Kothe, Douglas B., and William J. Rider. “Comments on modeling interfacial flows with volume-of-fluid methods.” Submitted for publication (1995).

Kothe, Douglas, W. Rider, Stewart Mosso, J. Brock, and John Hochstein. “Volume tracking of interfaces having surface tension in two and three dimensions.” In 34th Aerospace Sciences Meeting and Exhibit, p. 859. 1996.

tl;dr

Algorithmic stability is an essential concept for solving problems with computers. Studying stability provides a foundation for everything a computer does. Any algorithm for any purpose can exhibit stability issues that are fatal. Simply put, a lack of stability arises from a small change in data, yielding a huge change in results in an anomalous way. The archetype of stability is the numerical solution of differential equations. This arose from key wartime applications and early computing use (WW2 and the atom bomb). Von Neumann’s numerical algorithm for computing shock waves failed miserably. It became a key topic to study and understand. This spurred essential developments in computational science. Despite progress, problems still exist needing attention. There are parallels to AI that we should look to for better outcomes with that technology.

“Mathematics is the art of explanation.” ― Paul Lockhart

In the Beginning

By 1944, the American effort behind creating the atomic bomb was beginning to make genuine progress. Part of the Manhattan Project was nascent computational physics efforts led by the vision of John Von Neumann. In Los Alamos, they were examining design ideas numerically. The effort was led by Hans Bethe and Richard Feynman, both future Nobel Prize winners in Physics. Von Neumann had led the concept of using computing for science. He also devised a computational scheme for shock hydrodynamics. Bethe and Feynman did early calculations using Von Neumann’s method. It failed, catastrophically. The method produced horrible oscillating results (ringing). Today, we would recognize this as numerical instability.

This outcome was recognized as a problem. Scientists in Los Alamos (Peierls) also conceptualized ways to mitigate it. These changes to Von Neumann’s method would not be realized until after the war. During the war, success was achieved with another algorithm devised by the British mission at Los Alamos. The method was first proposed by Peierls and then refined by Skyrme. It was fundamentally different than Von Neumann’s. It involved some similar methods to Von Neumann’s, but computed the shock wave explicitly via tracking. The shock was treated with precision by Feynman via the Rankine-Hugoniot conditions as an internal boundary. Von Neumann’s method was more general, but fatally flawed.

“Mathematics is the cheapest science. Unlike physics or chemistry, it does not require any expensive equipment. All one needs for mathematics is a pencil and paper.” ― George Polya

From the foundational work of Courant, Friedrichs, and Lewy in 1928, the methods used a heuristic for stability. This is the famous CFL condition named for them. This is a simple argument about the domain of dependence for information (waves). This means a finite speed of propagation of sound waves should not move more than the spacing of the mesh in a time step. It is simple and rational, and still the core of stability today. It is also subtly flawed, as I’ve written. Both of the two WW2 methods used it, and it holds today (for explicit methods). For Von Neumann’s method, it was insufficient. Something else was needed to keep it viable. This seeded two developments key to the future of computational physics.

A key question to consider is whether current difficulties are seeding any similar fundamental mathematics. The failures and problems are things to explore that lead to discovery. AI has vast swaths of problems needing math. Efforts are generally lacking and sub rosa.

“But in my opinion, all things in nature occur mathematically.” ― Rene Decartes

Context and Modern Significance

“If rounding errors vanished, 95% of numerical analysis would remain.” – Nick Trefethen

My focus here is numerical stability for solving differential equations, especially partial differential equations. Other forms of numerical stability are equally important for computation. The classical case is Gaussian elimination, which drove much of the early work, including von Neumann’s efforts with Goldstine. This study algorithms and how to structure them for stable computation. It also draws attention to when computation is untrustworthy, defined by the structure of the problem. By and large, these algorithms are designed to produce exact or precise results, except when numerical errors occur. Roundoff error and changes in the order of operations can lead to stability issues. These approaches focus on eliminating those problems and ensuring reliable results.

The primacy of stability in numerical computations powered the growth of the technology. Contrast this with the lack of a coherent, encompassing theory for large language models. This is true when you look at how these models behave during training or use. That behavior is stochastic, so it is perhaps logical that stability would not be a key concern. I would counter that stability concepts might yield order to AI where it is lacking today. The absence of stability theory erodes confidence in the underlying techniques. Addressing it should be a priority going forward, and would likely yield practical benefits.

“Do not imagine that mathematics is hard and crabbed, and repulsive to common sense. It is merely the etherealization of common sense.” ― Lord Kelvin

Unfortunately, as noted previously, the United States is loath to invest in this kind of work of late. The applied math that has been so crucial to computational physics is largely absent today in AI. It is absent in almost every scientific endeavor. The partnership between the two is all but dead. This all points to an indictment of the American strategy in AI (or lack thereof). We assume we’re in the lead, but we’re increasingly failing to do the things that would sustain and expand that lead. Instead, the USA is doing everything it can to lose the lead in the long run. This applies to what we are and are not doing. My goal is to outline a key methodology for the birth and growth of computational physics. In contrast, it will highlight the lack of a similar framework for our current revolution. This is badly needed and would be a huge boon to AI in the future, improving every aspect of the technology.

Many things stand in the way, but one thing is galling in the extreme. I’ll note that claims of industrial espionage are true, but it is also propaganda. They’re also used to make Americans think they’re stealing the edge of technology through spying. It belittles an adversary we should fear for their own creativity and creation. We avoid recognizing our own self-defeating philosophy: our systematic internal attack on science and technology. These self-defeating actions are the core of the danger.

“If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat.” —Sun Tzu

Industrial espionage happens and is part of the picture, but it pales in comparison to the absolute incompetence and the attack on science and technology that the government and industry are waging on themselves. We’re the ones laying the groundwork for Chinese supremacy in science and technology, because China is inherently competent at doing the right things while we do all the wrong things. It will become clear that the Americans are the ones who need to engage in industrial espionage of the Chinese quite soon, because we will increasingly be firmly behind. The coffers of science and technology that the Chinese might be emptying were filled decades ago, and those coffers will now be empty and threadbare. Given American attacks on its own internal science and technology, any claims otherwise are simply hubris and empty patriotism, so common these days.

What’s clear today is that the United States is losing its dominance, and it’s losing it because of its own actions. The Chinese are pulling ahead because they’re competent and doing the right things, while the USA is undermining its science and technology.

The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman

To anyone who questions the current American thought, I’ll share a personal anecdote from right before I left Sandia.

I was taking the required training for a conference where British scientists would be present for classified discussions. The training is standard before these meetings, and the Sandia person leading it was a young fellow. During the training, he made comments belittling Russian or Soviet accomplishments in nuclear weapons. He seemed to believe those achievements were illegitimate and based on espionage, particularly pointing to Klaus Fuchs.

While the Russians did engage in espionage and stole much of the design for early atomic bombs, it would be wrong to belittle their capabilities or the brilliance of how they executed their program. Once they knew an atomic weapon could be produced, they could build it from scratch. The only thing the American design did at Trinity was confirm that it could be done. Once you know it can be done, it becomes a much simpler matter to do it.

This also belittles the brilliance of scientists like Andrei Sakharov, who did unique and brilliant work in support of the Soviet hydrogen bomb program. They had many other extraordinary theoretical physicists like Landau and Zel’dovich. I corrected the young man and encouraged him to look at the real history, as explained by Richard Rhodes. The literature has some obscure publications by Russian scientists who were key to their program. These are eye-popping. The same fictions are at work today with regard to the Chinese and their advances in AI. The same fictions in a host of fields. We Americans need to be guided by facts and not pulled in by faux patriotism and hubris about our scientific prowess. We should be completely in sync with the rather politically incorrect notion that the Manhattan Project was powered more by the efforts of immigrants than by homegrown American science.

“Intellectual freedom is essential to human society — freedom to obtain and distribute information, freedom for open-minded and unfearing debate, and freedom from pressure by officialdom and prejudices.” — Andrei Sakharov

Another key aspect of this dynamic is the cost of information protection. The American system operates under the premise of assumed superiority. This then leads to protectionist policies in classification and export control. These policies can be quite effective in limiting spying and keeping information from being lost. It is also effective in controlling this information domestically. The impact harms innovation and progress in the USA. American institutions have cracked down on information dissemination more and more. This has played a significant role in dragging American science down. Even worse, if American science is behind, these policies will act as friction, undermining catching up. We may already be behind, and the approach taken is foolish at best.

The question is whether the United States will wake up to a Sputnik-like moment and turn things around. Alternatively, the United States may be done, and the nearly century-long era of international dominance, fueled by supremacy in science and technology, could come to an end. That chapter has not been written yet, but the current signs are worrying. The American public seems asleep and is not taking the steps needed to return to the approach that helped us achieve our dominance.

With that table setting out of the way, let’s get back to our story.

“All that it is reasonable to ask for in a scientific calculation is stability, not accuracy.” – Nick Trefethen

Shock Capturing Methods and Applications

After World War 2, science jumped to the attention of everyone. This was powered by the stunning use of atomic bombs against Japan, shining a light on the Manhattan Project. In the wake of this, resources and importance flowed toward scientific work. This allowed the powerful vision of John Von Neumann to begin to come to fruition. Part of his vision was the use of computers for scientific work. Part of his vision was chastened by the lack of success for his differencing scheme. Its results were unstable. This was not an isolated problem, as other schemes that seemed reasonable produced bad results. He sought to understand this.

By the summer of 1946, he had produced an analytical tool to examine stability. This was his spectral stability analysis of finite difference schemes. He presented it in Los Alamos that summer for parabolic equations. It only applies to linear equations, but provides results that guide nonlinear methods and equations. It is still an essential tool for understanding methods. It can provide answers to subtle stability problems invisible to inspection. The method is still used broadly today. This method leaked out over the remainder of the 1940s, most notably by Crank and Nicolson. Von Neumann published the method in 1950 along with another key invention.

The method used successfully in the War was a tracking method. As application complexity grew, the tracking method became intractable. Los Alamos was looking at the H-bomb (super, as they called it). A more general method was needed to support progress. Robert Richtmyer, who was leading the Theoretical Division, sought to produce this. He worked on the suggestion of Peierls to add dissipation to Von Neumann’s method. By 1947, Richtmyer worked out the solution. He would add an extra term in and near shock waves to produce the entropy needed by a shock’s passage. This was dubbed artificial viscosity. In my opinion, an unfortunate choice. It is not artificial, but entirely physical. Shocks create singularities, and entropy rise is necessary to navigate the singularity. It allowed shocks to be captured and not tracked. Von Neumann’s method was salvaged, and the stage was set for numerical methods to flourish on complex applications.

This was the first shock-capturing method. The paper Von Neumann and Richtmyer published had three major advances. First, there was the differencing scheme Von Neumann devised in 1944, but it failed in use. The second was the stability technique Von Neumann devised. Finally, the third was the artificial viscosity invented by Richtmyer to stabilize shocks. As I’ve noted before, this viscosity has also become the foundational method in Large Eddy Simulation. The reason for this commonality is dissipation in turbulence that functionally acts similarly to shocks. In both cases, the large-scale dissipation acts as viscosity vanishes in nearly identical manners. The understanding of shocks is primarily one-dimensional. For turbulence, this is fully three-dimensional and introduced by Kolmogorov as his “4/5 law.”

“As technology advances, the ingenious ideas that make progress possible vanish into the inner workings of our machines, where only experts may be aware of their existence. Numerical algorithms, being exceptionally uninteresting and incomprehensible to the public, vanish exceptionally fast.” – Nick Trefethen

Many of the most important aspects of computational science arose out of this single thread of science and math. These works are foundational to the entire field. They show us the path not taken today with AI. We should heed this as a warning.

Subtle Stability

“Computing has changed not only the way mathematics is practiced, but mathematics itself.” —Peter Lax

Given the success of this method, numerical solutions exploded onto the scene. Applications of numerical methods multiplied, especially in fluid dynamics. The key aspect was the proof that it could be done. Everyone knew it was possible, especially in Los Alamos. This seeded ideas with luminaries like Peter Lax and Frank Harlow. Both of these men pioneered whole fields of practice in what we now call CFD. Harlow started to invent a host of methods still used today, first in compressible fluids, then in incompressible fluids. He also invented an important aspect of turbulence modeling. Lax produced fundamental methods for compressible fluid dynamics. He produced several methods that are foundations today, and the theory of conservation form, which underpins computational aerodynamics.

“One must watch the convergence of a numerical code as carefully as a father watching his four year old play near a busy road.” — J. P. Boyd

More importantly, he produced foundational mathematics for computational science. Lax’s equivalence theorem connects stability, consistent approximations and convergence. Convergence is the promise that more computing power yields better answers. This has underpinned the pursuit of better computers for better science. It took the stability work discussed above and connected it to numerical approximation accuracy to ground all efforts. The practice of code verification is dependent on this theory. Scientists design stable, accurate methods to solve equations and implement these in code. If they can be shown to converge to the correct solution, we have assurances of correctness. Lax provided the theory to see this.

“The unreasonable effectiveness of mathematics in the natural sciences.” – Eugene Wigner

The importance of the applications to the world meant that the theory for partial differential equations came first. Ordinary differential equation theory from Dahlquist actually came afterward. One might logically think the order would be reversed, but not. This is due to the importance and energy for PDEs.

The inspiration for this post is my discovery of stability problems that infest our codes today. These are associated with strong expansions I’ve discussed recently. The problems are multiple, with under-estimates of wave speeds leading to unstable time steps and schemes. The saving grace is that the instability is a mid-frequency and not at the grid scale. Shocks produce instability at the grid scale Thus they are catastrophic almost immediately. The expansion instability is mitigated by a little resolution and a few time steps. The question is whether the instability has a lasting influence on the solution. Does the instability leave a lingering corruption of the solution that is never healed? Right now, theory can’t tell us. Evidence says that this may well be the case/

Postscript

By embracing the past, the current administration is killing the future. The future is about moving forward and adapting to the problems we already have. The solutions of the past will not work in the future. That is true in every area, from warfare to science. This administration is focused on the past, and its actions will destroy a positive future for the United States. I spent my entire professional career at nuclear weapons laboratories. I watched them decline and lose capability. I can say without equivocation that we are not ready for what is coming. Our science is not prepared to compete in the world that is about to unfold.

We’ve arranged a society on science and technology in which nobody understands anything about science and technology, and this combustible mixture of ignorance and power, sooner or later, is going to blow up in our faces.” — Carl Sagan

We need to get our shit together, fast. The current administration is damaging our ability to compete on every front. True, whether it is national security or economics. We are not ready for the world that is coming. If you want to highlight how unprepared we are, just look at Iran or Ukraine. In both wars, American power is failing in spades. We are not adapting to the world that is already here.

America has rejected expertise, and it is going to do our country harm. Whether we are talking about nukes, fusion, AI or drones, we need experts to define our strategy. Then to execute it and adapt to an every changing landscape. I have seen this rejection of expertise all the way down to the working level at a national laboratory. There, my own expertise was deemed too cutting and too critical. It was way too much for the incompetent leaders to take seriously.

Our current leaders and their approach to leadership are failing the country at this critical juncture, and time is running out. Looking to billionaires to lead us is foolish. They are increasingly driven by their own greed and could not care less about society as a whole. Just look at how they have stewarded the technologies that have driven their wealth. In every case, those technologies have caused real harm to our society, to our children, and to the way we live. They show no responsibility other than maximizing their own wealth and power. Letting them guide something with the power of AI is suicidal.

“In mathematics you don’t understand things. You just get used to them.” —John von Neumann

This essay shows a small vignette of how fundamental math supports computational science. The computational science has become essential for work supporting a host of applications. These include everything from nuclear weapons to climate science to car design. It encapsulates much of what is missing from science today (including, but not limited to AI).

“Mathematics, rightly viewed, possesses not only truth, but supreme beauty—a beauty cold and austere, like that of sculpture, without appeal to any part of our weaker nature, without the gorgeous trappings of painting or music, yet sublimely pure, and capable of a stern perfection such as only the greatest art can show.” ― Bertrand Russell

References

Von Neumann, John. Proposal and analysis of a new numerical method for the treatment of hydrodynamical shock problems. Applied Mathematics Group, Institute for Advanced Study, 1944.

VonNeumann, John, and Robert D. Richtmyer. “A method for the numerical calculation of hydrodynamic shocks.” Journal of applied physics 21, no. 3 (1950): 232-237.

Mattsson, Ann E., and William J. Rider. “Artificial viscosity: back to the basics.” International Journal for Numerical Methods in Fluids 77, no. 7 (2015): 400-417. Morgan, Nathaniel R., and Billy J. Archer. “On the origins of Lagrangian hydrodynamic methods.” Nuclear Technology 207, no. sup1 (2021): S147-S175.

Margolin, Len G., and K. L. Van Buren. “Richtmyer on Shocks:“Proposed Numerical Method for Calculation of Shocks,” an Annotation of LA-671.” Fusion Science and Technology 80, no. sup1 (2024): S168-S185.

Lax, Peter D. “Hyperbolic difference equations: A review of the Courant-Friedrichs-Lewy paper in the light of recent developments.” IBM Journal of Research and Development11, no. 2 (1967): 235-238.

Lax, Peter D., and Robert D. Richtmyer. “Survey of the stability of linear finite difference equations.” Communications on pure and applied mathematics 9, no. 2 (1956): 267-293.

Kolmogorov, Andrey Nikolaevich. “A refinement of previous hypotheses concerning the local structure of turbulence in a viscous incompressible fluid at high Reynolds number.” Journal of Fluid Mechanics 13, no. 1 (1962): 82-85.

Smagorinsky, Joseph. “General circulation experiments with the primitive equations: I. The basic experiment.” Monthly weather review 91, no. 3 (1963): 99-164.

Higham, Nicholas J. Accuracy and stability of numerical algorithms. Society for industrial and applied mathematics, 2002.

Grcar, Joseph F. “John von Neumann’s analysis of Gaussian elimination and the origins of modern Numerical Analysis.” SIAM review 53, no. 4 (2011): 607-682.

Lax, Peter D. “The flowering of applied mathematics in America.” Siam Review 31, no. 4 (1989): 533-541.