“One of the greatest pains to human nature is the pain of a new idea.” — Walter Bagehot
The Quest for Understanding
One of the animating ideas of my career in science is a quest to understand the origin, development, and acceptance of ideas. This has been a periodic challenge I took on throughout my career. You will find it in my dissertation, which was mostly self-driven and self-taught. It ultimately became a professional achievement, a more complete accounting of the early history of computational fluid dynamics (CFD). I have written an essay about this on this blog and given a presentation on the topic. It is perhaps the talk I have delivered most often. Beginning in gestational form at the JRV Symposium in 2013. There was a history of CFD around limiters after 1970 by Bram van Leer. It was titled “The History of CFD Part 2”.
I asked Bram, “Where is part 1?”
In response, Bram told me I needed to create part 1 of the history. He said, “That is for you to do!”
Afterward, expanding into a comprehensive presentation that I have given six or seven times over the years. It feels like this will be one of my understated achievements. Under the hood of this history is a desire to understand how we got where we are today. Not simply account for what happened, but why and how.
I have posed the affirmative question before: why should a method be forced to satisfy only one requirement of quality when we know there are two? Those two requirements are the preservation of adiabatic solutions and the conservation form. Right now, the field simply picks one and walks away from the other. We choose rather than demand both. The choice then becomes calcified into doctrine. What I am pondering here is not the choice itself, but the reason we have never insisted on meeting both. The answer is that we refuse to hold ourselves to a higher standard. Progress is still needed and still possible, but it has stagnated to nothing because of an outright refusal to accept good ideas and merge them together. That refusal is not a technical limitation. It is a failure of will.
One of the key formative moments in developing these ideas came in the year 2000. That summer, a special conference took place at Los Alamos to honor the 70th birthday of Burt Wendroff. Burt, except for a few years, worked his entire career at Los Alamos. He is best known for his PhD work under the advisement of Peter Lax. This produced the celebrated dual achievements of the Lax-Wendroff theorem and the Lax-Wendroff method. It was then documented in a paper in Communications in Pure and Applied Mathematics. Both achieved massive success over the years and have shaped modern computational fluid dynamics in profound ways. It has had profound success in compressible aerodynamics, defining much of what is currently done.
The occasion of this conference was my opportunity to actually meet Peter Lax. I have met many great scientists over the years. A few became acquaintances, and far fewer became friends. What I have learned from engaging with them is that these people are all human. They are all extremely smart. Some are extremely lucky, and their greatness comes from being smart enough while also being in the right time and place to create what they create. Of all the people I have met, Lax is perhaps the greatest, or nearly the greatest. That puts him in rather prominent company.
There was a photograph taken at the conference on the first day. I remember it quite well. I don’t actually remember who took the photograph. Burt and Peter came together warmly, and then the photographer noted that the ordering was wrong. He asked Peter and Burt to exchange places to get the ordering of the authorship for the Lax-Wendroff paper. Then the photo was taken. I was standing behind where the photo was taken, watching all this happen. It remains a very fond memory. Its great that the internet remembers the photo too.
Another source of great memory in that conference was my presentation there. I felt immensely lucky to do so. One of the more memorable moments came when Burt pulled me aside to let me know that Peter suffered from narcolepsy. I should not be alarmed if he fell asleep during my talk. Sure enough, when I gave my talk, Peter fell asleep halfway through. Bert’s warning saved me from being mortified, as Peter had already become quite the hero of mine by then.
My talk came from a period of my technical history when I was beginning to explore turbulence modeling. My interest was specifically in implicit large eddy simulation. I was trying to understand the role that things like limiters play in the ability of these methods to act as effective turbulence models. In addition, I was searching for the effective subgrid model implied by these methods. In particular, I was looking at the role of nonlinear dispersion.
Speaking in front of Peter was a particular honor, but also something that made me quite nervous. Peter had studied dispersion in calculations, inspired by the work of John von Neumann. In his original shock method, which was tried and successfully used during World War II, dispersion was rampant. Lax examined the solutions von Neumann’s method created, which produced a huge amount of ringing. This work was done in conjunction with another exceptional scientist, David Levermore. Presenting my work to Peter was both a source of pride and something that felt very dangerous. He was an exceptional genius after all, even by Los Alamos standards.
Nonetheless, looking back, this work was the beginning of perhaps my most successful research. It certainly grew into the best and most obviously successful project I ever worked on. One that resulted in many highly cited papers and a book. That book tied together many researchers in the area for the first time. It led to my engaging and meeting many other scientists working on similar things, including Jay Boris, one of the inventors of limiters. Paul Woodward was in the book too, but I already knew him.
Expanding My Understanding of Creation
“Novelty emerges only with difficulty, manifested by resistance, against a background provided by expectation.” – Thomas Kuhn
Over time, I came to understand how many of these ideas came into being. How they merged with other ideas. How some of them became distorted and lost their original intent. This is perhaps more evident in turbulence simulation than anywhere else. There, the Smagorinsky model is the original model used to represent subgrid turbulence in LES. Over the years, the basic identity of the Smagorinsky model has been lost. It was first and foremost a rearticulation of the artificial viscosity model developed by Richtmyer in conjunction with von Neumann. It has been repurposed to clean up calculations of geophysical flows. This repurposing came at the suggestion of Jule Charney, von Neumann’s collaborator in applying computing to geophysical fluid mechanics. Over the years, this connection to shock capturing was lost, if not outright ignored. This seems to be a combination of genuine and willful ignorance.
The greater animating purpose of this essay is the general lack of acceptance and use of Lax’s work at its place of origin, specifically Los Alamos. It includes the many labs that followed Los Alamos’ lead. This includes Livermore, Sandia, and overseas Labs. This requires a bit of history, involving how Lax came to these ideas. Then, some deeper pondering of why these key breakthroughs have generally gone unaccepted and unused at the place where they were born.
Lax’s ideas and achievements in the area of hyperbolic conservation laws are incredible and undeniably great. This was recognized when he received the Abel Prize in 2005. The fact remains that they hold very little sway and acceptance at places like Los Alamos or Livermore seems mysterious. They are revered in CFD around the rest of the World. This is the world shaped by the Manhattan Project. This ushered both computers and computational science into being. The original achievements of John von Neumann are more broadly accepted in the Los Alamos-connected work. Even though they lack the rigor Lax provides. Why?
“Most men can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they have delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.” – Tolstoy
Consider the Lax equivalence theorem as another example. The practice of verification that relies upon this theorem is carried out begrudgingly in places like Los Alamos. The same holds for the Lax-Wendroff theorem. The theorem with its demands, and its promised benefits, of conservation form. In a deep sense, it has greater rigor and applicability than the equivalence theorem. Conservation, too, is seldom used. The extensive mathematical theory of hyperbolic conservation laws, the crowning achievement of Lax’s work in the area, is also rarely leaned upon as a technical basis either.
Given the greatness of Lax’s work, one has to question the reasons. The question becomes even more pregnant when you realize that the origin of this work traces directly to Los Alamos. It is grounded in the achievements of von Neumann, which demonstrated the capacity of computers to solve this class of problems. This inspiration made Lax’s work timely and essential. It was clearly a motivation for greater understanding. Much great math was delivered, yet not used where the inspiration originated.
“One resists the invasion of armies; one does not resist the invasion of ideas.” — Victor Hugo
Lax was drafted into the U.S. Army in 1944. He was ultimately assigned to Los Alamos as part of the Manhattan Project, a recognition of his potential and burgeoning genius. There, he did not work on fluid dynamics, but rather on neutron diffusion. After leaving Los Alamos, he returned to New York, where he completed his PhD at the Courant Institute under Friedrichs. Upon receiving his PhD, he promptly went back to Los Alamos and spent a year there as a staff member. He worked closely with one of the famous Keller brothers, both great mathematician.
One of the most amazing things to discover was the write-up of Lax’s plan of attack on the mathematical theory of hyperbolic conservation laws. It was completed in conjunction with Keller during his time at Los Alamos. The outline in this report is complete and defines the next 25 years of research. One can trace all of Lax’s completed works in conservation laws from it. Ultimately, ending with the mathematical theory of hyperbolic conservation laws published in 1973. It is all laid out there in astonishing detail.
Lax, Peter D. Hyperbolic systems of conservation laws and the mathematical theory of shock waves. Society for Industrial and Applied Mathematics, 1973.
The advantage Peter had when he returned to Los Alamos was the knowledge that one could successfully compute solutions to hyperbolic conservation laws. The basic premise had been proven out in World War II by Feynman and Bethe with their method, based on Tony Skyrme’s work. By the time Lax returned to Los Alamos, Richtmyer had devised the artificial viscosity and made shock-capturing methods possible. Now computers and methods could be turned loose on shock waves. Thus, Lax worked with the knowledge that all of this could be done. It was now a matter of bringing order to it, along with some degree of mathematical rigor and knowledge to guide it. The question remains: why has his magnificent work had so little impact on its place of origin?
How Original Ideas Remain Dominant?
“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.” — John Kenneth Galbraith
If one looks at the codes developed and used at a place like Los Alamos, only one of them is written in what we call conservation form. This is a code I worked on, one that goes by the moniker xRage these days. My friend Rob and I could not ascertain the basic structure of the high-order Godunov code that xRage purported to be. Rob had trained under Phil Roe and Bram van Leer for his PhD. I am basically self-trained in the same area. The basic recipe, construction, and maxims that Rob and I had absorbed in our training could not be discovered in the way this code was written. We expected to find a viable first-order Godunov method under the hood. No such thing existed in the code. The second-order spatial differencing was novel as well. It worked, but not in a standard way. The same mentality applied to the Riemann solver and the multi-material treatment. I was part of adding interface treatments to the code. Without those, the material interfaces were too diffusive.
This code was originally written by a genius of a code developer, Mike Gittings. Mike was able to work magic with codes and get things to work by magic and code wizardry. The code worked and was astoundingly robust, loved by its users. In many ways not much different than the dynamic at Sandia with CTH (or its predecessor, CSQ). Yet the technical basis of the code was unrecognizable to those of us who had been formally trained. Here is the one example in the weapons complex where Lax’s framework might actually apply, and it is almost impossible to see what is actually there. To me, this is rich with irony, especially at Los Alamos.
“Nature, to be commanded, must be obeyed.” — Francis Bacon
One of the things that I believe is that the importance of hydro for multi-physics codes is not well understood. I was reminded recently in a LinkedIn comment by my friend Nathaniel Morgan from Los Alamos. The reason it is so important is that it provides the material map on which all the physics in a multi-physics code are computed. It also provides the state of that material thermodynamically and, more importantly, its position. Physicists naturally think about most of the physics in the Lagrangian frame of reference. Thus, this is comfortable for them. Thinking about it in a different frame of reference only muddies the picture and introduces new physical effects. The Lagrangian frame perspective is taken as primal.
“For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.” – Richard Feynman
That said, most of the physics that we are solving are extremely muddy and difficult. In almost every case where you’re dealing with a high-speed, high-energy flow, turbulence and mixing are present. As soon as the flow begins to twist and distort, the Lagrangian frame becomes untenable, and you lose the ability to clearly think about problems in that frame of reference. In a sense, this is the core of the problem with failing to progress away from the von Neumann view of how to compute this. Von Neumann’s method was originally composed in 1D with relatively limited computing. It is almost obvious that the simplicity it embodied is not appropriate for the type of computing and expressibility of ideas we have available to us today. Why is this approach still relied upon?
The reason why is the power of being the first mover. That first mover, the person whose reputation remains unsullied, is John von Neumann. Von Neumann is perhaps the greatest polymath of the 20th century, a genius of almost unparalleled magnitude, and perhaps one of the smartest people in the history of humanity, certainly on par with someone like da Vinci.
If one looks at the codes in use across the labs today, the model of computing is very much derivative of von Neumann’s method, paired with Richtmyer’s artificial viscosity. What cannot be said is that these codes bear any of Lax’s rigor in how they are used. Given the power and quality of Lax’s work, I have always found this mysterious. Moreover, much of what Lax did actually generates genuine animosity on the part of people at the labs. I find this difficult to understand. In a field with few results to lean upon, some of the best ideas are ignored. This entire phenomenon places an extreme amount of importance on the legitimacy conferred by being first and on the power of incumbency. The nature of things is that once something works at all, it is hard to displace, even when its shortcomings become increasingly obvious over time.
“He that will not apply new remedies must expect new evils; for time is the greatest innovator.” — Francis Bacon
Indeed, the main place where ideas related to Lax’s work found acceptance is through the work of van Leer principally, and limiters in general. Limiters provided a means to solve a problem that Lagrangian von Neumann-type methods ran into in multiple dimensions: the inevitability of mesh tangling. Mesh tangling meant that remesh-and-remap technology was necessary for these calculations to go to 3D and ultimately solve the problems they were designed to solve. Van Leer’s method made low dissipation and higher accuracy possible under these conditions, and it was accepted almost immediately across the entire lab complex.
What goes unacknowledged is that van Leer’s work was shaped by the work of Peter Lax. Lax provided the foundation, and everything van Leer did was built upon it. In a sense, Lax’s work did have its impact, but only in a derivative sense. Van Leer also engaged with Paul Woodward from Livermore. Paul’s collaboration with Bram helped bridge the methodology to the Labs. Van Leer’s methods were almost immediately adopted at Livermore, Los Alamos, Sandia, and AWE in England. It was rapid and pervasive.
These methods, once they leave the Lagrangian frame, have a problem: they do not conserve energy. This comes from the form of the energy equation used. There is a method to conserve energy devised originally by Roger DeBar of Livermore. Generally, that method is not robust enough to be used for practical problems. Scientists at Los Alamos have devised a much better version of it that may be good enough for practical use. Nonetheless, it is not in an obvious flux-conservation form; thus, Lax-Wendroff may not apply. The same thoughts are needed for conservation in the Lagrangian frame for staggered mesh methods. It can happen, but it depends on subtle, discrete details. For example, the first version to conserve relied upon a bizarre definition of kinetic energy (could be negative). Modern methods are strictly flux-conservative with some very specific choices. These include the time integration, where a particular predictor-corrector of second-order is needed. This also includes a corner mass invariant definition for momentum-kinetic energy.
This lack of energy conservation is exactly the underlying is that prompted my decision to retire. Even 40-plus years after van Leer remap became standard, energy conservation is uncommon. There are ways to make it work, but they are seldom used in practice. It is not in conservation form; thus, Lax-Wendroff is not applicable. I find this head-scratching and difficult to square with the importance of the work done with codes.
“The most difficult subjects can be explained to the most slow-witted man if he has not formed any idea of them already; but the simplest thing cannot be made clear to the most intelligent man if he is firmly persuaded that he knows already.” — Tolstoy
Here we are, 65 years after the Lax-Wendroff theorem was published. People remain completely unwilling to acknowledge this work or utilize its fundamental results in what they do. This rejection creates an absolute crisis of legitimacy, and it reflects decisions made over and over again. I have asked my friends at Livermore, and they report that the energy-conserving methodologies developed at Livermore, which do not use conservation form, are not utilized in their codes once they go into a mode using remap. Those codes have the same problems, albeit to a lesser degree than CTH. That is more a reflection of the higher quality of the methods, and of being modern codes. Nevertheless, the basic premise of this essay persists: Lax’s work is not accepted at its place of origin.
“It is difficult to get a man to understand something, when his salary depends upon his not understanding it.” — Upton Sinclair
I have written about this mentality before, in my essay on the shortcomings of current methods, where the field has chosen to honor the physical concept of computing adiabatic solutions rather than computing solutions in conservation form. The unwillingness to demand that methods meet both requirements means that the method that routinely and casually produces adiabatic results is chosen. Conservation is rejected as a preeminent requirement. Really, we should have both requirements met.
I will reiterate that I believe this is a fundamental mistake. The requirements and gifts of conservation exceed those of an adiabatic solution. Moreover, an adiabatic solution is a desired outcome, but it is utterly pathological, representing a rather profound resistance to the second law of thermodynamics, which manifests itself in mixing and turbulence most often in fluid dynamics. Adiabatic solutions are ephemeral and pathological. That means they shouldn’t be the foundational character of the method. They are wonderful if you can engineer them. Making them the premise upon which you design and accept new numerical methods is an act of faith, if not borderline lunacy.
Adiabatic solutions are largely an article of faith, mostly mythological. Shock waves are ubiquitous. You want to compute a weak solution, and you need to compute the physically relevant weak solution. I place this demand first, and the preservation of adiabatic solutions second, based on this analysis. Accepting what people have always done is easy. Change is hard. This alone explains most of what we see. The incumbent has an immense advantage. Without a devotion to progress and change, you simply do things the way you always have. You make excuses about why change is unnecessary, and you point to the successful track record of the past as the only proof you need to keep doing things the same way.
“The difficulty lies not in the new ideas, but in escaping from the old ones, which ramify… into every corner of our minds.” — John Maynard Keynes
The Essence of the Problem: Change is Hard
When I consider what John von Neumann would have thought of all this, I come to the following conclusion: he would have recognized the correctness and genius of Lax’s work. He was ever devoted to progress, and he would have seen the need to merge Lax’s ideas into his own. He would not be pleased that the ideas he pioneered continue to be used without the modifications necessary to make them more reliable and more generally useful for solving humanity’s most enduring and difficult problems. Perhaps we can turn over a new page, get to the place where we combine these ideas, and give each of them the air they deserve and the progress we all need.
Power from being the first mover, or from initial success in something, comes from the fact that it works. It’s very similar to the advantage conservatives have over progressives. The policies conservatives espouse have already been tried and have some functional basis in society. The same happens in science: why change something that works for something new? For this reason, very old solutions with significant flaws live on. Change is hard and uncertain. This is especially true in a world where trust itself is scarce and increasingly existential.
There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things.” — Machiavelli
The Regularized Singularity 