“If you do not work on an important problem, it’s unlikely you’ll do important work.” — Richard Hamming,
High-resolution methods are a bit stale these days (IMHO). I actually thought this 20 years ago, and little has changed. I’ve seen a little progress over time, but the deficiencies and lack of progress seem to be moving faster. Lots of high-order non-oscillatory methods, and DG without real improvement in capability. I have an explanation or two below. I’ve written about the reasons before. Lack of responsiveness to evidence or failing to get evidence. Sod’s shock tube results are a key bit of the problem. The practice of only applying qualitative comparison is pathological.
“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” — Lord Kelvin
I also need to remember why I started doing V&V in the first place. My primal interest was in numerical methods and improving them. I saw test problems with quantitative results as a means to improving methods. You would measure results and use the measurement to decide where progress was being made. This seemed utterly logical and scientific. It was obvious. It is still not done today. Even with decades of expansion of V&V as a technical endeavor, evidence is poorly sought and even more poorly used. Progress in simply doing science for numerical methods is remarkably backwards.
We see multiple fields engaged at cross purposes in advancing the field. Mathematics is a key to progress. It is essential and also the root of the issue. Finding rigor and proof is amazing, as it is limited. Rigor comes with limits and assumptions that are restrictive. Real problems have discontinuities and chaotic-unstable results. Math is still rather limited there. This is a challenge, but also used to justify the lack of progress. For nonlinear, complex, and chaotic problems, there are still great gaps in our mathematical knowledge. It is work that is needed and necessary.
Physics and engineering tend to be somewhat application-focused on practical things. This leads to lots of corner-cutting. The lack of math rigor is then an excuse for this. Lots of things we do work well, but we don’t understand why. A subfield where I’ve actually worked shows this: implicit large eddy simulation. It remains an observation without a systematic explanation. I’ve contributed much of what passes as explanation. Too often we just need rules of thumb and utilization of common belief. There is not nearly enough effort in systematically understanding. This understanding is needed to reliably engineer things. Numerical methods remain too poorly understood.
V&V is thrust into these chasms. Verification interacts with the math deeply. It is largely an interface between math and numerics. It is supposed to be quantitative, and it is too rarely that. More frequently, it is only done in the ideal case. When things become difficult (Sod’s shock tube), we drop quantified results. Validation is the interaction with physics and engineering. Remarkably. the quantitative practice becomes even less common. In both cases the existing practice refuses to respond to evidence.
At least we are becoming more honest about how we ignore V&V. Recent trends simply reject V&V as a necessary part of simulation. It is simply too much bad news. It’s a bummer. Stagnation simply assures this.
This refusal is the origin of stagnation. CFD and other simulations tumble into witchcraft and wizardry. The path forward seems relatively obvious. It is difficult, but we’ve made it more so.
Things Aren’t Moving
“In science if you know what you are doing you should not be doing it. In engineering if you do not know what you are doing you should not be doing it.” — Richard Hamming,
I have a list of issues to explore in my head. This post is a means to document this. I’ve touched on most of this before, but it makes sense to put it all together in one place. It would make for a great set of PhD theses or research programs. If V&V were healthy, we could provide evidence of stagnation and progress against that. Fat chance of either happening today. But a guy can dream, can’t he?
Many of these things were already in my mind 20 years ago as I left Los Alamos. I had put effort into a number of these during my time at LANL. Several came into my consciousness at Sandia. The key is the lack of time and resources to tackle these there. The same lack was growing at LANL. I don’t think staying there would fix this. The issues are things I’ve spilled so much ink about. The nature of V&V resistance. The national obsession with exascale computing and big iron over mathematics and algorithms. The lack of risk-taking and trust in research today. The retreat of applied mathematics from practical importance.
As an example, I’ll mention an idea that my friend Vince had. It plays a role later in this essay. If you look at contact algorithms that are used in solid-mechanics codes, there are usually giant heaps of cyclomatic complexity, with a whole bunch of nested if-then-else statements. These are a nightmare for V&V, reproducibility, and computation in general. Vince wanted to study taking all of those out and replacing them with a smooth sigmoidal function that would make the code continuously differentiable.
He put in a research proposal at Sandia or modern day America that never had a chance. The reason? It was pointed at a sacred cow. Never mind that it was an incredibly good idea that met all sorts of requirements for programmatic impact. It simply was too different to support; it wasn’t HPC. There has been far too little emphasis on algorithms as a path to performance. In the process, progress has been lost, and performance has been hurt.
The major player in the long-term deficit in computational science is the multi-decade obsession with high-performance computing hardware. This hardware obsession has sapped the balance out of computational science and left a deficit. Almost everything I talk about here is dealing with the methods that, in one way or another, are at the foundation of many of our most important simulation tools. This is true for climate, astrophysics, nuclear weapons, nuclear reactors, clean energy, and on and on. If you’re generally solving any sort of multi-physics where hydrodynamics plays a major role, these methods are extremely important.
The cost of our failure to focus on these algorithms is probably most acutely measured in efficiency. The focus on high-performance computing is the most inefficient way to improve our simulation capacity. We’ve taken the same approach for artificial intelligence. Again, there are probably massive algorithmic efficiencies that should be explored with AI, but right now the focus is almost entirely on computing power.
“Numerical analysis is the study of algorithms for the problems of continuous mathematics.” — Nick Trefethen
So without further ado, let’s make a simple list of where I see some interesting issues to explore and improve upon:
1. What makes a real difference in accuracy on real problems
1a. What is the optimal mix of time and space accuracy in method design
2, What is the optimal mix of accuracy and efficiency on real problems
3. Computing nonlinearly stable time steps for nonlinear problems
4. Understanding nonlinear stability of solutions for space and time
4a. Exploring ideas around nonlinear stability for discretization
5. Combining adiabatic-entropic solutions with conservation. Why do solutions for very strong expansions not converge?
6. What are the right concepts for convergence in physical instabilities
7. Explaining implicit large eddy simulation’s effectiveness.
So, let’s dig into each.
1. What makes a real difference in accuracy on real problems
For real problems, accuracy is limited to first-order (or less) in almost every case. The lack of smoothness is the reason. High-order methods have value, but the great expense does not deliver commensurate with the effort. The question is what aspects of high-order methods deliver value in terms of accuracy. Evidence seems to point to some value being found with parts of high-order methods. There is definitely a huge leap from first to second-order, but the accuracy gains seem to saturate. How does one strike the balance? At the same time, high-order accuracy is more fragile and prone to failures.
To solve this issue, a number of things are needed. Rigorous guidance is a huge challenge for applied mathematics. Breakthroughs in math would be golden, but may not be possible. Instead of this guidance, we need to test methods and parts of discretization elements on real problems. Part of this is verification using analytical results. There is then the issue of how this transfers to validation problems. Right now, we are operating on a mix of blind faith and rules of thumb. We need to turn this toward science and real measurement. Examples abound, including shock wave problems and direct numerical simulations of turbulence. Neither is guided by genuine quantitative examination of results.
2. What is the optimal mix of accuracy and efficiency on real problems
This is a follow-on to the first issue. How does one balance accuracy and efficiency? The issue is that accuracy is expensive. Convergence rates are low. Accuracy is also found at the expense of robustness. This is a third issue to throw into the balance. Again, the vehicle is testing. It seems unlikely that mathematics adds as much as the first issue. The other major issue is the definition of efficiency. I’ve defined it as accuracy (fidelity) per unit cost. In a dull sense, given an accuracy of solution, the lowest cost is the most efficient. In the process, we can find the best ways to achieve accuracy (with robustness).
3. Computing nonlinearly stable time steps for nonlinear problems
For many problems the time step control is done via linearization. The dynamics of these initial value problem can contain much faster time scales. Hydrodynamics is a key example. The evolution of the problem can immediately include phenomena that is an order of magnitude faster. This is a relatively difficult problem to solve. One simple idea I had is to test a time step at the end of a cycle. Ask the question, was that time step stable given the dynamics at the end of the time step. If it was not stable, reject the time step and do it over with a smaller (stable) time step. This is simple and the main critique is the cost of storing another solution vector. It seems to me that the cost of an unstable calculation is vastly greater.
“Newton said, ‘If I have seen further than others, it is because I’ve stood on the shoulders of giants.’ These days we stand on each other’s feet.” — Richard Hamming
4. Understanding nonlinear stability of solutions for space and time
I have written about this before in several posts.
5. Combining adiabatic-entropic solutions with conservation. Why solutions for very strong expansions do not converge?
I wrote a whole set of blog posts on these topics. They remain largely open issues.
“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman
6. What are the right concepts for convergence in physical instabilities
If one has a physical instability like those found in turbulence or material mixing solutions do not converge normally. The concepts of convergence for initial value problems do not apply. There is a huge leap of faith and confidence that finer meshes lead to better results. Still convergence as a notion is mostly a leap of faith. Many classical hydro problems show more and more structure with resolution.
In a sense the notion is that given two solutions of the Euler equations, a “swirlier” calculation is a better calculation. This is seen in classical Kelvin-Helmholtz instabilities. Applied math has been noodling on this via solutions as distributions, but progress has been spotty. This is both difficult hard work, and hugely necessary. For many essential initial value problems, the massive calculations are leaps of faith. This includes all of direct numerical simulation of turbulence. That said, ideas in turbulence are the best hope here.
“Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.” — George Box
7. Explaining implicit large eddy simulation’s effectiveness.
This is finishing a project I gave up 20 years ago when I left Los Alamos. I had started looking at the modified equations for modern methods from MUSCL to WENO and many in between. Working closely with Len Margolin we identified some important parallels between LES modeling and the truncation error. In particular there is a term that shows up at second order with major significance. It comes from having a stable second-order method in conservation form. It looks much like the self-similarity model in LES. That model is notoriously unstable. With modern methods it appears and is selectively stabilized. I believe there is much more waiting to be found with ILES.
“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.” — Max Planck
The Regularized Singularity 