The beginning of wisdom is the definition of terms.

― Socrates

Accuracy, Fidelity, Resolution and Stability, Robustness, Reliability

These six words matter a lot in the discussion of numerical methods yet their meanings are quite poorly understood or even regularly conflict. Even worse they are poorly articulated in the literature despite their prominent importance in the character of numerical methods and their instantiation in code. Maybe I can make some progress toward rectifying this issue here. Consider this post to be a follow-on and compliment to the earlier post on improving CFD codes, https://williamjrider.wordpress.com/2015/07/10/cfd-codes-should-improve-but-wont-why/.

One of the biggest issues in providing clarity in these matters is the ultimate inability of classical mathematics to provide clear guidance as we move from the abstract and ideal world to the real and messy reality. In using methods to solve problems with a lot of reality things that work well or even optimally in an idealized version of the World fall apart. In the ideal World very high-order approximations are wonderful, powerful and efficient. Reality tears this apart and all of these characteristics change to problematic, fragile and expensive. The failure to bridge methods over to the real world undermines progress and strands innovation behind.

Accuracy is a simple thing to define, the deviation of an approximation from reality. For an idealized mathematical representation this measurement is “easy”. The problem is measuring this for any real world circumstance is quite difficult to impossible. One way to define accuracy is related to the asymptotic rate of convergence as the degree of approximation is changed this is “order-of-accuracy”. The connection of order of accuracy, and numerical precision is rather ill defined and fuzzy. Moreover reality makes this even harder.

A clear way that the World becomes real is shock waves (and other similar nonlinear structures). A shock wave or similarly discontinuous solution renders most numerical methods first-order accurate at best. The current knowledge about how formal ideal high-order accuracy connects to measurable accuracy in the presence of discontinuities is sketchy. This manifests itself in having research in methods focus on high-order accuracy without any understanding of how this would translate to accuracy for real World problems. Today the efficacy of high-order methods is merely an article of faith.

We have alternative definitions of accuracy focus on the terms fidelity and resolution. Both of these terms are even more fuzzy that accuracy. These both get applied to methods that provide their value to (more) real World circumstances where formal accuracy is diminished. Thus important classes of methods are defined as “high-fidelity” or “high-resolution”. Both of these definitions are used to imply capability to provide good solutions when the ugliness of reality intrudes into our idealized reality.

Peter Lax provided a definition of resolution in an unfortunately obscure source. There and in a fascinating interview with Lax by Phil Colella (http://history.siam.org/oralhistories/lax.htm) the concept was discussed with an astounding proposition, perhaps higher than second-order was too accurate and produced solutions that did not have enough “room” to capture true solutions. It is not a final answer, but it does yield a direction to thinking about such things.

Here is the key passage on the topic of resolution:

“COLELLA: So you have shocks, why second-order? What were you thinking?

LAX: Well, you want greater accuracy, but even more you want greater resolution. I defined a concept of resolution. If you take a difference method and you consider a set of initial value problems of interest, which in practice could be some ball in L1-space, anything will do, and then you look at the states into which it develops after the unit-time, any given time, that’s another set. The first surprise is that this set is much smaller for nonlinear…for linear equations where time is reversible, the size of this set is roughly the same as the original set. For nonlinear equations, which are not reversible and where the wave information is actually destroyed, it’s a much smaller set. And the measure of the set that is relevant is what’s called entropy or capacity with respect to some given scale delta. So the first thing to look at is what is the capacity or entropy of this set of exact solutions. Then you take a numerical method, you start, you discretize the same set of initial data, then you look at what you get after time t goes to whatever the test time was. A method has a proper resolving power if the size of this set is comparable to the size of the exact solution; if it’s very much smaller it clearly cannot resolve. And first-order methods have resolution that is too low, and many details are just washed out. Second-order methods have better resolution. In fact, I was trying to – well, I want to bring up the question: could it be that methods that are even higher order (third, fourth) have perhaps too much resolution, more resolution than is needed? I just bring this up as a question.”

I might offer a bit of support for that concept in the case of genuinely nonlinear problems below. In a nutshell, the second-order methods with conservation form provide truncation error that matches important aspects of the true physics. Higher order methods will not capture this aspect of the physics. I’ll also note that Len Margolin and I have followed a similar, but different line of thinking looking at implicit large eddy simulation (ILES). ILES is an observation that high-resolution methods appear to provide effective turbulence modeling without the benefit of explicitly added subgrid modeling.

So let’s talk about the archetype of nonlinear, real World, messy computations, shock waves. In some ways shocks are really nice, they are inherently dissipative even in the case where the system is free of explicit molecular viscosity. Dissipation in the limit of zero viscosity is one of the most profound aspects of our mathematical description of reality. For physical systems with a quadratic nonlinearity including shocks and turbulence, this dissipation scales, with being the velocity and being a constant. At its core is the imposition of reality on idealized math to describe reality, and provide a useful, utilitarian description of mathematically singular structures. This character is present in turbulence as well. Both have basically the same scaling law and deep philosophical implications.

This form of nonlinear dissipation comes directly from the application of the conservation form to methods with second-order accuracy. For energy this term is precisely the form of the asymptotic law except for its connection to the discrete system. If the method achieves a formally higher than second-order accuracy this term disappears. For very simple second-order schemes there are truncation errors that compete with this fortuitous term, but if the linear accuracy of the method is higher order, this term is the leading and dominant truncation error. This may explain why schemes like PPM, and FCT methods produce high quality turbulence simulations without explicit modeling, but methods like minmod or WENO do not. The minmod scheme has a nonlinear truncation error that dominates the control volume term. For WENO method the higher order accuracy means the dissipation is dominated by a combination hyperviscous terms.

These deep philosophical implications are ignored by the literature for the most part, with shocks and turbulence defining a separation of focus. The connections between these topics are diffuse and unfocused. A direct connection would be a stunning breakthrough, but entrenched interests in both areas conspire against this. This remarkable similarity of the limiting dissipation in the absence of viscosity have been systematically ignored by scientists. I see it as utterly compelling or simply brutally obvious for a quadratic nonlinearity. Either way the similarity is meaningful. One of the key problems is that turbulence is almost completely grounded in the belief that it can be completely described by incompressible flow. No one seems to ever question this assumption.

Incompressibility is a physically limited approximation of reality, but not reality. It renders the equations to be intractable in some ways (see the Clay prize for proving the existence of solutions!). The unphysical nature of the equations is two-fold: sound speeds are infinite and thermodynamics are removed (especially harmful is the loss of the second law). Perhaps more problematically is the loss of the very nonlinearity known to be the source of dissipation without viscosity for shock waves, that is the steepening of arbitrary disturbances into discontinuous shock waves.

I’ve written before about stability and robustness with a focus on the commonality of their definition https://williamjrider.wordpress.com/2014/12/03/robustness-is-stability-stability-is-robustness-almost/. The default basic methodology for stability analysis was discussed too https://williamjrider.wordpress.com/2014/07/15/conducting-von-neumann-stability-analysis/. If we add the term “reliable” the situation is quite analogous to the issues with accuracy. We ultimately don’t have the right technical definitions for the useful character of practical reliability and robustness of numerical methods and their executable instantiation in code. Stability is necessary for robustness and reliable, but robustness and reliable imply even more. Typically the concept of robustness applies to practical computational methods used for real World (i.e., messy) problems.

The key issue for high order methods is the inherently non-smoothness and lack of clean structure in the real world. This messiness renders high-order methods of questionable utility. Showing that high-order methods improve real world, practical, pragmatic calculation is challenge for the research community working in this area. Generally high-order methods show a benefit, but at a cost that makes their viability in production software questionable. In addition the high-order methods tend to be more fragile than their lower order cousins. The two questions of robustness in use and efficiency are the keys to progress.

Given all of these considerations, what is a path forward to improving existing production codes with higher order methods?

I will close with a set of proposals on how we might see our way clear to improving methods in codes by balancing requirements for high-order accuracy, high-resolution, robustness and stability. The goal is to improve the solution of “real” “practical” problems, not idealized problems associated with publishing research papers.

1. For practical accuracy high-order only matters for linear modes in the problem. Therefore seek high-order only for the leading order terms in the expansion. Full nonlinear accuracy is a waste of effort. Full nonlinear accuracy only matters if the flow is fully resolved and the fields contain the level of smoothness equal to the scheme (they never do!). This would allow the quadratures usually invoked by formally high-order methods could be reduced along with the costs.
2. For nonlinear structures, you just need second-order accuracy, which gives you a truncation error that matches the asymptotic structure of dissipation analytically. Removing this term may actually harm the solution rather than improve. The reasoning follows Lax’s comments above.
3. Nonlinear stability is more important than linear stability in fact nonlinear stability will allow you to use methods that are locally linearly unstable. Extending useful nonlinear stability beyond monotonicity is one of the keys to improving codes.
4. Developing nonlinear stability principles beyond monotonicity preservation is one of the keys to progress. A test of a good principle is its ability to allow the use of linearly unstable methods without catastrophe. The principle should not create too much dissipation outside of shocked regions (this is why ENO and WENO are not good enough principles). In a key way monotonicity-preserving methods naturally extended the linear monotone and first order methods. The next step beyond monotonicity preservation has not built upon this foundation, but rather introduced an entirely different concept. A more hierarchical approach may be needed to achieve something of more practical utility.
5. The option of fully degrading the accuracy of the method to first-order accuracy must always be in play. This step is the key to robustness. Methods that do not allow this to happen will never be robust enough for production work. This is another reason why ENO and WENO don’t work for production codes.

Logically, all things are created by a combination of simpler less capable components

– Scott Adams (or the Dogbert Principle that applies to high resolution schemes in Laney, Culbert B. Computational gasdynamics. Cambridge University Press, 1998).

References

Lax, Peter D. “Accuracy and resolution in the computation of solutions of linear and nonlinear equations.” Selected Papers Volume I (2005): 184-194.

Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39.9 (2002): 821-841.

Grinstein, Fernando F., Len G. Margolin, and William J. Rider, eds. Implicit large eddy simulation: computing turbulent fluid dynamics. Cambridge university press, 2007.

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225.2 (2007): 1827-1848.

Broadwell, James E. “Shocks and energy dissipation in inviscid fluids: a question posed by Lord Rayleigh.” Journal of Fluid Mechanics 347 (1997): 375-380.

Bethe, Hans Albrecht. Report on” The Theory of Shock Waves for an Arbitrary Equation of State”… 1942.

It is dangerous to be right in matters on which the established authorities are wrong.

― Voltaire

Last week I received a question via email that prompted this post. It proposed that the title of the post is true. It talked about the benefits of pushing the envelope with high performance computing. The gist of the thought is that by pushing the envelope with computing we can effectively use the mainstream high end computing resources better. This is true without a doubt. It is a benefit of having a bleeding edge research program in any area. A better question is whether it is the most beneficial way to allocate our current efforts.

Everything in excess is opposed by nature.

― Hippocrates

The ability of such a program to impact the World positively is a much more difficult and nuanced question. For high performance computing we have seen decades long focus on the power and speed of the hardware that has fueled a growth in peak computing speed consistent with Moore’s law. Unfortunately a host of essential capabilities for realizing this computing power as a scientific capability have not been similarly supported. Without these other capabilities such as physical models, solution methods, algorithms, the computing hardware is nothing more than a very expensive way to use electricity. The very things that make computers really useful for the purposes of modeling and simulation are the things we have not invested in for these same decades.

The distance between insanity and genius is measured only by success

― Ian Fleming

The issue with it is that this benefit does not exist in a vacuum. There are limits to the financial and human resources that may be devoted to the objective of “predictive” modeling and simulation. My politically incorrect assertion is that the focus on high performance computing hardware is a suboptimal approach to achieving the end result of maximizing the capability for modeling and simulation. The devotion to progress in hardware is sapping the resources that might be applied to attacking this problem in a more balanced manner.

If we have no heretics we must invent them, for heresy is essential to health and growth.

― Yevgeny Zamyatin

This imbalance is primarily exemplified by the failure to invest in people, experiments and models. When I speak of investing in people, it goes far beyond simply paying people. Investing in people means creating systems where people can develop and grow in their capability while feeling safe and secure to take huge risks. Talented people who take risks are necessary for progress, and without such risk-taking progress stagnates. Without taking risks we cannot develop talent, the two are intertwined.

An expert is someone who knows some of the worst mistakes that can be made in his (her) subject, and how to avoid them.

– Werner Heisenberg

We have destroyed the vitality of our experimental sciences, which further amplifies the destruction of our scientific staff. Experimental science is absolutely necessary to advance science. This has the knock-on effect of undermining the creation of new, better models for science. Having the World’s fastest computer cannot replace any of these shortcomings.

It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.

― Richard Feynman

A big part of the problem is that aspects of modeling and simulation closer to the modeling are starved. The modeling associated with high performance computing is static and relatively free of progress. We are not progressing toward introducing new models into codes and ultimately practical use in a healthy way. The same can be said for solution methods that provide either more powerful or effective ways of solving models. These methods then provide the impetus for algorithms that systematically provide means of solution. Our current computing emphasis is geared toward efficiently delivering existing algorithmic solutions for existing methods for existing models.

History warns us … that it is the customary fate of new truths to begin as heresies and to end as superstitions.

― Thomas Henry Huxley

Inadequacies in models are simply allowed to persist while the fiction of their rescue by more powerful computing platforms a false promise. The real scientific answer to this issue is that a more powerful computer, better software, better algorithms or better methods cannot save a model that is incorrect. Despite this maxim we keep attacking modeling and simulation dominantly through computing hardware.

…a more powerful computer, better software, better algorithms or better methods cannot save a model that is incorrect.

The fastest computer has even taken a page from Cold War nationalism with the “missile gap” being replaced by the “supercomputer gap”. Chinese prowess in computing is demonstrated by their supercomputer and funding is supposed to follow the American effort to regain the crown of fastest. I, for one, am far more worried about the Chinese and Russian investments in human resources in modeling, methods and algorithms than computers. By all appearances these investments are significant. The American public should be far more worried by the encroachment of mediocrity into the research staff at our National Labs than our lack of the fastest computers.

In the republic of mediocrity, genius is dangerous.

― Robert G. Ingersoll

The management of our National research programs and devotion to intellectually questionable priorities is by far a greater threat to our National security than anything our adversaries are doing. A perfect example of the problem is the increasingly legacy nature of our computer codes. We looked at the local history of one type of code and noted that up until 1990 we had a new code every five years. Since 1990 we have simply kept the same old codes. This is 25 years with the same code with the same methods and same approach. We have basically lost an entire generation of staff. We have lost a generation of progress and research. The models and methods are frozen in time. This is a recipe for mediocrity at best, disaster at worst.

Progress is born of doubt and inquiry. The Church never doubts, never inquires. To doubt is heresy, to inquire is to admit that you do not know—the Church does neither.

― Robert G. Ingersoll

So, yes, having leading edge computing is a great, wonderful and important thing for the country. It’s true for any country desiring international leadership. Getting to properly defining what leading edge computing is actually comprised of becomes difficult. A completely naive and incorrect way to define leading edge is having the fastest or most powerful computer on Earth. Everyone knows what a fast computer is, but a powerful computer is a more subtle question. I would argue that in many respects my iPhone is more powerful and useful than virtually any supercomputer I’ve used. The problem in defining powerful is the limited utility of supercomputers. Supercomputers are important for solving scientific problems, which are necessarily limited in context.

The riskiest thing we can do is just maintain the status quo.

― Bob Iger
Moreover, this effort for leading edge computing lies in a resource constrained trade space and the focus on hardware leaves other efforts starved for funding or focus.  Even this discussion leaves most of the important nuance untouched, the dependence on people and their talent. The issues around the efficacy of the HPC efforts are subtle and far more nuanced than the mere power of the computer. A powerful supercomputer is useless without talented people to use it. The problem in the United States is that people are something we have chronically and systematically under-invested in. Our universities are in decline and part of a vastly corrupt system that underserves the public at a massive cost. The consequences of this decline in education are then amplified by the destruction of the social contracts associated with post-educational work.

Unhappiness lies in that gap between our talents and our expectations.

― Sebastian Horsley

Employees are viewed as commodities and infinitely replicable, even at National Labs. The lifetime employment necessary for deep sustainable expertise has been replaced by an attitude more appropriate for Wal-Mart. Over the past couple of decades the sort of strong scientific leadership once provided by the National Labs has been replaced by Lab employees who are little more than “sheeple” who bend to political will rather than speak up and offer their expertise instead of politically correct pablum. Today the Labs simply do what they are told. Their spirit has been beat out of them. I might even be so bold and to say that the attempt to lead in scientific computing says more about our lack of scientific leadership than our commitment to it.

The smart way to keep people passive and obedient is to strictly limit the spectrum of acceptable opinion, but allow very lively debate within that spectrum….

― Noam Chomsky
If the models, methods, and codes used on our fastest computer are lacking, the computer’s value is diminished to the point of being worthless. A computer will provide results that are as good as the codes running on it. If the people running the problems on the computer are similarly lacking, the computer’s value is diminished as well. The computer is only answering questions as good as the people asking the questions. I believe that we have systematically failed to make investments in the models; methods, codes and people sufficient to make the focus on computing power pay off. We have created a new generation of legacy codes (just because its written in C++ does not keep it from being a legacy code!), with legacy models and methods and a staff that cannot fully understand the codes or calculations they are running. The fiction that all we need to do is refine the mesh and run calculations on bigger computers to predict nature continues to hold sway.

Every truth in this world stretched beyond its limits will become a false doctrine.

― K.P. Yohannan
This is a situation that the mismanagement of the labs has created.  The DOE has done the same thing to its Labs that DOD did. DOD foolishly destroyed its research labs 30 years ago, and over the last 20 years, DOE has followed a similar path toward mediocrity. The national resource of these Labs is being allowed to fade and wither.  We have allowed the Labs to atrophy. Our approach to high performance computing is but one example of this. The situation is even worse when you look at what we have done to our experimental sciences. Under these conditions having the lead in computing hardware will do little actually support our national security because we have failed to keep competence in the fundamentals necessary for efficacy.

Change almost never fails because it’s too early. It almost always fails because it’s too late.

― Seth Godin
In other words for the hardware to really matter in the delivery of predictive simulation and modeling, everything upstream of the machine needs to be right.  This includes the people.  We have failed to invest in leading edge technology in the very things that make the supercomputer valuable. We have a skewed and unbalanced view of how computing works, which allows the justification of our current programmatic path, but fails to deliver true progress.

Heretics are the new leaders. The ones who challenge the status quo, who get out in front of their tribes, who create movements.

― Seth Godin

Happiness is the pleasantest of emotions; because of this, it is the most dangerous. Having once felt happiness, one will do anything to maintain it, and losing it, one will grieve.

― Kij Johnson

Rarely has any movie stuck with me like Inside-Out. It is hard to imagine how an animated movie about an 11-year-old girl has left such a lasting impression on me, but it has. It is advertised as a kid’s movie with hidden themes for their parents to enjoy, but that is wrong. It is a movie for adults that will entertain your kids. The movie is well done like you’d expect from Pixar and it packs a massive punch. It’s a really good movie, but it’s a better adult movie than kid’s movie.

What has given this movie such staying power?

It gave me a lot to consider that can apply to my own life. It provided an open playful theme to deeply think about extremely deep themes in how you personally manage your response to reality through your emotions. The main characters in Inside-Out are the emotions of the 11-year-old girl, Riley. The dominant character is Joy, with other key emotions Sadness, Anger, Fear and Disgust all vying to respond to her World. When the story begins, Joy dominates Riley’s life, but a major life event throws everything into turmoil. The main conflict is how much should you allow sadness into your life and what happens when you don’t.

Riley has spent 11 years growing up in Minnesota living a life of fun, family, friends, and especially hockey. Her family picks up, and moves to San Francisco and all hell breaks loose inside Riley. Everything she has known and loved in her life has been taken away. Sadness begins to creep into her reactions, and Joy tries to reign in this reaction. Push comes to shove and disaster ensues. Riley’s emotions fall apart along with her. Riley begins to make bad decisions based on inappropriate emotional responses, and her life spirals out of control. The foundations of her entire psyche begin to fall apart. Meanwhile Joy and Sadness try to get back into her emotionalwheelhouse (along with all the hijinks that the child movie viewers will enjoy).

Emotions are not irrational, but how we rationalize the world. The key is to use the right ones for the information we are being handed. We need to use our cognitive tools represented in emotions to process our own world and respond appropriately. Irrational responses are applying the wrong emotion to the situation, which can result in bad outcomes. The world throws a lot at us and the reactions need to be proper. We have a myriad of events that range from wonderful to horrible, and our emotions need to match the event. This is rational. Therefore sadness, fear, anger or disgust despite being negative are important and proper for many things. They are necessary for our survival.

In the end, Joy and Sadness return to save the day with a key realization. Sadness is appropriate and important as a response to traumatic events. Joy is not. Trying to make everything joyous and happy is simply wrong and inappropriate. Sadness adds a texture to formerly unambiguously happy memories that now have complexity the younger Riley lacked. Riley is growing up and rebuilding her emotional world. Joy is no longer so dominant; the other emotions now have much more sway in her emotional makeup. Letting Sadness in was the key to her response to crisis.

The lesson will always repeat itself, unless you see yourself as the problem–not others.

― Shannon L. Alder

A key lesson for us today is that unhappiness and sadness are the right response to many things in life. We have a culture that drives home the message that we should always be happy and satisfied, and feel joy. Many situations call for sadness, or anger, or fear, or disgust. If we don’t feel the right emotion our reaction to the situation is inappropriate and harmful to our long-term well-being. The movie has the deep and impactful message for all of us that “negative” emotions are more than simply okay, they are powerful and proper responses to many of life’s events.

Even more powerfully there are circumstances where happiness or joy is utterly wrong and harmful. The attempt to imprint joy onto these situations hurts us and provides an improper personal response to life’s travails. Life is about balance and the push and pull of events. For us to learn, grow and develop correctly, we must process and respond to events in a way that fits the events. When we do not respond properly we can hurt our future. Moreover the full repertoire of emotions is needed to live our lives with the tools to deal with what life throws at us. The so-called negative emotions should be embraced when they fit the circumstances. In the case of Riley sadness was proper and healthy, and the attempt to feel nothing but joy left her in a tailspin of almost disastrous magnitude.

Don’t cry because it’s over, smile because it happened.

― Dr. Seuss

 

If failure is not an option, then neither is success.

― Seth Godin

Alternate titles

Progress in CFD has stalled. Why?

Why are methods in CFD codes so static?

Why is the status quo in CFD so persistent?

Status quos are made to be broken.

― Ray Davis

There are a lot of reasons for lack of progress in CFD codes, and here I will examine one particular issue. The reality is that there is a myriad of issues plaguing modern codes. I’ve written about issues with our modeling and its lack of suitability for tackling modern simulation questions. One of the major issues is the declaration that success won’t be reached until computers are far more powerful. This is also testimony to the lack of faith in innovation and creativity in research (risk aversion and fear of failure being key). As a result funding and focus for improving the fundamentals of CFD codes has dried up. It like the community has collectively thrown up it hands and said, “its not worth it!”

The riskiest thing we can do is just maintain the status quo.

― Bob Iger

We have an overly focused research program toward utilizing the next generation of computing hardware. The major overarching issue is a general lack of risk taking in our research programs spanning from government funded pure research, through applied research programs and extending to industrially focused research. Without a tolerance for failure and hence a risk, the ability to make progress is utterly undermined. This more than anything explains why the codes are generally vehicles of status quo practice rather than dynamos of innovation.

Yesterday’s adaptations are today’s routines.

― Ronald A. Heifetz

If one travels back into the mid-1980’s there was a massive revolution in numericalmethods in CFD codes. Methods that were introduced at that time remain at the core of CFD codes today. The reason was the development of new methods that were so unambiguously better than the previous alternatives that the change was a fait accompli. Codes produced results with the new methods that were impossible to achieve with previous methods. At that time a broad and important class of physical problems in fluid dynamics were suddenly open to successful simulation. Simulation results were more realistic and physically appealing and the artificial and unphysical results of the past were no longer a limitation.  

These methods were high-resolution methods such as flux corrected transport (FCT), high-order Godunov, total variation diminishing (TVD), and other formulations for solving hyperbolic conservation laws. These terms are in other words the convective or inertial terms in the governing equations transporting quantities through waves most typically through the bulk motion of the fluid. These new (at that time) methods produced results that when compared with preceding options were simply superior by virtually any conceivable standard. In addition, the new methods were not either overly complex or expensive to use. The principles associated with their approach to solving the equations combined the best, most appealing aspects of previous methods in a novel fashion. They became the standard method almost overnight.

Novelty does not require intelligence, but ignorance, which is why the young excel in this branch.

― Anthony Marais

This was accomplished because the methods were nonlinear even for linear equations meaning that the domain of dependence for the approximation is a function of the solution itself. Earlier methods were linear meaning that the approximation was the same without regard for the solution. Before the high-resolution methods you had two choices either a low-order method that would wash out the solution, or a high-order solution that would have unphysical solutions. Theoretically the low-order solution is superior in a sense because the solution could be guaranteed to be physical. This happened because the solution was found using a great deal of numerical or artificial viscosity. The solutions were effectively laminar (meaning viscously dominated) thus not having energetic structures that make fluid dynamics so exciting, useful and beautiful.

When your ideas shatter established thought, expect blowback.

― Tim Fargo

The new methods would use higher accuracy approximations as much as possible (orsafe to do so), and only use the lower accuracy, dissipative method when absolutely necessary. Making these choices on the fly is the core of the magic of these methods. The new methods alleviated the bulk of this viscosity, but did not entirely remove it. This is good and important because some viscosity in the solution is essential to connect the results to the real world. Real world flows all have some amount of viscous dissipation. This fact is essential for success in computing shock waves where having dissipation allows the selection of the correct solution.

The status quo is never news, only challenges to it.

― Malorie Blackman

The dissipation is the essence of important phenomena such as turbulence as well. The viscous nature of things can be seen through a technique known as the method of modified equations. This method of numerical analysis derives the equations that the numerical method effectively solves. Because of numerical error when you solve an equation numerically, the solution more closely matches a more complex equation.

In the case of simple hyperbolic conservation laws that define the inertial part of fluid dynamics, the low order accuracy methods solve an equation with classical viscous terms that match those seen in reality although generally the magnitude of viscosity is much larger than the real world. Thus these methods produce laminar (syrupy) flows as a matter of course. This makes these methods unsuitable for simulating most conditions of interest to engineering and science. It also makes these methods very safe to use and virtually guarantee a physically reasonable (if inaccurate) solution.

The new methods get rid of these large viscous terms and replace it with a smaller viscosity that depends on the structure of the solution. The results with the new methods are stunningly different and produce the sort of rich nonlinear structures found in nature (or something closely related). Suddenly codes produced solutions that matched reality far more closely. It was a night and day difference in method performance, once you tried the new methods there was no going back.

Negative results are just what I want. They’re just as valuable to me as positive results. I can never find the thing that does the job best until I find the ones that don’t.

― Thomas A. Edison

This is the crux of the issue with moving on to even more advanced methods, the quantum leap in performance to be had then simply won’t be repeated. The newer methods will not yield a change like the initial movement to high-resolution methods. The newer methods will be better and more accurate, but not Earth-shatteringly so. In today’s risk adverse world making a change for the sake of continual improvement is almost impossible to sell. The result is stagnation and lack of progress.

The problems don’t end there by a long shot. Because of the massive improvement in solutions to be had with the first generation of high resolution methods to a very large extent cost wasn’t an issue. With the next generation of methods, the improvements are far more modest and the cost of using them is an issue. So far, these methods are simply too expensive to displace the older methods.

The issues don’t even stop there. The new methods also tend to have relatively large errors compared to their cost. In addition the newer methods tend to be fragile and may not handle difficult situations robustly. The demands of maintaining formally high-order accuracy are quite expensive (time, space integration demands are costly whereas the first generation high resolution methods are simple and cheap). The result is that the newer approaches are methods that “do not pay their way.”

The balance of accuracy and cost has not been negotiated well. This whole dynamic is worth a good bit of discussion.

The key to this issue is the lack of capacity for high-order accuracy to be achieved in practical problems. To get high-order accuracy the solution needs to be smooth and differentiable. Real problems conspire against this sort of character at virtually every turn with singular structures both in the solution itself, not to mention geometry or physical properties. Real objects are rough and imperfect, which tends to breed more structure in solutions. Shock waves are the archetype of the problem that undermines high-order accuracy, but the problem hardly stops there.

The measure of intelligence is the ability to change.

― Albert Einstein

All of these factors conspire to produce in real problems, results only improve their accuracy at first-order (or worse), which means that double the mesh produces half the error. In other words, the accuracy is linearly proportional to the mesh spacing. This is a big deal as the second-order means that halving the mesh yields a four times reduction in error. Third-order would yield an eight times reduction. The reality is everything gives first-order accuracy or worse. The key for high-order working at all is that the high-order methods give a lower starting point for the error, which it sometimes does. The problem is that high-order methods are too expensive to justify the improvements they provide. The question is whether the benefits of practical accuracy can be achieved without incurring the costs typical for such methods.

Sometimes a clearly defined error is the only way to discover the truth

― Benjamin Wiker

The higher costs of the high-order methods are associated with a multitude of the characteristics of these methods. The basic steps associated with creating the high-order approximations use more data and involve many more operations than existing methods. If this wasn’t bad enough these methods often require a multiple of evaluations to integrate their approximations using quadratures. In cases of time-dependent methods, these methods often require more steps and require smaller time steps than the standard methods. To make matters even worse these methods are often not applicable to complex geometries associated with real problems. If you add on relative fragility and small gains in practical accuracy, you get the state of affairs we see today.

Restlessness is discontent — and discontent is the first necessity of progress. Show me a thoroughly satisfied man — and I will show you a failure.

― Thomas A. Edison

Meanwhile the theoretical and mathematical communities will tie themselves to high formal order of accuracy even when the methods are inefficient. The very communities that we should depend on to break this log jam are not motivated to deal with the actual problem. We are left in a lurch where no progress is being made toward improving the work horse methods in our codes.

To improve is to change; to be perfect is to change often.

― Winston S. Churchill

The cost part is almost a uniformly disappointing part of these methods most of which is dedicated to achieving formally high-order results. The irony is that the formal order of accuracy is immaterial to their practical and pragmatic utility. Almost no effort has been devoted to understanding how this cost accuracy dynamic can be negotiated. Without progress and understanding of these issues, the older methods, which now are standard will simply not move forward. Thus we had a great leap forward 25-30 years ago followed by stasis and stagnation.

Change almost never fails because it’s too early. It almost always fails because it’s too late.

― Seth Godin

Here are some “fun” research papers to read on these topics.

[Harten83] Harten, Ami. “High resolution schemes for hyperbolic conservation laws.”Journal of computational physics 49, no. 3 (1983): 357-393.

[HEOC87] Harten, Ami, Bjorn Engquist, Stanley Osher, and Sukumar R. Chakravarthy. “Uniformly high order accurate essentially non-oscillatory schemes, III.” Journal of computational physics 71, no. 2 (1987): 231-303.

[HHL76] Harten, Amiram, James M. Hyman, Peter D. Lax, and Barbara Keyfitz. “On finite‐difference approximations and entropy conditions for shocks.”Communications on pure and applied mathematics 29, no. 3 (1976): 297-322.

[Lax73] Lax, Peter D. Hyperbolic systems of conservation laws and the mathematical theory of shock waves. Vol. 11. SIAM, 1973.

[LW60] Lax, Peter, and Burton Wendroff. “Systems of conservation laws.”Communications on Pure and Applied mathematics 13, no. 2 (1960): 217-237.

[Boris71] Boris, Jay P., and David L. Book. “Flux-corrected transport. I. SHASTA, A fluid transport algorithm that works.” Journal of computational physics 11, no. 1 (1973): 38-69.

(Boris, Jay P. A Fluid Transport Algorithm that Works. No. NRL-MR-2357. NAVAL RESEARCH LAB WASHINGTON DC, 1971.)

[VanLeer73] van Leer, Bram. “Towards the ultimate conservative difference scheme I. The quest of monotonicity.” In Proceedings of the Third International Conference on Numerical Methods in Fluid Mechanics, pp. 163-168. Springer Berlin Heidelberg, 1973.

[Shu87] Shu, Chi-Wang. “TVB uniformly high-order schemes for conservation laws.”Mathematics of Computation 49, no. 179 (1987): 105-121.

[GLR07] Grinstein, Fernando F., Len G. Margolin, and William J. Rider, eds. Implicit large eddy simulation: computing turbulent fluid dynamics. Cambridge university press, 2007.

[RM05] Margolin, L. G., and W. J. Rider. “The design and construction of implicit LES models.” International journal for numerical methods in fluids 47, no. 10‐11 (2005): 1173-1179.

[MR02] Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821-841.

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

― Abraham Maslow

A collection of my thoughts on issues with modeling and simulation for the future with an emphasis on modeling. Without modeling improvements, the promise of simulation cannot be achieved. Our current approach to high performance computing focused on faster computers without balance is intellectually bankrupt. Without changes in our fundamental philosophy on modeling in computational simulation, the investments in hardware will yield little benefit for society.

A common point of view these days regards the existing code base as a massive investment to be preserved and ported to the new generation of computers. What is often not articulated is the antiquated nature of the modeling available in these codes. The approach used to model material has been in use for over 50 years and has quite a lot in common with a “textbook” approach to material description found in undergraduate courses in engineering. These undergraduate courses define the basic approach to analysis of systems, which are by their very nature macroscopic. Are these descriptions appropriate for highly detailed computer models? And at what point must the description change to reflect the physics of the scales being explored?

Computers allow a far more detailed description of these systems discretizing the macroscopic system into pieces that should resolve ever more of the microstructure and its response. The problem is that description of the materials is still almost isomorphic to the philosophy expressed in the undergraduate textbook. The material’s description is invariant all the way from the macroscopic view to scales that are clearly uncovering microscopic details. This highlights several clear scientific problems that extending the current code base to exascale computation will only exacerbate.

A key to moving forward is the recognition that the class of problems that can be simulated has grown. The older homogeneous average response modeling is still useful and valid, but only for a restricted class of problems. New capabilities and models will enrich simulation’s impact through providing avenues to new classes of problem solving. The new class of problems is defined by creating simulations that more faithfully provide the role of experiments and device testing. The simulations should be able to selectively probe the cases where off-normal response of devices arises. This will allow the analysis to assist in the determination of the limits of operation and safety for engineered systems.

1. At a macroscopic level, systems are not deterministic, yet the models we rely upon are. The models are exercised in an overly deterministic manner.
2. The material descriptions are invariant to the scales, which they are described at.
3. The questions answered by the codes do not match the asked about these systems any more.
4. A scientific vibrant field would not tolerate the level of inflexibility implied by current modeling practice. Vibrant science would demand that the models evolve to better match reality.

Addressing this set of issues is going to require deep scientific investigations, and perhaps more deep cultural evolution. We have a wealth of approaches investigating and solving problems associated with multiscale, multiphysics which bridge detailed microstructural simulation to macroscopic scales. The problem is that these approaches are not being utilized to solve the applied problems we currently tackle with our code base. None of these methods are being forced to displace the ancient techniques we rely upon today. As a result the state of practice is stuck in quicksand and remains static.

The importance of modeling as a driver for simulation capability should be obvious as well as its role as the essence of utility for the entire enterprise. This importance is not as obvious when looking at the balance of efforts in simulation science. For example no amount of accuracy, or computer power, or software quality can rescue a model that is inadequate or wrong. Only a focus on improving the model itself can rescue it. Today improving models is far down on the list of priorities for simulation despite its primal role in the quality of the enterprise. Nearby issues of solution methods and algorithms for model are also poorly funded. Most of the emphasis is tilted toward high performance computing and are implicitly predicated on the models themselves being correct.

Even if the models were judged to be correct, the advances in experimental science should be providing pressure to improve models. Improvements in detection, information and analysis will all yield ever better experimental measurements and access to uniquely innovative experimental investigations. These should provide a constant impetus to advance models beyond their current state. This tension is essential to the conduct of high quality science. If science is healthy there is a push and pull with theory and experiment where a theoretical advance will drive experiments, or new experimental observations will drive theory to explain. Without modeling being allowed to advance in response to experimental evidence, the fundamental engine of science is broken.

Furthermore the culture of analysis in engineering and science reinforces these approaches. First and foremost is the commitment to deterministic outcomes in simulation. Experimental science makes it very clear that our everyday macroscopic world has stochastic elements. There is a deterministic aspect to events, but the non-deterministic aspects are equally essential. By and large our analysis of experiments and simulations works steadfastly to remove the stochastic. Usually this is the adoption of averaging (or regression fits to data). These average properties or events then become the working model of our systems. In the past this approach allowed great progress, but more and more our engineered systems are defined more properly by the extremes of behavior that they can exhibit.

Our entire modeling approach especially that used in simulation are completely ill suited to address these extreme behaviors. A fundamental change in modeling and simulation philosophy is necessary to advance our understanding. Our models do not produce actual physically realizable simulations because not system actually acts like an average system everywhere. Instead the average behavior results from variations in behavior throughout the system. Sometimes these variations produce effects that are exactly associated with the newer questions being asked about extreme behavior.

The new methods do not displace the need for the old methods, indeed the new methods should appropriately limit to the solutions found by the old methods. The new methods allow the resolution of scale-dependent behavior, and off-average behavior of the system, but need to be self-consistent with traditional methods for simulation. Perhaps just as importantly, those conducting simulations should be deeply aware of when the old methods lose validity both in terms of scale-dependent behavior, and the questions being addressed through the simulation.

This brings the idea of questions to the forefront of the discussion. What questions are being addressed via simulation? There is a set of questions that older simulation methods are distinctly capable of answering. These questions are not the same questions driving the need for simulation capability today. In providing new models for simulation, the proper questions are primal in importance.

The current simulation capability is tied to answering old questions, which are valid today, but less important as new topics are crowding them out. Examples of the older questions are “what is the performance of this system under average conditions?” “What is the yield of this production process?” “How large is the average margin of performance beyond the requirements for the system?” With the key aspect of the questions being answered being the capacity of the modeling to attack the average properties and performance of engineered systems. By the same token the uncertainty we can assess via modeling today via simulation is the lack of knowledge about the average behavior of these systems, which is not the same as the uncertainty in the behavior of the actual system.

This mindset influences the experimental comparisons done for the purposes of validation as well. Experimental data is often processed into an average, and then compared to the simulation. No single experiment is appropriately simulated, but rather the simulation is modeling the average of the experiments. As such, the simulations are not truly modeling reality because for many physical systems, the average response of the system is never produced in a single experiment. As discussed below, this mindset infects the interpretation of experiments in a deeply pernicious manner.

The new questions being asked of simulations are subtly different, and require different models, methods, algorithms and codes to answer. Key among these questions is “how much variation in the behavior of the system can be expected?” “How often will the system manifest certain extreme behavior?” “How will the entire population of the system behave under certain conditions?” “What is the worst behavior to expect from the system and how likely is it to happen?”

Ideally, the calculation should be the same as observations from a physical experiment (validation), not the average of all experiments. In this way our simulations do not model any reality today because they are almost invariably too homogeneous and deterministic in character. Experiments, on the other hand, are heterogeneous and variable yielding some degree of stochastic response. Systems truly have both a variable stochastic character, which usually acts as a non-deterministic component around a major homogeneous and deterministic aspect of the system. Today our models are predominantly focused on the homogeneous, deterministic aspect of these systems. This aspect is the focus of traditional models and the older questions. The new questions are clearly focused on the secondary stochastic aspects that we average away today. The result is a strong tendency to treat single experiments inappropriately as instances of average response when they are simply a single instance from a population of possible experiments. When the deterministic calculation is forced to compare too closely to the non-deterministic aspects of an experiment, problems ensue.

Of course this decomposition is only approximate. For nonlinear systems the separation between stochastic and deterministic is dependent on the circumstances, and the nature of the system itself. Some instances of the system will yield a different decomposition because of the coupling of the system’s response to variability. Examples of the newer questions to be addressed by simulation abound in areas such as device engineering, stockpile stewardship and weather/climate modeling. For example, a key aspect of an engineered device is the portion of the population of devices that can be expected to fail under the extreme conditions associated with normal use. This may have significant reliability consequences and economic side effects. Similar questions are key in stockpile stewardship in part to address shortcomings in the degree of testing in the field or as populations of devices diminish and reduce statistical method’s effectiveness. Extreme weather events such a rain, wind or snowfall have extreme consequences on the mortality and economic impacts on society. The degree to which climate change causes an increase in such occurrences has significant policy consequences. Simulations are being relied up to an ever-greater degree to estimate this issue.

In many cases the modeling in our workhorse engineering analysis codes is quite recognizable from our undergraduate engineering textbooks. Rather than forming a distinct field of study as the modeling unveils more mesoscopic and ultimately microscopic details, the modeling is still couched in terms of the macroscopic methods used in classical desk modeling and calculations. The modeling does not account for the distinct aspects of being applied to a discretized system where smaller scales are available. Many of these models are clearly associated with average, steady state behavior of the full macroscopic system. The multiscale modeling is simply short-circuited by the traditional view of modeling defined in many codes. For example continuum codes for fluids, solids, heat transfer and mechanics all use uniform, homogenized properties for solving problems. The philosophy is virtually identical to the macroscopic material description that would be familiar to undergraduate engineering students.

This is madness! This was a reasonable fifty years ago as these methods first came into use and the number of computational elements was small and the elements were large. Today these methods are quite mature, and the number elements is huge and their size is clearly separated from the large scale. The scale separation dictates that a model that more properly describes the material at the scale of simulation overturns the homogenized models. A homogenized material can only describe the homogenized outcome, or the average solution for the material. Furthermore this homogeneous model will not match any actual circumstances from reality.

One of the key aspects of real experiments is the ever-present random component of results. The initial and boundary conditions all have a random uncontrolled variability that yields the variation in results. In homogenized simulations, this aspect of reality is washed out and for this reason the simulation is unrealizable in the real World. At times the random component is significant enough that the result of the experiment will radically depart from the average response. In these cases, however small in probability, the simulations fall completely short of serving to replace experiments and testing. This aspect of simulation is woefully lacking from current plans despite in centrality to the role of the simulation in providing a transformative scientific tool.

Another place where current simulation approaches fall demonstrably short of serving the modeling of reality are ideal models. These models are often mathematically beautiful evoking Hamiltonian structure and deep provable properties that breed devotion by the mathematically inclined. All of this simply detracts from the lack of physical reality bound up in this idealization. These models lack dissipative forces, which define the presence of the second law of thermodynamics, a necessary element for continua associated with reality. By too greatly focusing on the beauty and majesty of the ideal model, the primal focus of modeling reality is ultimately sacrificed. This is simply too great a price to pay for beauty. More perniciously the approach produces models with seemingly wonderful properties and rigor that seduce the unwary into modeling the World in utterly unphysical manners. In many cases the modeling is constructed as the solution to the ideal model plus an explicit model for the non-ideal effects. It should be a focus of modeling to assess whether the intrinsically unphysical aspects of the ideal model are polluting the objective of modeling reality.

In computing there is a chain of activities that provide value to the World. Modeling is the closest thing to reality. No amount of computing speed, algorithmic efficiency, and methodological accuracy can rescue a model that is inadequate. Once a model is defined in needs to be solved on the computer via a method. The method can be streamlined and made more efficient via algorithmic advances. Finally we must consider that all of these need to have software for implementation and as well as mapping to the computing hardware. At the end of the chain the computing hardware is dependent on everything above it for its capacity to impact our reality. Again, modeling is the absolute key to any value at all in simulation.

Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.

― Isaac Asimov

 

 

Linearity Breeds Contempt

—Peter Lax

A few weeks ago I went into my office and found a book waiting for me. It was one of the most pleasant surprises I’ve had at work for a long while, a biography of Peter Lax written by Reuben Hersh. Hersh is an emeritus professor of mathematics at the University of New Mexico (my alma mater), and a student of Lax at NYU. The book was a gift from my friend, Tim Trucano, who knew of my high regard and depth of appreciation for the work of Lax. I believe that Lax is one of the most important mathematicians of the 20^th Century and he embodies a spirit that is all too lacking from current mathematical work. It is Lax’s steadfast commitment and execution of great pure math as a vehicle to producing great applied math that the book explicitly reports and implicitly advertises. Lax never saw a divide in math between the two and complete compatibility between them.

The publisher is the American Mathematical Society (AMS) and the book is a wonderfully technical and personal account of the fascinating and influential life of Peter Lax. Hersh’s account goes far beyond the obvious public and professional impact of Lax into his personal life and family although these are colored greatly by the greatest events of the 20^th Century. Lax also has a deep connection to three themes in my own life: scientific computing, hyperbolic conservation laws and Los Alamos. He was a contributing member of the Manhattan Project despite being a corporal in the US Army and only 18 years old! Los Alamos and John von Neumann in particular had an immense influence on his life’s work with the fingerprints of that influence all over his greatest professional achievements.

In 1945 scientific computing was just being born having provided an early example in a simulation of the plutonium bomb the previous year. Von Neumann was a visionary in scientific computing having already created the first shock capturing method and realizing the necessity of tackling the solution of shock waves through numerical investigations. The first real computers were a twinkle in Von Neumann’s eye. Lax was exposed to these ideas and along with his mentors at New York University (NYU), Courant and Friedrichs, soon set out making his own contributions to the field. It is easily defensible to credit Lax as being one of the primary creators of the field, Computational Fluid Dynamics (CFD) along with Von Neumann and Frank Harlow. All of these men had a direct association with Los Alamos and access to computers, resources and applications that drove the creation of this area of study.

Lax’s work started with his thesis work at NYU, and continued with a year on staff at Los Alamos from 1949-1951. It is remarkable that upon leaving Los Alamos to take a professorship at NYU his vision of the future technical work in the area of shock waves and CFD had already achieved remarkable clarity of purpose and direction. He spent the next 20 years filling in all the details and laying the foundation for CFD for hyperbolic conservation laws across the world. He returned to Los Alamos every summer for a while and encouraged his students to do the same. He always felt that the applied environment should provide inspiration for mathematics and the problems studied by Los Alamos were weighty and important. Moreover he was a firm believer in the cause of the defense of the Country and its ideals. Surely this was a product of being driven from his native Hungary by the Nazis and their allies.

Lax also comes from a Hungarian heritage that provided some of the greatest minds of the 20^th Century with Von Neumann and Teller being standouts. Their immense intellectual gifts were driven Westward to America through the incalculable hatred and violence of the Nazis and their allies in World War 2. Ultimately, the United States benefited by providing these refugees sanctuary against the forces of hate and intolerance. This among other things led to the Nazis defeat and should provide an ample lesson regarding the values of tolerance and openness as a contrast.

The book closes with an overview of Lax’s major areas of technical achievement in a series of short essays. Lax received the Abel Prize for Mathematics in 2005 because of the depth and breath of his work in these areas. While hyperbolic conservation laws and CFD were foremost in his resume, he produced great mathematics in a number of other areas. In addition he provided continuous service to the NYU and United States in broader scientific leadership positions.

Before laying out these topics the book makes a special effort to describe Lax’s devotion to the creation of mathematics that is both pure and applied. In other words beautiful mathematics that stands toe to toe with any other pure math, but also has application to problems in the real world. He has an unwavering commitment to the idea that applied math should be good pure math too. The two are not in any way incompatible. Today too many mathematicians are keen to dismiss applied math as being a lesser topic and beneath pure math as a discipline.

This attitude is harmful to all of mathematics and the root of many deep problems in the field today. Mathematicians far and wide would be well-served to look to Lax as a shining example of how they should be thinking, solve problems, be of service and contribute to a better World.

…who may regard using finite differences as the last resort of a scoundrel that the theory of difference equations is a rather sophisticated affair, more sophisticated than the corresponding theory of partial differential equations.

—Peter Lax

After all, I believe that legends and myths are largely made of ‘truth’.

― J.R.R. Tolkien

For the purposes of this post, “Code” = “Modeling & Simulation tool: instead of the set of instructions in a programming language. Some codes are viewed as being better and more worthy of trust than others. The reasons for such distinctions are many and varied, but most often vague and clouded in mystery. I hope to shed some light on the topic.

Legend does not contradict history. It preserves the fundamental after but magnifies and embellishes it.

― Adrien Rouquette

Often codes become useful by simply being the first one to achieve success with a difficult and important problem. In other cases the authors of the code are responsible for the code’s mythic status. Certain authors of codes bring with them a pedigree of achievement that breeds confidence in the product of their work. This is computer instructions or code, which comprises an executable tool. It is a combination of a model of physical reality, a method to solve that model including algorithms developed to optimize the method, and auxiliary software that connects the code to the computer itself. Together with the practices of the users of the code, and options enabled by the code itself, the modeling capability is defined. This capability is then applied to problems of interest and success occurs if the comparisons with observations of reality are judged to be high quality.

Computers are good at following instructions, but not at reading your mind.

—Donald Knuth

What sort of things produces a code of legend and myth?

Myth must be kept alive. The people who can keep it alive are the artists of one kind or another.

― Joseph Campbell

1. The code allows new problems to be solved, or solved correctly. Often a new interface or setup capability is key to this capacity as well as new models of reality. This has the same dynamic as discovery does in other areas. Being first is an incredibly empowering aspect of work and often provides a de facto basis for success.
2. The code allows problems to be solved better than before by whatever standard is used by the community. Sometimes being first is not enough because the quality of solution isn’t good enough. The discovery ends up being delayed until the results are good enough to be useful. As such success is related to quality of results and the expectations or standards of a technical community.
3. The code solves a standing problem that hasn’t been solved before, or to a degree that instills confidence. Sometimes problems are acknowledged and result in being a standing challenge. When someone creates a tool that makes an effective solution to this sort of problem, it creates a “buzz” and provides the push the code needs for adoption more broadly.
4. The code is strongly associated with success in application space (quality by association). If the code is strongly associated with a successful application product, the code can inherit its virtue. Usually this sort of success will be strongly associated with an institution or national program (like ICF, inertial confined fusion). The codes success can persist for as long as the application’s success, or in some cases outlast it.
5. The code is reliable (robust) and produces useful results as a matter of course. For some areas in modeling and simulation codes are fragile, or too fragile to solve problems of interest. In such cases a code will make a breakthrough when the code simply runs problems to completion and the results are physically or conceptually plausible. Depending on the situation the lowly standard will then transition to other forms of success as the standards for solution improve,
6. The code produces physically reasonable solutions under difficult circumstances. This is a similar situation to the robustness virtue, but a bit better. Sometimes robustness is achieved through producing really terrible solutions (often very heavily diffused, or smeared out). This often destroys significant aspects of the solution. A better answer without such heavy-handed methods will yield code new followers who evangelize its use, or perhaps embarrass those holding onto the past.
7. The code is associated with someone with a pedigree such as an acknowledged trailblazer in a seminal field to the application or code specialties. This is praise by association. Someone who is a giant in a field will produce a wake of competence, which is almost completely associated with a cult of personality (or personal achievement).
   

   Frank Harlow with Jacob Fromm

8. The code’s methods are uniquely focused on the application problem area and not generalized beyond it. Sometimes a code is so focused in an important niche area that it dominates the analysis like no general-purpose code can. Often this means that the code caters to the basic needs of the analysis specifically and provides a basis of solution of application-specific problems that no general-purpose code can compete with.
9. The code solves a model of reality that no other code can. In other cases, the code has models no other code provides. These models can be enabling because standard models are not sufficient to explain reality (i.e., fail validation). The new model may require some unique methodology for its solution, which together with the model provide a distinct advantage.
10. The code is really fast compared to alternatives. For a lot of analysis questions it is important to able to run the code many times. Analysts like getting answers faster more than slower, and a quick turn-around time is viewed as a great benefit. If a code takes too long to get an answer, the ability to fully explore problems via parameter or scenario variation can be negatively impacted.
11. The code’s solutions are amenable to analysis or comparisons to observations are enabled. This has a lot more to do with the auxiliary analysis than the code itself. A code that has good visualization or data analysis built into its analysis system can provide significant encouragement for the use of the code itself.

12. The code produces results that are comfortable to the community, or define the standard for the community. Sometimes the code simply either meets or sets the expectations for the community using it for analysis. If it confirms what they tend to believe already, the analysts have greater comfort using the code.
13. The code’s methodology is comfortable to the community (and its intrinsic bias). For example the model and its solution are solved in a Lagrangian frame of reference, and the community only trusts Lagrangian frame solutions.

Storytellers seldom let facts get in the way of perpetuating a legend, although a few facts add seasoning and make the legend more believable.

― John H. Alexander

Sometimes a code has one or more of these items going for it. Once the code becomes used and trusted, it is the incumbent and it is very difficult to displace from usage. This is even true with unambiguously better methods. This is just a fact of life.

Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning.

— Rich Cook

 

 

 

 

 

 

 

The road to Hell is paved with the best of conscious intentions.

― Elizabeth F. Howell

Let me get to one of the key punch lines for this post, “no amount of mesh refinement, accuracy or computer speed can rescue an incorrect model.” The entire reason for doing modeling and simulation is impacting our understanding or response to the reality of the Universe. The only fix for a bad model is a better model. Better models are not something we are investing much effort in. This gets to a fundamental imbalance in high performance computing where progress is now expected to come almost purely through improvements in the performance of hardware.

Success doesn’t come to you; you go to it.

― T. Scott McLeod

If the field were functioning in a healthy manner, the dynamic would be fluid and flexible. Sometimes a new model would spur developments in methods, algorithms for its solution. This would ultimately spill down to software and hardware developments. The dynamic that is working today would also manifest itself in the need for improvements in software and hardware to allow for solutions of meaningful models. The issue at hand today is the 20 year history of emphasis on hardware and its inability to yield progress as promised. It is time to recognize that the current trajectory is imbalanced and needs significant alteration to achieve progress commensurate with what has been marketed to society at large.

There can be no ultimate statements science: there can be no statements in science which can not be tested, and therefore none which cannot in principle be refuted, by falsifying some of the conclusions which can be deduced from them.

― Karl Popper

Modeling and simulation has become an end unto itself and lost some of its connection to its real reason for being done. The reason we conduct modeling and simulation is to understand, explain or influence reality. All of science has the objective of both uncovering the truth of the Universe and allowing man to apply some measure of control to it. As most things become to those practicing an art, modeling and simulation is a deep field combining many disparate fields together toward its accomplishment. This depth allows practitioners to lose track of the real purpose, and focus on the conduct of science to exclusion of its application.

Science has an unfortunate habit of discovering information politicians don’t want to hear, largely because it has some bearing on reality.

― Stephen L. Burns

Why would any of this make a difference?

By losing sight of the reason for conducting an activity causes a loss of the capacity to best utilize the field to make a difference. Science has a method and its manner conduct is important to keep in mind. Computational science is a bridge between the reality of physics and engineering and the computers that enable it. The biggest issue is the loss of perspective on what really determines the quality of modeling and simulation. Our current trajectory is focused almost exclusively on the speed of the computer as the route to quality. We have lost the important perspective that no computer can save a lousy model. It just assures a more expensive, high fidelity wrong solution.

The quest for absolute certainty is an immature, if not infantile, trait of thinking.

― Herbert Feigl

The wrong solutions we are getting are not terrible, just limited. Science works properly when there is a creative tension between experiments and theory. Theory can be powered by computing allowing the solution of models impossible without it. Experiments must test these theories either by being utterly new, or employing better diagnostics. Without the experiment to test, confirm or deny, theory can rot from within essentially losing connection with reality. This fate is befalling our theories today by fiat. Our models are almost assumed to be correct and not subject to rigorous testing. More powerful computers are simply assumed to yield better answers. No impetus is present to refine or develop better models where all evidence points toward their utter necessity.

…if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated.

― Richard P. Feynman

Applied mathematics is a closely related field where the same slippage from reality is present. Again the utility of applied mathematics is distinctly like that of computing; it is utterly predicated upon the model’s quality visa-vis reality. In the period from World War 2 until around 1990, applied mathematics eagerly brought order and rigor to modeling, simulation and related activities. It became an able and honored partner for the advance and practice of science. Then it changed. It began to desire a deeper sense of mathematical honor as pure mathematics had in its eyes. In doing so applied math turned away from being applied and toward being governed by mathematical qualities. The lack of balance has emptied applied math’s capacity to advance science. The same has happened with computing. We are all poorer for it.

Science is not about making predictions or performing experiments. Science is about explaining.

― Bill Gaede

All of this may be overcome and the balance may be resurrected. All that is needed is to reconnect these fields with application. Application is a Gordian knot whose very nature powers science. Without the riddle and difficulty of application, the fields lose their vigor. The vigor is powered by attempting the solution of seemingly intractable problems. Without the continual injection of new ideas, the science cannot prosper. Such prosperity is currently being denied by a lack of connectivity to the very reality the fields discussed here could help to master. Such mastery is being denied by the lack of faith in our ability to take risks.

The intention (of an artist) is (the same as a scientist)…to discover and reveal what is unsuspected but significant in life.

― H W Leggett

Bad models abound in use today. A lot of them should be modified and discarded, but in today’s direction for scientific computing, we are simply claiming that a faster computer will open the door to solution. Many idealized equation sets are used in modeling that yield intrinsically unphysical solutions. The Euler equations without dissipation are a prime example. Plasma physics is yet another place where unphysical models are used because dissipation mechanisms are small. In macroscopic models dissipation is omnipresent, and leads to satisfaction of the second law of thermodynamic. Ideal equations remove this essential aspect of modeling by fiat.

The law that entropy always increases holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations — then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation — well, these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.

—Sir Arthur Stanley Eddington

In no place is this more greatly overloaded with context than turbulence. There is a misbegotten belief that solving the incompressible Navier-Stokes equations will unveil the secrets of turbulence. Incompressibility is fundamentally unphysical and may remove fundamental aspects of turbulence through its invocation. Incompressibility implies infinite speed of sound and a lack of thermodynamics. Connections between the incompressible and compressible equations only exist for adiabatic (dissipation-free) flows. Turbulence is characterized by dissipation in the absence of finite viscosity, which implies derivative singularities in the flow. Compressible fluids have this character and its nature is highly associated with the details of thermodynamics. Incompressible flows have not been clearly associated with this character, and the lack of thermodynamic is a likely source of this failing.

The plural of anecdote is not data.

― Marc Bekoff

Another aspect of our continuum modeling codes is the manner of describing material response. We tend to describe materials in a homogeneous manner that is we “paint” them into a physical region of the problem. All the aluminum in a problem will be described by the same constitutive laws without regard to the separation of scales between the computational mesh, and the physical scales in the material. This approach has been around for over 50 years and shows no signs of changing. It is actually long since past the time when this should have changed.

It is more Important to be of pure intention than of perfect action.

― Ilyas Kassam

The key is to apply the scientific method with rigor and vigor. Right now scientific computing has vacated its responsibility to apply the scientific method appropriately. Too often modeling and simulation are touted as being the third leg of science equal to theory and experiment. Modeling should always be beholding to experimental and field observation, and should the model be found to be in opposition to the observation, it must be found faulty. Modeling is rather an approach to more generally and broadly find solutions to theory. Thus theory can be extended to more nonlinear and complex models of reality. This should aid the ability of theory to describe the physical universe. Often simulation can act as a Laboratory for theory where suppositional theory can be tested for congruence with observation (computational astrophysics is a prime example of this).

Intention is one of the most powerful forces there is. What you mean when you do a thing will always determine the outcome. The law creates the world.

― Brenna Yovanoff

The bottom line is whether we are presently on a path that allows modeling and simulation to take its proper place in impacting reality, or explaining reality as part of the scientific method? I think the answer today is a clear and unequivocal, no. A combination of modern day political correctness regarding the power of computational hardware, over-selling of computing, fear and risk avoidance all lead to this. Each of these factors needs to be overcome to place us on the road to progress.

The tiniest of actions is always better than the boldest of intentions

― Robin Sharma

What needs to happen to make things better?

• Always connect the work in modeling and simulation to something the real world,
• Balance effort with the benefit of the world to the real world
• Find a way to give up on determinism to an appropriate degree, model the degree of variability seen in reality,
• Do not over emphasize the capacity of computational power to simply solve problems by fiat,
• Take risks especially risks that have a high chance for failure, but large payoffs,
• Allow glorious failure and reward risk-taking if done in a technically appropriate manner,
• New methods and algorithms provide potential for quantum improvements in efficiency and accuracy as well as the promise of new uses for computational models,
• No single aspect of modeling and simulation should be starved of attention as every part of this ecosystem must be healthy to achieve progress in predictive science,
• Stop settling for legacy models, methods and codes just because they are “good” enough focus on quality and excellence.

In the republic of mediocrity, genius is dangerous.

― Robert G. Ingersoll

What’s the “best” computer? By what criteria should a computer be judged? Best for what? Is it the fastest? Or the easiest to use? Or the most useful?

The most honest answer is probably the most useful, or impactful computer in how I live my life or work, so I’ll answer in that vein.

Have the courage to follow your heart and intuition. They somehow already know what you truly want to become. Everything else is secondary.

― Steve Jobs

Details matter, it’s worth waiting to get it right.

― Steve Jobs

If I had to answer honestly, it’s probably the latest computer I bought, my new iPhone 6. It’s an absolute marvel. It is easy to use and useful all at once. I have a vast array of applications to use, plus I can communicate with the entire World and access an entire World’s worth of information. I can access maps, find a place to eat lunch, take notes, access notes, find out the answer to questions, keep up with friends, and make new ones. It also allows me to listen to music either stored or via “radio”. It is so good that I am rarely without it. It helps me work out at the gym with an interval timer that I can program to develop unique tailored workouts. Anything that links to the “cloud” for data is even better because the data on the iPhone is the same as other platforms I use. The productivity and efficiency that I can work with is now simply stunning. The word awesome doesn’t quite do it justice. If you gave it to me ten years ago, I’d have thought aliens delivered the technology to humans.

We don’t get a chance to do that many things, and every one should be really excellent. Because this is our life.

― Steve Jobs

The fastest computer I have access to isn’t very good, or useful. It is just fast and really hard to use. In all honesty it is a complete horror show. For the most part this really fast computer is only good for crunching a lot of numbers in a terribly inefficient manner. It isn’t merely not a multi-purpose computer; it is single purpose computer that is quite poor at delivering that single purpose. Except for its speed it compares poorly to the supercomputers I used over 20 years ago. I say this noting that I am not prone to nostalgia at all. Generally I favor the modern over the past by a wide margin. This makes the assessment of modern supercomputing all the more damning.

Don’t be trapped by dogma — which is living with the results of other people’s thinking.

― Steve Jobs

Your time is limited, so don’t waste it living someone else’s life.

― Steve Jobs

Unlike the iPhone with its teeming modernity, the modern supercomputer is an ever more monstrous proposition with each passing year. Plans for future supercomputers are sure to create a new breed of monsters (think Godzilla, a good name for one of the machines!) that promise to consume energy like American consumers drunk on demonstrating their God-given right to excess. They also promise to be harder to use, less reliable, and nearly impossible to program. They might just be truly evil monsters in the making. The evil being done is primarily the loss of opportunity to make modeling and simulation match the hype.

Anything worth doing, is worth doing right.

― Hunter S. Thompson

It isn’t that the hyped vision of modeling and simulation as a third way for science is so flawed; it is our approach to achieving this vision that is so counter-productive. The vision is generally sound provided that the steps we took actually led to such an outcome. The overbearing emphasis on computing speed as the key path to producing a predictive modeling capability is fatally flawed. It is a path lacks the sort of checks and balances that science needs to succeed. A faulty model cannot predict reality regardless of how fast it executes on a computer, or how refined the computational “mesh” is. Algorithmic improvements can provide new applications, solve unsolved problems, and provide greater efficiency that pure computational speed cannot deliver.

It’s not like I’m all into nostalgia and history, it’s just that I can’t stand the way things are now

― Novala Takemoto

The current fastest computer certainly isn’t the best supercomputer ever built. That crown lies on the head of the Crays of the 70’s, 80’s and 90’s built by that genius Seymour Cray. In the form of the X-MP, Y-MP, C90 or Cray 2 the supercomputer reached its zenith. In relative terms these Crays were joys to use, and program. They were veritable iPhones compared to the rotary phones we produce today. At that time with an apex in functionality and utility for supercomputing massively parallel computing was born (i.e., the attack of the killer micros), and the measure of a supercomputer became speed above all else. Utility, and usefulness be damned. The fully integrated software-hardware solution found in a Cray Y-MP became a relic in the wake of the “need for speed”.

Study the past if you would define the future.

― Confucius

In a sense the modern trajectory of supercomputing is quintessentially American, bigger and faster is better by fiat. Excess and waste are virtues rather than flaw. Except the modern supercomputer it is not better, and not just because they don’t hold a candle to the old Crays. These computers just suck in so many ways; they are soulless and devoid of character. Moreover they are already a massive pain in the ass to use, and plans are afoot to make them even worse. The unrelenting priority of speed over utility is crushing. Terrible is the only path to speed, and terrible is coming with a tremendous cost too. When a colleague recently quipped that she would like to see us get a computer we actually wanted to use, I’m convinced that she had the older generation of Crays firmly in mind.

The future is already here – it’s just not evenly distributed.

― William Gibson

So, who are the geniuses that created this mess?

We have to go back to the mid-1990’s and the combination of computing and geopolitical issues that existed then. The path taken by the classic Cray supercomputers appeared to be running out of steam insofar as improving performance. The attack of the killer micros was defined as the path to continued growth in performance. Overall hardware functionality was effectively abandoned in favor of pure performance. The pure performance was only achieved in the case of benchmark problems that had little in common with actual applications. Performance on real application took a nosedive; a nosedive that the benchmark conveniently covered up. We still haven’t woken up to the reality.

Remembrance of things past is not necessarily the remembrance of things as they were.

― Marcel Proust

Geopolitically we saw the end of the Cold War including the cessation of nuclear
weapons’ testing. In the United Stated a program including high performance computing was sold as the alternative to nuclear testing (the ASCI program, now the ASC program). This program focused on computing power as the sole determinant of success. Every other aspect of computing became a veritable afterthought and was supported on a shoestring budget (modeling, methods, algorithms, and V&V). The result has been fast, unusable computers that deliver a pittance of their promised performance and a generation of codes with antiquated models and algorithms (written mostly in C++). We’ve been on this foolish path ever since to the extent that it’s become the politically correct and viable path going forward. We have lost a generation of potential scientific progress at the altar of this vacuous model for progress.

It shocks me how I wish for…what is lost and cannot come back.

― Sue Monk Kidd

Why do we choose this path when other more useful and rational approaches are available?

Risk aversion.

In the past forty some odd years we have as a society lost the ability to take risks even when the opportunity available is huge. The consequence of failure has become greater than the opportunity for success. In computing this trend has been powered by Moore’s law, the exponential growth in computing power over the course of the last 50 years (its not a law, just an observation). Under Moore’s law you just have to let time pass and computer performance will grow. It is a low-risk path to success.

When did the future switch from being a promise to being a threat?

― Chuck Palahniuk

Every other aspect of modeling and simulation entails far greater risk and opportunity to either fail, or fail to deliver in a predictable manner. Innovation in many areas critical to modeling and simulation are prone to episodic or quantum leaps in terms of capabilities (especially modeling and algorithms). These areas of potential innovationare also prone to failures where ideas simply don’t pan out. Without the failure you don’t have the breakthroughs hence the fatal nature of risk aversion. Integrated over decades of timid low-risk behavior we have the makings of a crisis. Our low-risk behavior has already created a fast immeasurable gulf in what we can do today versus what we should be doing today.

You realize that our mistrust of the future makes it hard to give up the past.

― Chuck Palahniuk

An aspirational goal for high performance computing would be the creation of a computing environment that meant as much for scientific work as my iPhone means for how I live my life. Today we are very far from that ideal. The key to the environment isn’t the speed of the hardware, but rather the utility of how the hardware is integrated with the needs of the user. In high performance computing the user needs to produce scientific results, which depend far more on the modeling’s fundamental character than the speed of the computer.

The future depends on what you do today.

― Mahatma Gandhi

 

I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

― Abraham Maslow

High performance computing is a big deal these days and may become a bigger deal very soon. It has become a new battleground for national supremacy. The United States will very likely soon commit to a new program for achieving progress in computing. This program by all accounts will be focused primarily on the computing hardware first, and then the system software that directly connects to this hardware. The goal will be the creation of a new generation of supercomputers that attempt to continue the growth of computing power into the next decade, and provide a path to “exascale”. I think it is past time to ask, “do we have the right priorities?” “Is this goal important and worthy of achieving?”

Lack of direction, not lack of time, is the problem. We all have twenty-four hour days.

― Zig Ziglar

I’ll return to these two questions at the end, but first I’d like to touch on an essential concept in high performance computing, scaling. Scaling is a big deal, it measures success in computing, in a nutshell describes efficiency of solving problems particular with respect to changing problem size or computing resource. In scientific computing one of the primary assumptions is that bigger faster computers yield better, more accurate results that have greater relevance to the real world. The success of computing depends on scaling and breakthroughs in achieving it, defines the sort of problems that could be solved.

Nothing is less productive than to make more efficient what should not be done at all.

― Peter Drucker

There are several types of scaling with distinctly different character. Lately the dominant scaling in computing has been associated with parallel computing performance. Originally the focus was on strong scaling, which is defined by the ability of greater computing resources to solve a problem of fixed size faster. In other words perfect strong scaling would result from solving a problem twice as fast with two CPUs than with one CPU.

Lately this has been replaced by weak scaling where the problem size is adjusted along with the resource. The goal is to solve a problem that is twice as big with two CPUs just as fast as the original problem is solved with one CPU. These scaling results depend both on the software implementation and the quality of the hardware. They are the stock and trade of success in the currently envisioned high performance-computing program nationally. They are also both relatively unimportant and poor measures of the power of computing to solve scientific problems.

Two things are infinite: the universe and human stupidity; and I’m not sure about the universe.

― Albert Einstein

Algorithmic scaling is another form of scaling and it is massive in its power. We are failing to measure, invest and utilize it in moving forward in computing nationally. The gains to be made through algorithmic scaling will almost certainly lay waste to anything that computing hardware will deliver. It isn’t that hardware investments aren’t necessary; they are simply grossly over-emphasized to a harmful degree.

The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom.

― Isaac Asimov

The archetype of algorithmic scaling is sorting a list, which is an amazingly common and important function for a computer program. Common sorting algorithms are things like insertion, or quick-sort, and each comes with a scaling for the memory required and the number of operations to work to completion. In most cases the best that can be done is linear scaling, in other words for a list that is items long, it takes order operations. This means that for a sufficiently large list the cost is proportional to some constant times the length of the list, . Other high-grade algorithms like quicksort take order , but may carry a smaller constant. These can be faster for shorter lists. If one chooses very poorly the sorting can scale like . There are alsoaspects of the algorithm and it’s scaling that speak to the memory-storage needed and the complexity of the algorithm’s implementation. These themes carry on to a discussion of more esoteric computational science algorithms next.

In scientific computing two categories of algorithm loom large over substantial swaths of the field: numerical linear algebra, and discretization-methods. Both of these categories have important scaling relations associated with their use that have a huge impact on the efficiency of solution. We have not been paying much attention at all to the efficiencies possible from these areas. Improvements in both areas could yield improvements in performance that would put any new computer to shame.

For numerical linear algebra the issue is the cost of solving the matrix problem with respect to the number of equations. For the simplest view of the problem one uses a naïve method like Gaussian elimination (or LU decomposition), which scales like where is the number of equations to be solved. This method is designed to solve a dense matrix where there are few non-zero entries. In scientific computing the matrices are typically “sparse” meaning most entries are zero. An algorithm specifically for sparse matrices lowers the scaling to . These methods both produce “exact” solutions to the system (modulo poorly conditioned problems).

If an approximate solution is desired or useful one can use lower cost iterative methods. The simplest methods like the Jacobi or Gauss-Seidel iteration also scale at . Modern iterative methods are based on the Krylov subspace with the conjugate gradient method being the classical method. As exact solutions these methods scale as , but as iterative methods for approximate solutions the scaling lowers to $N^\frac{3}{2}$. One can do even better with multigrid methods, lowering the scaling to .

Each of this sequence of methods has a constant in front of the scaling, and the constant gets larger as scaling gets better. Nonetheless it is easy to see that if you’re solving a billion unknowns the difference between and is immense, a billion billion. The difference in constants between the two methods is several thousand. In the long run multigrid wins. One might even do better than multigrid with current research in data analysis producing sublinear algorithms for large-scale data analysis. Another issue is the difficulty of making multigrid work in parallel, as the method is inherently NOT parallel in important parts. Multigrid performance is also not robust and Krylov subspace methods still dominate actual use.

Learn from yesterday, live for today, hope for tomorrow. The important thing is to not stop questioning.

― Albert Einstein

Discretization can provide even great wins. If a problem is amenable to high-order accuracy, a higher order method will unequivocally win over a low-order method. The problem is that most practical problems you can get paid to solve don’t have this property. In almost every case the solution will converge at a first-order accuracy. This is the nature of the world. The knee-jerk response is that this means that high-order methods are not useful. This shows a lack of understanding on what they bring to the table and how they scale. High-order methods produce lower errors than low-order methods even when high-order accuracy cannot be achieved.

As a simple example take a high-order method that delivers half the error of a low order method. To get equivalent results the high-order method would take half the mesh defined by a number of “cell” or “elements” per dimension . If one is interested in time-dependent problems, the number of time steps is usually proportional to . Hence a one-dimensional problem would require degrees of freedom. For equivalent accuracy the high-order method would require cells and one-fourth of the degrees of freedom. It breaks even at four times the cost. In three dimensional time dependent problems, the scaling is and the break-even point is 16 in cost. This is imminently doable. Even larger improvements in accuracy would provide an even more insurmountable advantage.

The counter-point to these methods is their computational cost and complexity. The second issue is their fragility, which can be recast as their robustness or stability in the face of real problems. Still their performance gains are sufficient to amortize the costs given the vast magnitude of the accuracy gains and effective scaling.

An expert is a person who has made all the mistakes that can be made in a very narrow field.

― Niels Bohr

The last issue to touch upon is the need to make algorithms robust, which is just another word for stable. Work on stability of algorithms is simply not happening these days. Part of the consequence is a lack of progress. For example one way to view the lack of ability of multigrid to dominate numerical linear algebra is its lack of robustness (stability). The same thing holds for high-order discretizations, which are typically not as robust or stable as low order ones. As a result low-order methods dominate scientific computing. For algorithms to prosper work on stability and robustness needs to be part of the recipe.

If we knew what it was we were doing, it would not be called research, would it?

― Albert Einstein

Performance is a monotonically sucking function of time. Our current approach to HPC will not help matters, and effectively ignores the ability of algorithms to make things better. So “do we have the right priorities?” and “is this goal (of computing supremacy) important and worthy of achieving?” The answers are an unqualified NO and a qualified YES. The goal of computing dominance and supremacy is certainly worth achieving, but having the fastest computer will absolutely not get us there. It is neither necessary, nor sufficient for success.

This gets to the issue of priorities directly. Our current program is so intellectually bankrupt as to be comical, and reflects a starkly superficial thinking that ignores the sort of facts staring them directly in the face such as the evidence of commercial computing. Computing matters because of how it impacts the real world we live it. This means the applications of computing matter most of all. In the approach to computing taken today the applications are taken completely for granted, and reality is a mere afterthought.

Any sufficiently advanced technology is indistinguishable from magic.

― Arthur C. Clarke