“Experiment is the sole source of truth. It alone can teach us something new; it alone can give us certainty.” ― Henri Poincaré

“What we observe is not nature itself, but nature exposed to our method of questioning.” ― Werner Heisenberg,

There is a real tension existing between these two quotes, truth and our perception of truth. Experiments are the key to science including theory where it is tested and ideas are prodded from our heads based on what we observe. At the same time our observations are imperfect and biased. In moving science forward both concepts are key to progress and keeping things in perspective. For the person engaged in computational science the challenge is uniquely fraught with conflict. This includes the new concept of computational experiments and their rightful role in advancing knowledge. Their perspective is undoubtably useful although having an artificial view of reality taking the role of “truth” is largely inappropriate. That said, the “truth” of experimental observations is also an illusion to the extent that observations are flawed as well; however these flaws are of an entirely different sort than simulation’s flaws.

Observations are flawed by our ability to correctly sense reality, or the distortions made through our means of detection, or the outright changes to reality made through our attempts to observe something. Simulations largely do not suffer from these issues in that we can perfectly observe them, but instead the reality we observe through simulation is itself intrinsically flawed. On the one hand we have a flawed view of the truth, and on the other we have a flawed truth with perfect vision. The key is that neither is perfect, and that both are useful.

Science is fundamentally predicated on experiments. Experiments are the engine of discovery and credibility. Not all experiments serve the same intent, nor should they follow the same protocols. There are many different types of experiments and it is useful to develop taxonomy of experiments to keep things organized. Ultimately since we all want to be better scientists, it might just help us do better science.

The classic experiment is the test of a hypothesis and it still holds the center of any discussion of science. Every other kind of experiment is a subset of this kind, but it useful to enrich the discussion with other experiments types. The differing types of experiments are constructed with a particular end in mind, and with that end in mind the choice to emphasize different qualities can be made. A key example is the notion of a specific validation experiment where the goal is to primarily provide data for ascertaining the credibility of computational simulations.

Measurement is the key to experiments. Measurement is by its very nature imprecise, we cannot exactly measure everything. Moreover, we don’t necessarily measure the right things. Often what we choose to measure is guided by theory, and if the theory is too flawed, we may not measure the important things. In other cases we simply cannot measure what is really important. In other words, the core of measurement is error. We need to be very exacting in our analysis of how much error is associated with an experimental measurement. Too often we aren’t very clear about this. For example, some experiments measure a quantity that actually fluctuates. The tendency is to report the mean value measured, and then some statistical measure of variation like the standard deviation. Rarely, if ever, the statistical choices made by the experimental analysis are justified. Does the quantity actually fall into a normal distribution? In spite of the fluctuations what is the experimental measurement error? Is this error biased?

Replicate experiments are another area where far too few examples exist. Experiments are often complex and expensive. In addition they are not repeatable, nor are they repeated. This results in certain uncertainties being completely unknown. Or to take the famous Donald Rumsfeld quip, the repeatability becomes a known unknown that is willfully unexplored. Usually the temptation to do a different experiment is too great to overcome. In this case any statistical evidence simply does not exist even though many of these cases are extremely sensitive to the initial conditions. If one is looking at a system described by a well-posed initial value problem and the initial conditions are impeccably well described, a single experiment might be justified. If all of this does not hold, the single experiment is outright dangerous. For complex systems the situation where the experiment is demonstrably repeatable does not usually present itself. An archetype of the sort of experiment that is not repeatable is the Earth’s climate, and in this case we have no choice.

Discovery experiments are where science most classically lives, or at least it is the ideal. A scientist makes a hypothesis about something, and an experiment is devised to test it. If the experiment and related measurements are good enough, a result is produced. The hypothesis is either confirmed (or no evidence against it), or it is disproven. These experiments are in fact far and few between, but when they can be done (correctly) they are awesome.

Computational experiments are a modern invention, and rightly the source of great controversy. I’d argue strongly they should be even more controversial than they are generally characterized. Generically, a computer code is a model (or a hypothesis) and a problem can be devised based on the model. Calculations can then be done to test the given hypothesis. The problem most succinctly is that the computational experiments are not proofs in the same sense as a physical experiment. Just as physical experiments have measurement error, computational experiments have computational error, but they also have more problems. The model itself may not be correct, or incomplete. The data used by the code may be incorrect or the experiment may be set up in flawed manner. Because of the artificial nature of the computational experiment, the whole enterprise is subject to an extra level of scrutiny. If such scrutiny produces evidence of correctness, the experiment can be taken more seriously, but rarely as seriously as the physical experiment. The benefit of computation is that it is more flexible than nature and most often much cheaper or less dangerous.

Often the statement is made that the computation is a “direct numerical simulation (DNS)” or “first-principles”. Very rarely is this statement actually justified or supported by any evidence. Most often it is false. These labels seem to be an excuse to avoid doing any analysis of the errors associated with the calculation, or worse yet claim they are small and unimportant without the slightest amount of justification. This is proof by authority, and it ultimately harms the conduct of science. If one is claiming to do DNS then the burden of proof should be very high. To be blunt, the use of DNS usually is offered with even less proof than admittedly cruder approximations. This isn’t to day that DNS should not be employed as a scientific tool, but rather its application should be taken with a rather large grain of salt. Scientists should demand more evidence of quality from a proposed DNS, and reject its results if such evidence is not provided. Doing anything less threatens both science in general, and poses an existential risk to computational science.

The concept of validation experiments is a new “invention,” or more properly a refinement on the basic concepts in experimental science. The primary purpose of these experiments is the validation of computer simulations. A simple-minded view would say that any other experiment would serve this purpose. The simple-minded view is correct, but this purpose is served poorly by classic experiments and the standards of reporting results. More importantly, many essential details for a successful simulation of the experiment are left out of the description. The definition of the experiments is more complete in the sense of providing key details for a high fidelity simulation of the precise experimental setup. Usual experimental science often leaves out many details that can cloud the sense of validation received by comparison, or at the very least offer substantial uncertainty as to the source of any discrepancies.

The point of this discussion isn’t to over-complicate things, but rather clarify differing intent for experiments. One simply doesn’t “experiment” for the same reasons, but rather many different reasons. The texture of the distinction can help provide a better environment for focus on why things are done and where the emphasis should be. Exploring a scientific hypothesis in the classical sense is different than validating a computer code. These differing purposes call for a refinement of emphasis in the conduct of the experiment. I will note that validation is a form of hypothesis testing, i.e., “is a computer simulation a representation of reality and to what degree and purpose can it be trusted?”   Computational experiments are another problem altogether, and require even greater attention to detail.

This is the 10^th edition of the conference and the first one in Europe. It is a mix of astrophysicists, plasma physicists, particle physicists, experimental physicists and a handful of nuclear engineers (like me).

This week was spent at a conference generally outside my field. Doing this is a mixed bag, great at being exposed to new things and new people, but often being way out of my depth. I had been invited to give a talk about V&V, as calculations are very important in both astrophysics and high energy density experiments. While calculations are important, the physicists’ mode of investigation almost seems to be intrinsically at odds with V&V. While outside my field I felt increasingly like I had gone back in time as the week of talks unfolded. I felt like the community acted like the classical Los Alamos physics community I had come to understand while working there. I came away thinking that they need more V&V, but not in the same way applied programs need it. Interactions here are likely to be instructive for the subtleties to be found elsewhere.

Physicists are highly motivated to study the impact of the completeness of the model of reality to such a degree that it inhibits virtually any attention to verification. Validation in a loose sense is the focus, but it tends to take on an ad hoc character, as models are changed whole cloth during a computational investigation while seeking to see the fidelity of the modeling to observations. These are essentially sensitivity studies, but the practice that is accepted is very ad hoc and lacks a certain systematic flavor. More commonly, the whole study embodies a curiosity driven approach. Perhaps, this is generally OK; however, some of the calculations left me feeling very uneasy.

HEDLA is involved with doing a broad spectrum of experimental work with astrophysical significance. A host of phenomena can be profitably examined using the modern facilities in high energy density physics. These facilities include laser fusion centers (NIF, Rochester, LMJ,…) and electromagnetic centers (magpie and Z). The topics to study are dynamics such as jets, radiating shock waves, material characterization and equation of state, and so on. The goal is to understand the physics in a more controlled environment than the pure observational environment of astronomy. The problem with the approach is the difficulty of measuring quantities in the environment offered by the experiments, which is generally very very hot and very small. The other opportunity is the more direct validation of the physical models available in the computer codes developed? These codes share the dual role of providing design and analysis of the experiments and exploring astrophysical theories and concepts.

This issue with combining astrophysics with experimental physics isn’t the quality of the science. The science in this community is strong, exploratory and interesting. The problem is that the experiments are hard to do, hard to diagnose and painfully expensive. Under these conditions the curiosity-driven approach to science becomes problematic. Experiments need to be carefully designed and the quantitative aspects of the work grow in priority. It clashes with the more qualitative mode of investigation that dominates astrophysics where the key is to understand the basic principles governing observed phenomena. An example is the sensitivity of dependence to initial conditions where the experiments could provide a measure of repeatability except replicate experiments are never done; they are beaten out by more interesting unique experiments. This is in spite of the replication issue being capable of addressing the true error bar for every experiment that is done.

Take for example core collapse supernova where computation has played a major role in understanding what is probably happening. Early on pure hydrodynamic simulations could not recover the behavior apparent from observations (the mixing of elements into the envelope of the exploding star). Adding multiple physical effects has provided a better qualitative picture of what is likely to be happening. When the simulations added asymmetry in the initial conditions, neutrino transport with coupling to the hydro, magnetic fields and rotation every thing became better. Suddenly the character of the simulations became much more like the observations. The issue of initial conditions comes up in spades here. Supernovas are difficult to make explode, and the question remains about how often there are duds that don’t explode. We really only see the supernovas that explode, the duds may happen, but we don’t see them.

The question is whether this successful approach can be used for very expensive experimental design and analysis. I’m not so sure.

Using codes in conjunction with expensive, complex experiments should naturally evoke refined V&V. V&V is natural in the sort of engineering uses of computation that experimental design engenders. Conversely V&V seems to be almost unnatural for physics investigations. V&V implies a certain stability of modeling and theory that this field does not have. The careful and complete investigation of a stable model in an anathema to open-ended physics investigations. In other words the places where V&V is well grounded and natural are exactly the areas where the physics research community isn’t interested in. So the key is to craft a path forward that at once provides better quality of simulation for high energy density physics without clashing with the sorts of investigations important to the vibrancy of the community.

In a strong sense I think this is a perfect example for the flexible approach to V&V I’ve been advocating. In essence the idea is to apply V&V in a limited and carefully defined manner crafted to the needs of the community. The codes should probably have a greater level of foundational V&V in terms of the implementation of the basic numerical methods and physical models. Beyond the foundational V&V the application specific V&V should be far greater when the codes are applied to experimental design and analysis to assure that the outcomes of the experimental work has sufficient value. On the other hand the hard-nosed V&V concepts are inappropriate for the curiosity-driven astrophysics investigations. This isn’t that they couldn’t be applied, but rather that they would be potentially counter-productive. Once a mechanism is well enough established to transition to an experimental study, more V&V should kick in.

 We also visited the French version of NIF, the LMJ, which is a CEA run facility. We had a wonderful tour and since I saw NIF a year or so ago, it was useful to compare notes. Mostly the facility is similar, but seems more austere and less boastful. It is lower power and probably consciously avoids the word “ignition”. Interestingly the facility is still being constructed, but overall looks quite a bit like NIF (minus the landscaping, façade and other window dressing). The French are much more transparent about the connection of LMJ to their defense work. In addition the tour was dramatically more technical (although they probably have a smaller number of visitors by a lot).

 Overall it was a good experience and gave me lots to think about. V&V should connect all the way from engineering and a heavy hand to physicists and a much lighter touch. Wherever codes are used seriously in design and analysis V&V should play some role, even if it is minor. After my talk I met a blogger attending the meeting (Adam Frank who blogs at http://www.npr.org/blogs/13.7/). He asked me the question about V&V and climate change. It was a good question that led to a much longer discussion. In a nutshell my opinion climate change needs to have serious discussion about V&V issues, but the atmosphere is so poisonous toward dialog that will never happen. One should be able to criticize how climate science is done without being labeled a denier. Right now, that cannot happen, and we are all poorer for it.

Verification as usually defined for computational science is an important, but confusing aspect of simulation. The core concepts of verification are three-fold:

• Does a simulation converge?
• What is the rate of this convergence?
• And what is the magnitude of numerical error?

Answering these questions provides evidence of the quality of the numerical solution. This is an important component of the overall quality of simulation. Scientists who tend to focus on the fidelity of the physical modeling often systematically overlook this aspect of the overall quality in simulation.

V&V often comes with a confusing “word soup” defining important terms. This “word soup” may be at its worst in verification. Verification and validation mean the same thing within a non-technical context, but in the framing of simulation quality they have quite specific technical meanings. So that the overall simulation quality can be assessed and understood, the activities surrounding each are distinctly different. The pithy statement of what the two words mean is useful; verification is the determination of whether the model is being solved correctly, and validation is the determination of whether the model is correct. Each involves the accumulation of evidence that this correctness is present.

Scientists tend to focus on the correctness of the model itself. The determination of correctness of the solution of the model is a mathematical problem involving basic numerical analysis. Validation necessarily involves observational or experimental data, and its comparison to the simulation. A necessary observation is that validation involves several error modes that color any comparison: the size of the numerical error in solving the model, and the magnitude of the experimental or observational error. Too often, one or both of these are overlooked. For high quality work, both of these must be accounted for in the assessment of model correctness.

For the numerical error, verification is used, but this differs from the verification process used to determine the correctness of the model solution. Thus, the distinction in verification is made between its two uses. Code verification is the process of determining model solution correctness and necessarily involves comparison of numerical solutions with analytical solutions that are unambiguously correct. For the purpose of error estimation several procedures may be used, but solution (or calculation) verification is perhaps the most convincing methodology.

A big issue is the implementation of verification is the confounding definitions and purpose of verification. In this vein the outcomes of the different forms of verification focus on differing metrics. I am going to try to address these confounding definitions and idiosyncrasies clearly.

For code verification and the determination of implementation and solution procedure correctness, the key metric is the rate of convergence. This rate of convergence is then compared with the analysis of the formal order of accuracy for the method being tested. If the solution to the problem is sufficiently smooth, the computed order of accuracy should closely match the order of accuracy from the numerical analysis as the mesh density becomes high.

In addition, the magnitude of the error is available in code verification. The code and computational physics communities systematically overlook the utilization in practical terms of the error magnitude in code verification. This aspect could be used to great effect in determining the efficacy of numerical methods. The determination of order of accuracy and error magnitude is not limited to smooth solutions. If solutions are discontinuous, the convergence rate and error magnitude are usually completely overlooked. Comparisons between the analytical solution and the numerical result are limited to the viewgraph or eyeball norm. This is a mistake and a missed opportunity to discuss the impact of numerical methods. Most practical problems have various forms of discontinuous behavior, and the magnitude of error for these problems define the efficiency of the method.

Solution verification is important for estimating the numerical error in applied simulation. No analytical solution exists in these cases, and the goal is two fold: determine whether the model is converging toward a mesh independent solution, and the magnitude of the error. Often scientists will show a couple of mesh solutions to assess whether the solution is sensitive to the mesh resolution. This is better than nothing, but only just. This does not provide the key property of the magnitude of numerical error in the solution. The error magnitude is a function of the mesh resolution; different mesh resolution has different error magnitudes (for a convergent simulation). An auxiliary quantity of interest is the rate of convergence, but the error magnitude is the primary metric of interest.

Lastly, the expectations for the rate of convergence are not often clearly enough stated. Concisely, the rate of convergence is a function of the details of the numerical method, and the nature of the solution. This is true for both code and solution verification. If the solution does not possess sufficient smoothness (regularity) or certain degenerate features, the convergence rate will deviate from the design order of accuracy, which a numerical method can achieve under ideal circumstances. Typically, the observed convergence rate is expected to be the minimum of the design order of accuracy and the solution regularity.

If instead, another error estimation procedure is utilized (such as adjoint methods, PDE-based methods, Z-Z, etc…), there is a secondary burden for the simulation code to address. In these cases the error estimation itself needs to be verified (code verified using analytical error estimate comparison, and solution verification comparison for applied use). I have rarely observed the successful use of verification for such estimation procedures.

Finally, I’ll mention the concerns I have about commercial CFD codes, or codes downloaded and used without detailed knowledge of the solution procedures therein. In the vast majority of cases these codes do not have a well-evidenced pedigree. The codes often report to having a good pedigree, but the evidence of that pedigree is sorely lacking. Those writing, selling and distributing these codes rarely provide the necessary evidence to have good faith in such codes. This lack of evidenced pedigree and deep knowledge of the solution procedures greatly limits the effective estimation of numerical error when using such codes.

This week I’m traveling all week (and next week too). Traveling is a mixed bag; I’m seeing new things, and exposed to new ideas; I’m away from home and family. It’s a yin and yang sort of thing. I love it, and I hate it. Today, I’ll focus on the good part, the part that keeps my brain humming along. Without mixing up whom you interact with your ideas may become stale, or fail to consider issues that come from a differing perspective. You also limit your ability to contribute to wider debates, and learn about new ideas developed elsewhere. You need it for intellectual vibrancy.

First, I went to visit the University of Notre Dame where DOE has funded a center to study a really cool material processing idea. The physical idea itself is fascinating involving reacting shock waves and the production of exotic materials. At the university they are combining disciplines to design and predict the experimental results. The process has never been executed before, but they believe it can be if they get everything right. They are using computational physics, multiscale modeling and advanced computing ideas along with the laboratory work. The key is engagement across areas that often don’t mix in an academic setting. This mixing is itself a challenge at a university where the tendency is for work to be done in narrow-deep silos of knowledge. I’m there leading a team of National Lab scientists who are there to help and advise the Center as well as make sure the work stays aligned with things the Lab values. Key among those values is multi-disciplinary work requiring disparate skills for success. This part was “easy” since it was the first two days of the two-week trip, and full of cool mind-expanding ideas. The hard part was having to “on” for the entirety of the meeting as the chairman of the committee.

The last three days have been spent at the third ASME V&V Symposium. The meeting has a real sense of being important to the V&V community. There seems to be a nexus at hand where the field will either grow and flourish, or decay and die. Opportunity lies to the right, and danger lies to the left; or is it the other way around. Decisions are being made that are important such as whether to have a V&V journal; how V&V figures into the regulation of medical devices and other aspects of the biological sciences. The danger with a Journal is that it becomes a cul-de-sac where all V&V work is done, and offers an excuse for not applying V&V in application areas. The distinction is important, and the spectrum of detailed V&V work needs to run from the development of methodology, to its application in purely applied work. For V&V to flourish, it needs both theoretical work and committed application in domain science and engineering. Regulatory work offers similar, but different challenges. The tension exists between strong regulation of action where people do things “right” and too much rigidity to allow for innovations in scientific methods to continue to benefit the quality of work.

For me, there was a big event. I was giving a plenary talk the second morning of the meeting. My talk’s abstract was published on the blog earlier, “What kind of person does V&V?” and in retrospect it drew attention appropriately to the talk. It also raised expectations. I had to deliver. Part of the talk was a desire to work on my own approach to giving a talk. In other words, I was pushing myself. I integrated elements of “Presentation Zen” and TED talks into the presentation. These elements meant using more images and fewer words while focusing on stories and free flowing narratives to give the message. My intent was to provoke thought and self-reflection in the V&V community. The talk was crafted into five narrative arcs some of which have been posted here. The first of these was the analogies between V&V and Human Resources, next I spoke about the danger of technology being too easy or simple (V&V is technology), the third is the use of “you idiot” appended to questions to screen bad questions out, followed by imploring V&V to act more as coaches and less as referees.

The final story revolved around the ongoing revolution in computing and data science coupled with the end of Moore’s law. V&V is needed to help manage the transition in computational science that will occur in the next decade. Without V&V computational science might be lost or go seriously off the rails. Without Moore’s law in effect we no longer have bigger, faster computers to simply crush problems with resolution. Instead, we will have to rely upon being smarter and improving methods, but new methods produce different answers, and V&V is necessary to build trust and confidence in the new methodology. Furthermore, the direction that computing is going offers the possibility to leverage the technology in a myriad of creative ways. V&V is core to success in many of these opportunities.

I felt very good at the end, and the objectives of the talk seemed to be generally achieved. I was imploring V&V to be collaborative, flexible and emotionally intelligent. As I’ve discovered if you give a good talk you’ll get questions and comments. People will want to talk to you. I received both in spades.

I’ll draw to a close with few observations about the meeting. The topic of V&V is quickly maturing. The practice of V&V is improving across the board. More and more talks are getting at subtleties in the practice rather than the basics. This is clearest in the new biological science use of V&V and particular medical device modeling. The quality of the work is much better than the previous year, and in fact the pace of improvement is astounding especially compared to the physical sciences that birthed V&V. A deeper concern is the penetration of V&V into the application sciences. Is V&V being done better and more extensively where it is needed in engineering practice? Or key scientific endeavors such as climate science. Climate science is politically charged and needs V&V, but the sensitivity to criticism acts to effectively poison any V&V despite the magnitude of the need. Pat Roache compared the discussions the climate community has about quality to a couple having a hushed argument behind closed doors fearing their children might overhear. There is a distinct lack of domain science expertise at the meeting, and that is a major concern.

I will also mention the extensive use of commercial codes as a concern. The verification work on many of these codes is not up to standards. I can’t see it doesn’t exist, I just haven’t seen it. As many have noted V&V is evidence based, and the evidence isn’t there. The use of commercial codes for CFD is extensive. Laboratory and government built or company internal software is now dominated by commercially sold CFD codes. The actual quality of these codes is difficult to understand, and they aren’t very open to discussing their “secrets”. In the end we need better transparency so that the solutions they create can be trusted. Like many applied codes the “robustness” of the code takes precedent over accuracy. As such low order numerical methods are highly favored. Another concern is the utilization of antiquated and simple methods for multiphysics coupling.

The good thing about the meeting was seeing lots of old friends, great conversations and a whole lot to think about when I get home. People who manage science federally and have restricted our attendance at meetings fail to understand what is important about conferences. Yes, we present our work to our peers, and our peers give us feedback, but much more happens. First and foremost, we see the work our peers are doing. We engage people socially, and we laugh, argue and eat together. The social aspects of science are critical to a well-functioning activity. Conferences are essential to the conduct of science because they allow people to interact as people. Presenting a paper at a conference is only one small aspect of a much broader engagement.

Next week I’m in France and seeing what the application folks in high energy density physics are up to. I have my hopes and have my fears.

“It is not the answer that enlightens, but the question.” – Decouvertes

Research is all about answering questions. The nature and quality of the question determines the power of the answers. I’ll just assert that we haven’t been asking very good questions lately and the quality of the research is showing the shortcomings. Lack of real risk coupled to intolerance to failure in research agendas are major problems today. Together these tendencies are tantamount to choosing a research agenda that produces little or nothing of value. These twin ills are reflected in the quality of the research questions. Poor questions that fail to probe the boundaries of knowledge lead to poor research that keeps those boundaries fixed. It is easier to continue to ask the same questions as before. There is a distinct predisposition to asking softball questions because you can be sure of the answer. If I’m sure of the answer, I haven’t asked a good question. The answer will do little to enlighten me beyond what is already self-evident.

For example, I realize now that major opportunities were missed in my previous life in Los Alamos. Up there, the nuclear weapons’ designers are kings. They also project a certain distain for computer codes despite using them virtually every day in the conduct of their work. I missed some really good questions that might have opened some doors to deeper discussions that are sorely necessary for progress. Instead we just beat around the proverbial bush and avoided the issues that hold back progress. I can imagine a dialog (past the third line its not clear where it would actually lead),

Me: “why do you believe your calculation is right?”

Designer: “I don’t, the code always lies to me”

Me: “then why do you use it?”

Designer: “it helps me solve my problems”

Me: “even if it lies?”

Designer: “I know how to separate the truth from the lies”

Me: “So it does contain some useful information?”

Designer: “Yes.”

Me: “How do you know where the utility ends, and the lies begin?”

Designer: “My judgment”

Me: “How do you know your judgment is sound?”

Designer: “I match the calculations against a lot of experimental data”

Me: “Do you know that the path taken to solution is unique, or can it be done multiple ways?”

Designer: “There is probably more than one way, but lots of experiments provide more confidence,”

Me: “What are the implications of this non-uniqueness?”

Designer: “I haven’t thought about that.”

Me: “Why? Isn’t that important or interesting?”

Designer: “It is a little frightening.”

This is the point where the discussion starts to veer into interesting and essential territory. We are confronted with systems dripping with uncertainty of all sorts. Many scientists are inherently biased toward solving well-posed initial value problems. For instance they will generally interpret experiments as a unique instantiation of the physical system, and expect the simulation to get that precise answer. This is reasonable for a stable system, but completely unreasonable for unstable systems. Remarkably, almost every technological and natural system of great interest has instabilities in it. Even more remarkably these systems often have a large enough ensemble of unstable events for them average out to reliable behavior. Nonetheless, they are not, nor should not be simulated as well-posed problems. Dealing with this situation rationally is a huge challenge that we have not stood up to as a community despite its pervasive nature.

Recently, I picked up a Los Alamos glossy (LANL publication, National Security Science) that discussed the various issues associated with nuclear weapons in today’s world. The issues are complex and tinged with deep geopolitical and technical issues. Take for instance the question of what the role of nuclear weapons is in national security today. Maybe a better question would be to answer the question, “imagine a world where the USA didn’t have nuclear weapons, but other nations did, what would it be like?” “Would you be comfortable in that World?”

The importance and training of a new generation of weapons’ designers was also highlighted in the glossy. In the dialog associated with that discussion, the gem of the “codes lie” shows up. This is a slightly more pejorative version of George Box’s quote “All models are wrong” without the positive retort “but some are useful.” I strongly suspect that the “codes lie” would be followed by “but they were useful” if the article had probed a bit deeper, but glossy publications don’t do that sort of thing. The discussion in the LANL glossy didn’t go there, and lost the opportunity to get to the deeper issues. Instead it was purely superficial spin. My retort is that codes don’t lie, but people sure do. Codes have errors. Some of these errors result from omission of important, but unknown physical effects. Other errors are committed out of necessity, such as numerical integration, which is never perfect. Other errors are merely the finite nature of knowledge and understanding such as the use of mathematics for governing equations, or imperfect knowledge of initial conditions. The taxonomy of error is the business of verification and validation with uncertainty quantification. The entire V&V enterprise is devoted to providing evidence for the quality (or lack thereof) of simulation.

We analyze systems with computer codes because those systems are deeply nonlinear and complex. The complexity and nonlinearity exceeds our capacity to fully understand. The computer code allows us to bridge our human capability for comprehension to these cases. Over time intuition can be developed when combined with concrete observation leads to confidence. This confidence is an illusion. Once the circumstances depart from where the data and simulations have taken us, we encounter a rapid degradation in predictive intuition. There is where danger lies. The fact is that the codes have errors, but people lie. People lie to gain advantage, or more commonly they lie to themselves because to answer truthfully requires them to stare in to the abyss of ignorance. In that abyss we can find the research questions worth answering and allowing mankind’s knowledge to advance.

The key is to get to a better question. It is about pulling a thread, doing an interrogation of the topic that peels away the layers of triviality, and gets to something with depth. First, the codes are more powerful than they will admit, but more deeply the path to solution is not unique. Both aspects are deeply important to the entire enterprise. I might imagine doing the same dialog with regard to climate science where similar issues naturally arise. Answers to these questions gets to the heart of computational science and its ability to contribute to knowledge.

The punch line is to push you to get at better, deeper questions as the route to better research. We need to ask questions that are uncomfortable, even unsettling. Not uncomfortable because of their personal nature (those are the “you idiot” questions where adding that phrase makes sense at the end out the question), but uncomfortable because they push us up to the chasm of our knowledge and understanding. These are questions that cause one to rethink their assumptions and if answered expand their knowledge.

I had an episode the other day that provided such a thread to pull. The issue resolves around the perniciousness of calibration and the false confidence that it produces. People looking at reactor criticality hold their calculations to a withering standard demanding five digits of accuracy. When I saw how they did this, my response was “I don’t believe that”. This was a sort of question, “can you justify those five digit?” The truth is that this answer is highly calibrated where the physical data is adjusted (homogenized) to allow this sort of accuracy, but its not “accuracy” in the sense that numerical modeling is built upon. It is precision. It is a calibrated precision where the impact of data and numerical uncertainty has been compensated for. This procedure and capability lacks virtually any predictive capability at the level of accuracy asserted. The problem is that reactor criticality is a horribly nonlinear problem, and small deviations are punished with an exponential effect. Practically speaking, the precision of getting the criticality correct (its an eigenvalue problem) is enormously important and this importance justifies the calibration.

A similar issue arises in climate science where the global energy balance must be nailed lest the Earth heat or cool unphysically. There a calibration is conducted that only applies to the specific mesh, numerical integration and subgrid models. If any of these things change the calibration must change as well to maintain the proper energy balance. The issue is whether the overall approach can be trusted at all as the system being modeled departs from the observed system that has been calibrated. For computational science this may be one of the most important issues to answer, “how far can a calibrated model be trusted?” “How can a calibrated model be trusted to assist in decisions?” Without the calibration the model is functionally useless, but with the calibration is it useful?

Questions are a way on encapsulating the core of what is wrong with computational science’s obsession with high performance computing. The question that would be better to ask is “are we focused on leveraging the right technological trends to maximize the impact of computational science on society at large?” I believe that we are not. We are missing the mark by a rather large margin. We are in the process of “doubling down” on the emphases of the past while largely ignoring how the World has changed. The change we see today is a merely the beginning of even bigger things to come. The approaches of the past will not suffice moving forward. For instance, the real hard truth is that the secrets of physical systems we are interested in will not simply submit to brute force computational power. Rather we need to spend some time thinking deeply about the questions we are trying to answer. With a little bit of deep thought we might actually start asking better questions and start down the path of getting more useful answers.

Scientific computing was once a major playing in the computing industry. Now it is merely a gnat on a whale’s back. The scientific computing community seems to be trying to swim against the incoming tidal wave instead of trying to ride it. Opportunity lies in front of us; can we muster the bravery to grasp it?

“The uncreative mind can spot wrong answers, but it takes a very creative mind to spot wrong questions.” – Anthony Jay

While I’ve opined about the dismaying state of the National Labs (meaning the DOE, NNSA or Weapons’ Labs), the state of affairs I deal with are splendid when compared to what NASA is dealing with. NASA is in terrible shape, particularly with respect to aeronautics. I’d say the support for the planetary exploration is dismal, but it is vibrant compared to the support for things aircraft related. When I’ve visited their centers, the sense of decay and awful morale is palpable. Aeronautics and almost everything about it is woefully stagnant with research support having dried up. Part of this lack of support has been a signaling by industry that research isn’t needed. The tragedy is that it shouldn’t be this way. How can something that has been so transformative to society be so abysmally supported?

Air flight has been one of the several things to utterly transform the World in the past century. Travel across the country or even more across the ocean used to be life altering lasting months or years and had the potential to completely change the course of one’s life. Now we can do any of these things in less than a day. Prior to the Internet, the ubiquity of air travel and the speed of transport had remade the globe. Despite our massive investment in air travel (planes, airports, defense, etc.) the support for scientific research has all but disappeared. I think the current lack of progress is primarily a self-fulfilling prophecy. If no effort it put into progress, progress will stop. There are several issues at play here, not the least of which is a lack of vision, and prognostications that are blatantly pessimistic including one that has become infamous. These prognostications have lacked balance and perspective on where the engines of progress arise.

I was reminded of this state of affairs during my weekly visit to “Another Fine Mesh” (http://blog.pointwise.com). Every Friday, this site publishes a set of links to interesting computational fluid dynamics (CFD) stories. I usually find at least one if not more items of significant interest. Last Friday a real gem was first up in their weekly post, a pointer to a NASA White Paper with a fantastic vision for CFD in 2030 (NASA Vision, CFD 2030 Vision, http://ntrs.nasa.gov/search.jsp?R=20140003093). The report is phenomenal. It provides a positive and balanced view of what can and needs to be accomplished to push CFD for aeronautics forward. A lot of what they discuss is totally on target. The biggest idea is that if we invest in progress we will be able to do some great things for aeronautics. Despite being truly visionary I would say that the authors don’t go far enough in spelling out what could be accomplished.

This NASA vision is running counter to the trend of declining effort, which has a lot of foundational reasons, not the least of which is disappearing support for research federally. Moreover the money spent on federal R&D is horribly inefficient due to numerous useless strings attached. In spite of significant money spent toward research, the amount of real effort has been seriously declining through systematic mismanagement and other disruptive forces. Congress who whines incessantly about waste in spending is actually the chief culprit. They add enormous numbers of wasteful requirements and needless accounting structure onto an already declining budget. They propagate the environment that kills risk taking by demanding no effort fail, and by virtue of this imperative virtually assure failure.

Intellectually, Phillip Spalart of Boeing Aerospace has a paper to which the decline in aeronautics is tied. It isn’t clear if this is more of reason as opposed to being simply an excuse. Spalart projected that the next turbulence modeling advance known as Large Eddy Simulation (LES) would not be truly useful til 2030 or even 2045 because of the computational needs for computing full wing or aircraft flows at flight conditions. You can read Spallart’s important paper at https://info.aiaa.org/tac/ASG/GTTC/Future%20of%20Ground%20Test%20Working%20Group/Reference%20Material/spalart-2000-DNS-scaleup-limits-IJHFF.pdf .

Philosophically, the largest issue with his approach is the fundamental scarcity mentality playing into the assumptions used in making the estimates. Unfortunately, the thinking involved in the LES estimates seems to be common today. It is both too common and dangerous, if not out-and-out destructive to our future.

There are serious problems with how Spalart approached his estimates. Most critically he applied the estimation techniques of 1999 too far into the future. He is assuming that no major discoveries will be made that will impact the efficiency of LES. These changes would be major model improvements, algorithms and theory developments that would more radically change computational efficiency than the computers themselves. I’ve written over the past couple of months about this. The elements in computational science outside computing hardware have always yielded more effective gains. Instead of waiting til 2030 or 2045 we might be looking at meaningful LES capability for applied aeronautics now, or within the next 10 years. Instead we disinvested in aeronautical research and killed the possibility. We have the literal self-fulfilling prophecy.

It gets worse, much worse. The estimate of 2030 to 2045 is based on the advance of computing hardware continuing unabated for that period. This almost certainty will not happen without a sea change in how computers are made. Moore’s law is dying. By 2020 it will be gone. Without the advances in theory, models, methods and algorithms we will never get there. In other words the study of fluid dynamics on a full aircraft via LES will not yield due to overpowering it with hardware. We need to think, we need to innovate, and we need to invent new ideas to solve this problem. Thankfully, this path is being described by the new NASA Vision, which hopefully will overthrow the stale viewpoint justifying the decline aeronautics.

Even worse than the estimates of the computing power are the assumptions that we will continue to use computers like we do today. Each new generation in computing has brought new ways of using them. New applications and new approaches to problem solving will arise that will render such estimates ridiculous. Ingenuity is not limited to increasing the efficiency of our current approaches, but developing new problem-solving regimes. Beyond the realm of computing are deeper discoveries in knowledge. For example, we are long overdue for meaningful developments in our understanding of turbulence. These will likely come through experiments that will utilize advances in material science, computing, optics and other fields to yield heretofore-impossible diagnostics. We will likely observe things that our present theory cannot explain, which in turn will drive new theory. The entire notion of what LES is may be transformed into something unforeseeable today. In other word, the future will probably look nothing like we think today because we can’t imagine what our fellow man will create. We can, however, believe in our fellow man’s potential to solve seemingly impossible problems,

Another argument is that we don’t need to develop better aeronautics because our aircraft are not changing any more. In fact the aircraft can’t change due to the regulatory environment. The belief is that current work is adequate for the mainstream issues in aircraft design. This might be true. Eventually things will need to change. I have a hard time imagining that in 100 years we will be flying planes that look just like today’s planes. Instead someone will decide to push knowledge forward. They will advance science including aerospace science. Doing so, they will develop new airplanes that will be much better than the current ones. The people who do this will own that future. If it isn’t the USA, then we will all be riding in planes built somewhere else. It doesn’t have to be that way, but it will if we don’t change. The same principles hold for computers, cars, toasters, TVs etc. If we allow ourselves to believe that we can’t changes, can’t do better, we won’t. Someone else who does believe they can do better will invent the future, when they invent the future they will own it, and us.

The last bit of wisdom missing from the whole dialog is the serendipity of finding entirely new applications for computational science. Part of progress is inventing entirely new ways of doing business, new ways of solving problems, and ways of thinking that are completely beyond the imagination currently. Our lack of investment in aerospace helps to assure this won’t happen. Even a casual examination of humanity’s march forward shows the folly of our current approach. Man has continuously created the new way of doing things, combined ideas and technology into new things. In fact, looking from the year 1999 it was unreasonable to assume that one could even begin to understand how things would be done in 2045, and certainly such a dismal outlook. A more defensible and honest assessment would have seen processes and progress that would seem otherworldly from the 1999 perspective including discoveries that would undo our limitations. Imaging that current limitations would hold in 2045 is blindness.

This whole episode with aeronautics is just one cautionary tale, in one field. There are many more examples today where small minded, scarcity based thinking is killing our future.

If you work actively in modeling and simulation you will encounter the codes of bygone days. If I were more of a historian it could come in really handy although comments in these codes often leave much to be desired. These codes are the stuff of myth and legend, or at least it seems like. The authors of the codes are mythically legendary. They did things we can’t do any longer; they created useful tools for conducting simulation. This is a big problem because we should be getting steadily better at conducting simulations. This becomes an even bigger problem when these people no longer work, and their codes live on.

“What I cannot create, I do not understand.” – Richard Feynman

Too large a portion of the simulation work done today is not understood in any deep way by those doing the simulation. In other words the people using the code and conducting the simulation don’t really understand much about how the answer they are using was arrived at. People run the code, get an answer do analysis without any real idea of how the code actually got the answer. This is dangerous. This actually works to undermine the scientific enterprise. Moreover, this trend is completely unnecessary, but some deep cultural undercurrents that extend well beyond the universities, labs and offices where simulation should be having a massively positive impact on society drive it.

The people who created these codes are surely proud of their work. I’ve been privileged to work with some, and all of them are generally horrified with how long their codes continue to be used to the exclusion of newer replacements. It was their commitment to apply applied technical solutions to real problems that made a difference. Those that create those earlier technical solutions were committed to applying technology to solving problems, and they were good at it. The spirit of discovery that allowed them to create codes and then see them used for meaningful work has dissipated in broad swaths of science and engineering. The disturbing point is that we don’t seem to be very good at it any more. At least we aren’t very good at getting our tools to solve problems. The developers of the mythic codes generally feel quite distressed by the continued reliance on their aging methods, and the lack of viable replacements.

Why?

I don’t believe it is the quality of the people, nor is it the raw resources available. Instead, we lack the collective will to get these things done. Our systems are letting us down. Society is not allowing progress. Our collective judgment is that the risk of change actually outweighs the need for or benefit of progress. Progress still lives on in other areas such as “big data” and business associated with the Internet, but even there you can see the forces of stagnation looming on the horizon. Areas where society should place its greatest hope for the future is under threat by the same forces that are choking the things I work on. This entire narrative needs to change for the good of society and the beneficial aspects of progress.

Remarkably, the systems devised and implemented to achieve greater accountability are themselves at the heart of achieving less. The accountability is a ruse, a façade put into place to comfort the small-minded. The wonder of solving interesting problems on a computer seems to have worn off being replaced by a cautious pessimism about the entire enterprise. None of these factors are necessary, and all of them are absolutely self-imposed limitations. Let’s look at each of these issues and suggest something better.

All the codes were created in the day when computing was in its infancy, and supported ambitious technological objectives. Usually a code would cut its teeth of the most difficult problems available and if it proved useful, the legend would be born. The mythic quality is related to the code’s ability to usefully address the problems of importance. The success of the technology supported by the code would lend itself to the code’s success and mythic status. The success of the code’s users would transfer to the code; the code was part of the path to a successful career. Ambition would be satisfied through the code’s good reputation. As such, the code was part of a flywheel with ambitious projects and ambitious people providing the energy. The legacy of the code creates something that is quite difficult to overcome. It may require more willpower to move on than the code originally harnessed in taking on its mantle of legend.

We seem to have created a federal system that is maximizing the creation of entropy. It is almost as if the government were expressing a deep commitment to the second law of thermodynamics. Despite being showered with resources, the ability to get anything of substance done is elusive. Beyond this, the elusive nature of progress is growing in prominence. Creating a code that has real utility for real applied problems takes focus, ingenuity, luck and commitment. Each of these is in limited supply. The research system of today seems to sap each of these in a myriad of ways. It seems almost impossible to focus on anything today. If I told you how many projects I work on, you’d immediately see part of the problem (7 or 8 a year). This level of accounting comes at me from a myriad of sources, some entirely local, and some National in character. All of it tinged with the sense that I can’t be trusted.

It takes a great deal of energy to drive these projects toward anything that looks coherent; none of this equals the creation and stewardship of a genuine capability. Ingenuity is being crushed by the increasingly risk adverse and politically motivated research management system. Lack of commitment is easy to see with the flighty support for most projects. Even when projects are supported well, the management system slices and dices the effort into tiny bite-sized pieces, and demands success in each. Failure is not tolerated. Wisdom dictates that the lack of tolerance for failure is tantamount to destroying the opportunity for success. In other words, our risk aversion is causing the very thing that it was designed to avoid. Between half-hearted support, and risk aversion the chance for real innovation is being choked to death.

The management of the Labs where I work is becoming ever more intrusive. Take for example the financial system. Every year my work is parceled into ever-smaller chunks. This is done in the name of accountability. Instead the freedom to execute anything big is being choked by all this accountability. The irony is that the detailed accounting is actually assuring that less is accomplished, and the people driving the micromanagement aren’t accountable for the damage they have caused in the slightest. The micro accounting of my time is also driving a level of incrementalism into the work that destroys the ability to do anything game changing. This incrementalism goes hand-in-hand with the lack of any risk-taking. We are dictated to succeed by fiat, and by the same logic success on a large scale will also be inhibited.

When it comes to code development the incremental attitude results in work being accreted onto the same ever-older code base. The low risk path is to add a little bit more onto the already useful (legacy) code. This is done despite the lack of real in-depth knowledge of how the code actually works to solve problems. The part of the code that leads to its success is almost magical, and as magic can’t be tampered with. The foundation for all the new work is corrupted by the lack of understanding which then poisons the quality of the work built on top of the flawed base. As such, the work done on top of the magical foundation is intrinsically superficial. Given the way we manage science today superficiality should actually be expected. Our science and engineering management is focused almost to exclusion on the most superficial aspects of the work.

The fundamental fact is that a new code is a risk. It may not replace or improve upon the existing capability. Success can never be guaranteed, nor should it be. Yet we have created a system of managing science that cannot tolerate any failure. Existing codes already solve the problem well enough for somebody to get answers, and the low risk path is to build upon this. Instead of building upon the foundation of knowledge and applying this to better solutions, it is cheaper and lower risk to simply refurbish the old code. Like much of our culture today the payoff is immediate rather than delayed. You get new capability right away rather than a much better code later. Right away and crummy beats longer term and great every time. Why? Short attention spans? No real accountability? Deep institutional cynicism?

A good analogy is the state of our crumbling physical infrastructure. The United States’ 20^th Century infrastructure is rotting before our eyes. When we should be thinking of a 21^st Century infrastructure, we are trying to make the last Century’s limp along. Think of an old bridge that desperately needs replacement. It is in danger of collapse and represents a real risk rather than a benefit to its users. More often than not in today’s America, the old bridge is simply repaired, or retrofitted regardless of its state of repair. You can bumble along this path until the bridge collapses. Most bridges don’t, but some do to tragic consequences. Usually there is a tremendous amount of warning that is simply ignored. Reality can’t be fooled. If the bridge needs replacing and you fail to do so, a collapse is a reasonable outcome. Most of the time we just do it on the cheap.

We are doing science exactly the same way. In cases where no one can see the bridge collapse, the guilty are absolved of the damage they do. Just like physical infrastructure, we are systematically discounting the future value for the present cost. The management (can’t really call them leaders) in charge simply is not stewards of our destiny; they are just trying holding the crumbling edifice together until they can move on with the hollow declaration of success. Sooner or later, this lack of care will yield negative consequences.

All this caution is creating an environment that fails to utilize existing talent, and embodies a pessimistic view of man’s capacity to create. This might be the saddest aspect of the overall malaise, the waste of potential. Our “customers” actually ask very little of us, and the efforts don’t really push our abilities; except, perhaps, our ability to withstand work that is utter dreck. The objectives of the work are small-minded with a focus on producing sure results and minimizing risks. The system does little to encourage big thoughts, dreams, or risks behind creating big results. Politicians heighten this sense by constantly discussing how deeply in debt our country is as an excuse for not spending money on things of value. Not every dollar spent is the same; a dollar invested in something of value is not the same as a dollar spent on something with no return. All of this is predicated on the mentality of scarcity, and a failure to see our fellow man or yourself as engines of innovation and unseen opportunities. History will not be kind to our current leadership when it is realized how much was squandered. The evidence that we should have faith in man’s innate creative ability is great, and the ignorance of the possibility of a better world is hard to stomach.

The first step toward a better future is change in the assumptions regarding what an investment in the future looks like. One needs to overthrow the scarcity mentality and realize that money invested wisely will yield greater future value. Education and lifetime learning is one such investment. Faith in creativity and innovation is another investment. Big audacious goals and lofty objectives are another investment. The big goal is more than just an achievement; it is an aspiration that lifts the lives of all who contribute. It also lifts the lives of all that are inspired by it. Do we have any large-scale societal goals today? If we don’t, how can we tolerate such lack of leadership? We should be demanding something big and meaningful in our lives. Something that is worth doing and something that would make the current small-minded micromanagement and lack of risk taking utterly unacceptable. We should be outraged by the state of things, and the degree to which we are being led astray.

All of us should be actively engaged in creating a better world, and solving the problems that face us. Instead we seem to be just hanging on to the imperfect world handed to us. We need to have more faith in our creative problem solving abilities, and less reverence for what was achieved in the past. The future is waiting to be created.

“If you want something new, you have to stop doing something old” ― Peter F. Drucker

“When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.” – Clarke’s first law

It would be easy to point fingers at the crushing bureaucratic load we face at many of our premier research institutes. I think that this only compounds the real forces holding us back as a sort of mindless ally in the quest for mediocrity. I for one can feel my ability to think and create being syphoned away by meaningless paperwork, approvals, training and mindless formality. The personal toll is heartbreaking and the taxpayers should be up in arms. Of course most of this is driven by our scandal mongering political system and the increasingly tabloidesque media. These items are merely various forms of societal dissipation aimed at driving entropy into its all-consuming conclusion.

When I came across the article in the Daily Beast (Our Mindless Government Is Heading for a Spending Disaster) yesterday on the book “The Rule of Nobody,” by Phillip K. Howard it became clear that I’m not alone in feeling this way. Our Labs are actually not run by anyone, and certainty not the management of the Lab. The problem with this approach is not partisan, but rather associated with a tendency to be lazy in our rule. The core of what drives this trend is the inability to reinvent our governance. This failure to reinvent is then at the core of the deeper issue, the fear of risk or failure. We have a society-wide inability to see failure for what it is; failure is a necessary vehicle for success. Risk is the thing that allows us to step forward toward both accomplishment and failure. You cannot have one without the other. Somehow as a culture we have forgotten how to strive, to accept the failure as a necessary element for a healthy Country. Somehow this aversion has crept into our collective consciousness. It is sapping our ability to accomplish anything of substance.

In scientific research the inability to accept risk and the requisite failure is incredibly destructive. Research at its essence is doing something that has never been done before. It should be risky and thus highly susceptible to failure. Our ability to learn the limits of knowledge is intimately tied to failure. Yet failure is the very thing that we are not encouraging as a society. In fact, failure is punished without mercy. The aggregate impact of this is the failure to accept the sort of risk that leads to large-scale success. To get a “Google” or a “moon landing” we have to fund, accept and learn from innumerable failures. Without the failure the large success will elude us as well.

Another is the artificial limitation we place on our thinking in the guise of thinking “it’s impossible”. Impossible also implies risk and the large chance of outright failure. We quit pushing the limits of what might be possible and escape into the comfortable confines of the safe possible, A third piece is the inability to marshal our collective efforts in the pursuit of massive societal goals. These goals capture the imagination and drive the orientation toward success beyond us to greater achievements. Again, it is the inability to accept risk. The last I’ll touch upon is the lack of faith in the creative abilities of mankind. Man’s creative energies have continually overcome limitations for millennia and there is no reason to think this won’t continue. Algorithmic improvement’s impacts on computing are but one version of the large theme of man’s ability to create a better world.

It seems that my job is all about NOT taking risks. The opposite should be true. Instead we spend all our time figuring out how to not screw up, how to avoid any failure. This, of course, is antithetical to success. All success, all expertise is built upon the firm foundation of glorious failure and risk. Failure is how we learn and risk helps to stoke the flames of failure. Instead we have grown to accept creeping mediocrity as the goal of our entire society. When the biggest goal at work is “don’t screw up” it is hard to think of a good reason to do anything. We have projects that have scheduled breakthroughs and goals that are easy to meet. Very few projects are funded that actually attack big goals. Instead instrumentalism abounds and the best way to get funded is to solve the problem first then use the result to justify more funding. It’s a vicious cycle, and it is swallowing too much of our efforts.

Strangely enough, the whole viscous cycle also keeps us from doing the mundane. Since our efforts are so horrifically over managed there is no energy to actually execute what should be the trivial aspects of the job. Part of this related to the slicing and dicing of our work into such small pieces any coherence is lost. The second part is the lack of any overarching vision of where we are going. The lack of big projects with scope kills the ability to do consequential tasks that should be easy. Instead we do all sorts of things that seem hard, but really amount to nothing. We are a lot of motion without any real progress. Some of us noted a few weeks ago that new computer codes were started every five to seven years. Then about 25 years ago that stopped. Now everything has to be built upon existing codes because it lowers the risk. We have literally missed four or five generations of new codes. This is failure on an epic scale because no one will risk something new.

“Can we travel faster than the speed of light?” My son once asked me. A reading of the standard, known theories of physics would give a clear unequivical “No, it would be impossible.” I don’t buy this as the ultimate response. A better and more measured response would be “not with what we know today, but there are always new things to be learned about the universe.” “Maybe we can using physical principles that haven’t been discovered yet.” Some day we might travel faster than light, or effectively so, but it won’t look like Star Trek’s warp drive (or maybe it will, who knows). The key is to understand that what is possible or impossible is only a function of what we know today, and our state of knowledge is always growing.

In mathematics these limits on possibility often take the form of barrier theorems. These state what cannot be done. These barriers can be overcome if the barriers are looked at liberally with an eye toward loopholes. A common loophole is linearity. Linearity infuses many mathematical proofs and theorems, and the means to overcoming the limitations are appealing to nonlinearity. One important example is Godunov’s theorem where formal accuracy and monotonicity were linked. The limit only exists for linear numerical methods, and a nonlinear numerical method can be both greater than first order accurate and monotone. The impossible was possible! It was simply a matter of thinking about the problem outside the box of the theorem.

In most of the areas that have traditionally supported scientific computing are languishing today. Almost nothing in the way of big goal oriented projects exist to spur progress. The last such program was the ASCI program from the mid-1990’s, which unfortunately focused too much on pure computing as the route to progress. ASCI bridged the gap between the CPU dominated early era to the growth in massively parallel computation. If fact parallel computing has masked the degree to which we are collectively failing to use our computers effectively. This era is drawing to a close, and in fact Moore’s law is rapidly dying.

While some might see the death of Moore’s law as a problem, it may be an opportunity to reframe to quest for progress. In the absence of computational improvements driven by the technology, the ability to progress could be again given to the scientific community. Without hardware growing in capability the source of progress resides in the ability of algorithms, methods and models to improve. Even under the spell of Moore’s law, these three factors have accounted for more improvement in computational capability than hardware. What will our response be to losing Moore’s law? Will we make investments appropriately in progress? Will we refocus our efforts on improving algorithmic efficiency, better numerical methods and improved modeling? Hope springs eternal!

In the final analysis, such an investment requires a great deal of faith in man’s eternal ability to create, to discover and be inspired. History provides an immense amount of evidence that this faith would be well placed. As noted above, we have created as much if not more computational capability through ingenious algorithms, methods, heuristics, and models than our massive strides in computational hardware.

It is noteworthy that the phone in my pocket today has the raw computational power of a Cray 2. It sits idle most of the time and gets used for email, phone calls, texts and light web browsing. If you had told me that I’d have this power available to me like these 25 years ago, I would have been dumbstruck. Moreover, I don’t really use it for anything like I’d have used a Cray 2. The difference is that the same will almost certainly not happen in the next 25 years. The “easy” progress simply riding the coattails of Moore’s law is over. We will have to think hard to progress and take a different path. I believe the path is clear. We have all the evidence needed to continue our progress.

We are deeply biased by our perceptions and preconceptions all the time. We make many decisions without knowing we are making a decision constantly. Any recognition of this would probably terrify most rational people. We often frame our investigations to prove the conclusion we have already made. Computer modeling and simulation has advanced to the point where it is forming biases. If one’s most vivid view of an unseeable event is a simulation, a deep bias can be shaped in favor of the simulation that unveiled the unseeable. We are now at the point where we need to consider if improvement in modeling and simulation can be blocked by such biases.

For example in one modeling effort, for high explosives efforts had a favored a computer code that is Lagrangian (meaning the mesh moves with the material). The energy release from explosives causes fluid to rotate vigorously and this rotation can render the mesh into a tangled mess. Besides becoming inaccurate, the tangled mesh will invariably endanger the entire simulation. To get rid of the problem, this code converts tangled mesh elements into particles. This is a significant upgrade over the practice of “element death” where the tangled grid is completely removed when it becomes a problem along with mass, momentum and energy… Conservation laws are laws, not suggestions! Instead the conversion to particles allows the simulation to continue, but bring all the problems with accuracy and ultimately conservation that particles bring along (I’m not a fan of particles).

More tellingly, competitor codes and alternative simulation approaches will add particles to their simulation. The only reason the particles are added is to give the users something that looks more like what they are used to. In other words the users expect particles in interesting parts of the flow, and the competitors are eager to give it to them whether it is a good idea or not (it really isn’t!). Rather than develop an honest and earnestly better capability, the developers focus on providing the familiar particles.

Why? The analysts running the simulations have come to expect particles, and the particles are common where the simulations are the most energetic, and interesting. To help make the analysts solving the problems believe the new codes particles come along. I, for one, think particles are terrible. Particles are incredibly seductive and appealing for simulation, but ultimately terrible because of their inability to satisfy even more important physical principles, or provide sufficient smoothness for stable approximations. Their discrete nature causes an unfortunate trade space to be navigated without sufficiently good alternatives. In some cases you have to choose between smoothness for accuracy and conservation. Integrating particles is often chosen because they can be done without dissipation, but dissipation is fundamental to physical, casual events. Causality, dissipation and conservation all trump a calculation with particles without these characteristics. In the end the only reason for the particles is the underlying bias of the analysts who have grown to look for them. Nothing else, no reason based on science, it is based on providing the “customer” what they want.

“If I had asked people what they wanted, they would have said faster horses.”– Henry Ford.

There you have it, give people what they don’t even know they need. This is a core principle in innovation. If we just keep giving people what they think they want, improvements will be killed. This is the principle that code related biases create. They are biased strongly toward what they already have instead of what is possible.

Modeling and simulation has been outrageously successful over the decades. This success has spawned the ability to trick the human brain to believing that what they see is real. The fact that simulations look so convincing is a mark of massive progress that has been made. This is a rather deep achievement, but it is fraught with the danger of coloring perceptions in ways that cannot be controlled. The anchoring bias I spoke of above is part of that danger. The success now provides a barrier to future advances. In other words enough success has been achieved that the human element in determining quality may be a barrier to future improvements.

It might not come as a surprise for you to think that I’ll say V&V is part of the answer.

V&V has a deep role to play in improving upon this state of affairs. In a nutshell, the standard for accepting and using modeling and simulation must improve in order to allow the codes to improve. A colleague of mine has the philosophy, “you can always do better.” I think this is the core of innovation, success and advances. There is always a way to improve. This needs to be a steadfast belief that guides our choices, and provides the continual reach toward bettering our capabilities.

What can overcome this very human reaction to the visual aspects of simulation?

First, the value of simulation needs to be based upon the comparisons with experimental measurements, not human perceptions. This is easier said than done. Simulations are prone to being calibrated to remove differences from experimental measurements. Most simulations cannot match experimental observables without calibration, and/or the quality standards cannot be achieved without calibration. The end result is the inability to assess the proper value of a simulation without the bias that calibration brings. An unambiguously better simulation will require a different calibration, and potentially a different calibration methodology.

In complex simulations, the full breadth of calibration is quite difficult to fully grapple with. There are often multiple sources of calibration in simulation including any subgrid physics, or closure relations associated with physical properties. Perhaps the most common place to see calibration is the turbulence model. Being an inherently poorly understood area of physics; turbulence modeling is prone to being a dumping ground for uncertainty. For example, ocean modeling often uses a value for the viscous dissipation that far exceeds reality. As a friend of mine like to say, “if the ocean were as viscous as we model it, you could drive to England (from the USA).” Without strong bounds being put on the form and value of parameters in the turbulence model, the values can be modified to give better matches to more important data. This is the essence of a heavy-handed calibration common. An example might be the detailed equation of state for a material. Often a simulation code has been used in determining various aspects of the material properties or analyzing the experimental data used.

I have witnessed several difficult areas of applied modeling and simulation overwhelmed by calibration. The use of calibration is so commonly accepted, the communities engage in it without thinking. If one isn’t careful the ability to truly validate the state of “true” modeling knowledge becomes nearly impossible. The calibration begins to become intimately intertwined with what seems to be fundamental knowledge. For example, a simulation code might be used to help make sense of experimental data. If one isn’t careful errors in the simulation used in reducing the experimental data can be transferred over to the data itself. Worse yet, the code used in interpreting the data might utilize a calibration (it almost certainty does). At that point you are deep down the proverbial rabbit hole. Deep. How the hell do you unwind this horrible knot? You have calibrated the calibrator. Even more pernicious errors might be the failure to characterize the uncertainties in the modeling and simulation that is used to help look at the experiment. In other cases calibrations are used so frequently that they simply get transferred over into what should be fundamental physical properties. If these sorts of steps are allowed to proceed forward, the original intent can be lost.

These steps are in addition to a lot of my professional V&V focus, code verification and numerical error estimation. These practices can provide unambiguous evidence that a new code is a better solution on analytical problems and real applications. Too often code verification simply focuses upon the correctness of implementations as revealed by the order of convergence. The magnitude of the numerical error can be revealed as well. It is important to provide this evidence along with the proof of correctness usually associated with verification. What was solution verification should be called numerical error estimation, and it provides important evidence on how well real problems are solved numerically. Moreover, if part of a calibration is accounting for numerical error, the error estimation will unveil this issue clearly.

The bottom line is to ask questions. Ask lots of questions, especially ones that might seem to be stupid. You’ll be surprised how many stupid questions actually have even stupider answers!

It feels almost dirty to put a “#” hashtag in my title, but what the hell! The production of predictive models is the holy grail of modeling and simulation.  On the other hand we have the situation where a lot of scientists and engineers who think they have predictivity when in fact they have cheated. By “cheating” I usually mean one form or another of calibration either mindfully or ignorantly applied to the model. The model itself ends up being an expensive interpolation and any predictivity is illusory. 

When I say that you are modeling the real world, I really mean that you actually understand how well you compare.  A model that seems worse, but is honest about your simulation mastery is better than a model that seems to compare better, but is highly calibrated.  This seems counter intuitive as I’m say that a greater disparity is better.  In the case where you’ve calibrated your agreement, you have lost the knowledge of how well you model anything.  Having a good idea of what you don’t know is essential for progress.

A computational model sounds a lot better than an interpolation. In such circumstances simulation ends up being a way of appearing to add more rigor to the prediction when any real rigor was lost in making the simulation agree so well to the data. As long as one is simply interpolating the cost is the major victim of this approach, but in the case where one extrapolates there is danger in the process.  In complex problems simulation is almost always extrapolating in some sense.  A real driver for this phenomenon is mistakenly high standards for matching experimental data, which drive substantial overfitting of data (in other words forcing a better agreement than the model should allow). In many cases well-intentioned standards of accuracy in simulation drive pervasive calibration that undermines the ability to predict, or assess the quality of any prediction.  I’ll explain what I mean by this and lay out what can be proactively done to conduct bonafide modeling. 

I suppose the ignorant can be absolved of the sin they don’t realize they are committing. Their main sin is ignorance, which is bad enough.  In many cases the ignorance is utterly willful.  For example, physicists tend to show a lot of willful ignorance of numerical side effects. They know it exists yet continue to systematically ignore it, or calibrate for its effects.  The delusional calibrators are cheating purposefully and then claiming victory despite having gotten the answer by less than noble means.  I’ve seen example after example of this in a wide spectrum of technical fields. Quite often nothing bad happens until a surprise leaps up from the data.  The extrapolation finally becomes poor and the response of the simulated system surprises. 

The more truly ignorant will find that they get the best answer by using a certain numerical method, or grid resolution and with no further justification declare this to be the best solution. This is the case for many, many engineering applications of modeling and simulation.  For some people this would mean using a first-order method because it gives a better result than the second-order method.  They could find that using a more refined mesh gives a worse answer and then use the coarser grid.  This is easier than trying track down why either of these dubious steps would give better answers because they shouldn’t.  In other cases, they will find a dubious material or phenomenological model gives better results, or a certain special combination.  Even more troubling is the tendency to choose expedient techniques whereby mass, momentum or energy is simply thrown away, or added in response to a bad result. Generally speaking, the ignorant that apply these techniques have no general idea how accurate their model actually is, its uncertainties, or the uncertainties in the quantities they are comparing to.

While dummies abound in science, charlatans are a bigger problem.  While calibration when mindfully done and acknowledged is legitimate, the misapplication of calibration as mastery in modeling is rampant. Again, like the ignorant, the calibrators often have no working knowledge of many of innate uncertainties in the model.  They will joyfully go about calibrating over numerical error, model form, data uncertainty, and natural variability without a thought.  Of course the worst form of this involves ignorant calibrators who believe they have mastery over things they understand poorly. This ultimately is a recipe for disaster, but the near term benefits of these practices are profound. Moreover the powers that be are woefully prepared to unmask these pretenders.

At its worst calibration will utilize unphysical, unrealizable models to navigate the solution into complete agreement with data.  I’ve seen examples where fundamental physical properties (like equation of state or cross sections) are made functions of space, when they should be invariant of position. Even worse the agreement will be better than it has a right to be, not even include the possibility that the data being calibrated to is flawed. Other calibrations will fail to account for experimental measurement error, or natural variability and never even raise the question of what these might be.  In the final analysis the worst aspect of this entire approach is lost opportunity to examine the state of our knowledge and seek to improve it.

How to do things right:

1. Recognize that the data you are comparing to isn’t accurate, and variable. Try to separate these uncertainties into their sources, measurement error, intrinsic variability, or unknown factors.

2. Your simulation results are similarly uncertain for a variety of reasons. More importantly you should be able to more completely and mindfully examine their sources and estimate their magnitude. Numerical errors arise from finite resolution, uncoverged nonlinearities (the effects of linearization), unconverged linear solvers, and outright bugs.  The models often can have their parameters change, or even change to other models.  The same can be said of the geometric modeling.

3. Much of the uncertainty in modeling can be explored in a concrete manner by modifying the details of the models in a manner that is physically defensible. The values in or from the model can be changed in ways that can be defended in a strict physical sense.

4. In addition different models are often available for important phenomena and these different approaches can yield a degree of uncertainty.  To some degree different computer codes themselves constitute different models and can be used to explore differences in what would be considered reasonable defensible models of reality.

5. A key concept in validation is a hierarchy of experimental investigations that cover different levels of system complexity, and modeling difficulty. These sources of experimental (validation) data provide the ability to deconstruct the phenomena of interest into its constituent pieces and validate them independently. When everything is put together for the full model a fuller appreciation for the validity of the parts can be achieved allowing greater focus on the source of discrepancy.

6. Be ruthless in uncovering what you don’t understand because this will define your theoretical and/or experimental program.  If nothing else it will help you mindfully and reasonably calibrate while places limits of extrapolation.

7. If possible work on experiments to help you understand basic things you know poorly and use the results to reduce or remove the scope of calibration.

8. Realize that the numerical solution to your system itself constitutes a model of one sort or another.  This model is a function of the grid you use, and the details of the numerical solution.

9. Separate your uncertainties between the things you don’t know and the things that just vary. This is the separation of epistemic and aletory uncertainty.  The key to this separation is that that epistemic errors can be removed through learning more. Aletory uncertainty is part of the system that is harder to control. 

10. Realize that most physical systems are not completely well determined problems.  In other words if you do an experiment that should be the same over and over some of the variation in results is due to imperfect knowledge of the experiment.  One should not try to exactly match the results of every experiment individually; some of the variation in results is real physical noise.

11. Put everything else into the calibration, but realize that it is just papering over what you don’t understand.  This should provide you with the appropriate level of humility.