Experiment is the sole source of truth. It alone can teach us something new; it alone can give us certainty.

― Henri Poincaré

In looking at the dynamic surrounding verification and validation recently I’ve noticed a very grim evolution of the narrative. Two things have happened to undermine the maturity of V&V. One I’ve spoken about in the past, the tendency to drop verification and focus solely on validation, which is bad enough. In the absence of verification, validation starts to become rather strained and drift toward calibration. Assurances that one is properly solving the model they are claiming to be solving are unsupported by evidence. This is bad enough all by itself. The use of V&V as a vehicle for improving modeling and simulation credibility is threatened by this alone, but something worse looms even larger.

A more common and pervasive trend is the conflation of validation with uncertainty quantification. It has become very common for uncertainty quantification (UQ) to be defined as the whole of validation. To some extent this is fueled by a focus on high performance computing where UQ provides a huge appetite for computational cycles thus eliciting lots of love and support in HPC circles. Validation must be about experiments and a broad cross section of uncertainties that may only be examined through a devotion to multi-disciplinary work and collaboration. One must always remember that validation can never be separated from measurements in the real world whether experimental or observational. The experiment-simulation connection in validation is primal and non-negotiable.

There are three types of lies — lies, damn lies, and statistics.

― Benjamin Disraeli

A second part of the issue is the hot topic nature of UQ. UQ has become a buzzword and seems to be a hot issue in publishing and research. Saying you’re doing UQ seems to be a means to squeezing money out of funding agents. In addition UQ can be done relatively automatically and mechanically. Tools and techniques exist to enable UQ to be done without much deep thought even though it touches upon many deep technical topics. Actual validation is far harder and more holistic. The core to any work in validation is serious experimental expertise and hard-nosed comparison with simulations. The detailed nature of the experiment and its intrinsic errors and uncertainties is the key to any comparison. Without knowing the experimental uncertainty any computational uncertainty is context free. My grumpy intellectual would quip that validation requires thinking and that leads people to avoid it because thinking is so hard. The deeper issues are that validation is complex and mutli-disciplinary in nature making it collaborative and difficult. Experts in a single discipline can do UQ, so it is an easy out.

Five percent of the people think;

ten percent of the people think they think;

and the other eighty-five percent would rather die than think.

― Thomas A. Edison

One of the biggest issues is the stunning incompleteness of UQ in general. Most commonly UQ is done via an exploration of the variation of parameters in models. Complex models of reality have a lot of constants that are not known with great precision. Various techniques may be utilized to efficiently examine the variation in computational solutions due to changes in these parameters. Among the methods used are things like Markov Chain Monte Carlo (MCMC), polynomial chaos, and other sampling methods. The results from this work are useful and sound, but form a rather incomplete view of uncertainty. Even in these cases the sampling is often subject to lack of certainty with the assessment driven by the difficulty of determining uncertainty in high dimensional spaces. Modeling and simulation suffers from a host of other uncertainties not covered by these methodologies. For example most simulations have some degree of numerical error that may be quite large. Numerous techniques exist for exploring its magnitude and nature. Many systems being modeled have some stochastic or variability associated with them. Modeling assumptions are often made in simulating a system or experiment. The solution may change greatly on the basis of these assumptions or modeling approximations. A different computational modeler may make much different assumptions and produce a different solution.

Judge a man by his questions rather than by his answers.

― Voltaire

If validation is to be done properly a fairly complete accounting of modeling and simulation uncertainty is needed. One also needs to also understand the experimental error and uncertainty with equal completeness. One must be acutely aware of the intrinsic lack of certainty in the estimation of uncertainty. The combination of the solutions and the sizes of each uncertainty provides a modeling and simulation solution into proper context. Without knowledge of the uncertainties in each data source, the distance between solutions cannot be judged. For example if the experimental precision is very good and the uncertainty is quite small, the simulation needs to be equally precise to be judged well. Conversely a large experimental uncertainty would allow model to be much looser, and still be judged well. More critically the experiment wouldn’t provide actionable evidence on research needs, and expert judgment would reign.

The whole of the uncertainty provides an important source of scientific tension. If experimental uncertainty is small, it requires modeling and simulation to be equally precise to imply good results. It pushes the modeling to improve to meet the high standard of the experiment. If the modeling and simulation is very good, but the experiments have large uncertainty, it should push the experiments to improve because they fail to constrain and distinguish between models. By having a deep and complete understanding of uncertainty, we can define where we need to put resources to improve. We know what aspects of our current knowledge are the most in need of attention and limiting progress.

One must always be aware of the significant attraction of short changing uncertainty estimation. Doing a complete job of estimating uncertainty almost always results in an increase in the magnitude of uncertainty. This is where science as a fundamentally human enterprise comes into play. People would rather think uncertainties are small than large. Uncertainty is uncomfortable and people shy away from discomfort. By under-estimating uncertainty people unconsciously put themselves at ease by doing incomplete work. A more rigorous and complete approach almost always produces a discomforting result. When one combines discomfort with difficulty of accomplishment, the necessary factors for lack of effort and completeness becomes clear. With this temptation in mind the tendency to take the easy route must be acknowledged.

The bottom line is the necessity understanding uncertainty in a holistic manner can produce useful and defensible context for science. It can allow us to understand where we need to improve our knowledge or practice. Without this accounting the whole issue falls into the area of relying upon expert judgment or politics to make the decisions. We fail to understand where our knowledge is weak and potentially overlook experiments necessary for understanding. We may have the right experiments, but cannot make measurements of sufficient accuracy. We might have models of insufficient complexity, or numerical solutions with too much numerical error. All of these spell out different demands for resource allocation.

Much of the tension is captured in these two quotes although I hope Eddington was probably trying to be ironic!

Never trust an experimental result until it has been confirmed by theory

― Arthur Stanley Eddington

It doesn’t matter how beautiful your theory is … If it doesn’t agree with experiment, it’s wrong.

― Richard Feynman

It is not enough for code to work.

― Robert C. Martin

One of the big selling points for high performance computing is real world impact. A focus for this impact revolves around computer codes with the capability to produce answers for people working on real world problems. In organizations such as mine where modeling and simulation produces analysis used to assist decision-making, these codes are known as production codes, or the production codes. As these codes provide real value to our applied programs, and with this value provides generous financial support. This support is necessary for the codes to do their job and creates financial stability. With this support comes acute responsibility that needs to be acknowledged and serious effort needs to apply to meeting these.

Being a production code means producing results that are meaningful. The results are important by virtue of their utility in decision-making that impacts the real world. In the past those using production codes as expert users provided the credibility needed to make these codes important. In other words credibility was granted by the weight of the expertise of the users. This element in expertise-based credibility is still alive today, but increasingly being replaced by systematic approaches to bolster the purely human element. The appropriate and proper use of verification, validation, and uncertainty quantification along with software quality assurance provides a system for assessing credibility and constructing evidence. More and more this evidence is demanded to document credibility and reassure decision makers.

One of the main things production codes give is the ability to model important problems in the real world. This is the primary characteristic of production codes, modeling capability for real problems. While this character is primary in defining production codes everything else important in high performance computing is eclipsed by the modeling imperative. These codes are essential for the utility of high performance computing resources and often become the first codes to make their way and use high-end resources. They quite often are the explicit justification for the purchase of such computing hardware. This character usually dominates and demands certain maturity of software professionalism.

On the flipside there are significant detrimental aspects of such codes. For example the methods and algorithms in production codes are often crude and antiquated in comparison to state of the art. The same can be said for the models, the algorithms and often the computer code itself. The whole of the production code credibility is deeply impacted by these pedigrees and their impact on real World programs and things. This issue comes from several directions; the codes are often old and used for long periods of time. The experts who traditionally define the credibility drive this to some extent. It often takes a long time to develop the code to the level needed to solve the hard real world problems as well as the expertise to navigate the code’s capability into results that have real world meaning. Older methods are robust, proven and trusted (low order, and dissipative is usually how robust happens). Newer methods are more fragile, or simply can’t deal with all the special cases and issues that threaten the solution of real problems. Again, the same issues are present with models, algorithms and the nature or quality of the computer code itself.

Knowledge is something which you can use.
Belief is something which uses you.

― Idries Shah

In the final analysis the production code status must be earned and can not granted by fiat. Generally one might intend for a code to become a production code, but it only becomes a production code when it begins to produce. That production must be delivered with credibility and trust. It is an utterly organic process that cannot be forced. While significant support comes with production code status, it also comes with responsibilities as well. Increasingly in a modern context these responsibilities include software quality, verification and validation to be executed in a serious professional manner. Hopefully the “customers” for production code analysis will be more demanding and insistent on a more systematic pedigree. Even today this pull for systematic pedigree is poorly supported by the customers of production code results creating an environment where counter-productive practices and results continue to supported.

In far too many instances, the systematic pedigree defining steps are being skipped for the old system of person-centered credibility. The old person-centered system is simple and straightforward. You trust somebody and develop a relationship that supports credibility. This person’s skills include detailed technical analysis, but also inter-personal relationship building skills. If such a system is in place there is not a problem as long as the deeper modern credibility is also present. Too often the modern credibility is absent or shorted and effectively replaced by a cult of personality. If we put our trust in people who do not value the best technical work available in favor of their force of personality or personal relationships we probably deserve the substandard work that results.

Tell me what you pay attention to and I will tell you who you are.

― José Ortega y Gasset

 

Reality is that which, when you stop believing in it, doesn’t go away.

― Philip K. Dick

I applied the smoothed operators to the actual solution of a scalar advection law, and wanted to show how the methods impact the solution. This should put the discussion of the past couple of weeks into a bit sharper and more relevant focus. We can also explore the impact of the degree of regularization chosen in the smoothing. The good things below are finding out that my analysis seems to completely coincide with the results, and in that case the results are the lowest error and highest quality solution. More surprisingly, the best result is obtained with a smoothed function, not the original discontinuous ones!

We will solve the scalar advection equation for a square wave on a mesh of 200 cells in one dimension using a Sweby version of Fromm’s scheme. The solution will do one rotation on the grid using 400 time steps. I’ll plot the solution and the error plus display the L1, L2 and L-infinity norms of the error.

First with the standard unsmooth functions.

Error= 0.0314099 0.00649721 0.438233

Now with smooth functions all based on using the and .

Error= 0.0313124 0.00648103 0.4384

Now with smooth functions all based on using the and .

Error= 0.0350856 0.00694473 0.454124

Now with smooth functions all based on using the and .

Error= 0.0257311 0.00578931 0.420996

What we see is that the smoothed operators produce high quality and lower error solutions in most cases. The one case with the linear version of the smoothed sign function the larger error is supported by the analysis I gave last week. Generally as the regularization allows the function to come as close as we might like to the original functions.

To test a perfect theory with imperfect instruments did not impress the Greek philosophers as a valid way to gain knowledge.

― Isaac Asimov

Note: I got super annoyed with the ability of WordPress to parse LaTeX so there are a lot of math type in here that I gave up on. Apologies!

Last week I introduced a set of alternatives to discontinuous intrinsic functions providing logical functional capability in programming. In the process I outlined some of the issues that arise due to discontinuous aspects of computer code operation. The utility of these alternatives applies to common regression testing in codes and convergence of nonlinear solvers. The issue remains with respect to how useful these alternatives are. My intent is to address components of this aspect of the methods this week. Do these advantageous functions provide their benefit without undermining more fundamental properties of numerical methods like stability and convergence of numerical methods? Or do we need to modify the implementation of these smoothed functions in some structured manner to assure proper behavior.

The answer to both questions is an unqualified yes, they are both useful and they need some careful modification to assure correct behavior. The smoothed functions may be used, but the details do matter.

To this end we will introduce several analysis techniques to show these issues concretely. One thing to get out of the way immediately is how this analysis does not change some basic aspects of the functions. For all of the functions we have the property that the original function is recovered in an asymptotic limit that is becomes the original sign function, . Our goal is to understand the behavior of these functions within the context of a numerical method away from this limit where we have obviously deviated substantially from the classical functions. We fundamentally want to assure that the basic approximation properties of methods are not altered in some fatal manner by their use. A big part of the tool kit will be systematic use of Taylor series approximations to make certain that the consistency of the numerical method and order of accuracy are retained when switching from the classical functions to their smoothed versions. Consistency simply means that the approximations are valid approximations to the original differential equation (meaning the error is ordered).

There was one important detail that I misplaced during last week’s post. If one takes the definition of the sign function we can see an alternative that wasn’t explored. If we have . Thus we can easily rearrange this expression to give two very different regularized absolute value expressions, and . When we move to the softened sign function the behavior of the absolute value changes in substantive ways. In particular in the cases where the softened absolute value was everywhere less than the classical absolute value, it would be greater for the second interpretation and vice-versa. As a result functions like now can produce an absolute value, that doesn’t have issues with entropy conditions as the dissipation will be more than the minimum, not less. Next we will examine whether these different views have significance in truncation error.

Our starting point will be the replacement of the sign function or absolute value in upwind approximations to differential equations. For entropy satisfaction and generally stable approximation the upwind approximation is quite fundamental as the foundation of robust numerical methods for fluid dynamics. We can start with the basic upwind approximation with classical function, the absolute value in this case. We will base the analysis on the semi-discrete version of the scheme using a flux difference, $\frac{1}{h} \left[ f(j+1/2) – f(j-1/2) \right]  $. The basic upwind approximation is $ – \frac{1}{2} \left|a\right| \left[ u(j+1)  –  u(j)  \right]$ where is the characteristic velocity for the flux and provides the upwind dissipation. The Taylor series analysis gives the dissipation in this scheme as . We find a set of very interesting conclusions can be drawn almost immediately.

All of the smoothed functions introduced are workable as alternatives although some versions seem to be intrinsically better. In other words all produce a valid consistent first-order approximation. The functions based on analytical functions like or are valid approximations, but the amount of dissipation is always less than the classical function leading to potential entropy violation. They approach the classical absolute value as one would expect and the deviations similarly diminish. Functions such as or result in no change in the leading order truncation error although similarly the deviations are always produce less dissipation than classical upwind. We do find that for both functions we need to modify the form of the regularization to get good behavior to and .For the classical softmax function based on logarithms and exponentials is the same, but it always produces more dissipation than upwinding rather than less, . This may make this functional basis better for replacing the absolute value for the purpose of upwinding. The downside to this form of the absolute value is the regularized sign function’s passage through hard zero, which makes division problematic.

Let’s look at the functions useful for producing a more entropy satisfactory result for upwinding.  We find that these functions work differently than the original ones. For example the hyperbolic tangent does not as quickly become equivalent to the upwind scheme as . There is a lingering departure from linearity with proportional to the mesh spacing and . As a result the quadratic form of the softened sign is best because of the regularization. Perhaps this is a more widely applicable conclusion as will see as we develop the smoothed function more with limiters.

Where utility ends and decoration begins is perfection.

― Jack Gardner

Now we can transition to looking at a more complex and subtle subject, limiters. Briefly put, limiters are nonlinear functions applied to differencing schemes to produce non-oscillatory (or monotone solutions) with higher order accuracy. Generally in this context high-order is anything above first order. We have theory that confines non-oscillatory methods to first-order accuracy where upwind differencing is canonical. As a result the basic theory applies to second-order method where a linear basis is added to the piecewise constant basis the upwind method is based on. The result is the term “slope limiter” where the linear, slope, is modified by a nonlinear function. Peter Sweby produced a diagram to describe what successful limiters look like parametrically. The parameter is non-dimensionally described by the ratio of discrete gradients, $ r = \frac{(u(j+1) – u(j)}{u(j) – u(j-1)}$. The smoothed functions described here modify the adherence to this diagram. The classical diagram has a region where second-order accuracy can be expected. It is bounded by the function and twice the magnitude of this function.

We can now visualize the impact of the smoothed functions on this diagram. This produces systematic changes in the diagram that lead to deviations from the ideal behavior. Realize that the ideal diagram is always recovered in the limit as the functions recover the classical form. What we see is that the classical curves are converged upon from above or below, and produces wiggles in the overall functional evaluation. My illustrations all show the functions with the regularization chosen to be unrealistically small to exaggerate the impact of the smooth functions. A bigger and more important question is whether the functions impact the order of approximation.

To finish up this discussion I’m going to look at analyzing the truncation error of the methods. Our starting point is the classical scheme’s error, which provides a viewpoint on the nature of the nonlinearity associated with limiters. What is clear about a successful limiter is its ability to produce a valid approximation to a gradient with an ordered error of a least order . The minmod limiter produces a truncation error of $ \mbox{minmod}(u(j)-u(j-1), u(j+1) – u(j)) = u_x – \frac{h}{2} \left| \frac{u_{xx}}{u_x} \right| u_x $. The results with different sorts of recipes for the smoothed sign function and its extension to softabs, softmin and softmax are surprising to say the least and a bit unexpected.

Here is a structured summary of the options as applied to a minmod limiter, :

 

1. and . The gradient approximation is . The constant in front of the gradient approaches one very quickly as grows.
2. and . The gradient approximation is . The constant in front of the gradient approaches one very slowly as grows. This is smoothing is unworkable for limiters.
3. and putting a mesh dependence in the sign function results in inconsistent gradient approximations. The gradient approximation is . The leading constant goes slowly to one as .
4. and . The gradient approximation is .
5. and . putting a mesh dependence in the sign results in inconsistent gradient approximations. The gradient approximation is . This makes the approximation utterly useless in this context.
6. and . The gradient approximation is $ u_x \approx u_x – \left[ u_x\sqrt{n^2 + \left(\frac{(u_{xx}}{u_x}} \right)^2 \right]h+ \cal{O}(h^2)$.

Being useful to others is not the same thing as being equal.

― N.K. Jemisin

Sweby, Peter K. “High resolution schemes using flux limiters for hyperbolic conservation laws.” SIAM journal on numerical analysis 21, no. 5 (1984): 995-1011.

Everything that looks too perfect is too perfect to be perfect.

― Dejan Stojanovic

This post is going to delve directly into this blog’s name, how to regularize a singularity, but in this case we are talking about an artificial one. When one is writing a computer program to solve differential equations it is easy to introduce a discontinuity into how the program operates. This is particularly true when you’re implementing an elaborate model or numerical method. In the process of taking care of special cases or making a program robust for general use, logic is employed. Should a circumstance arise that causes the program to fail, it can be detected and avoided by making a logical change in the operation. The most common way to do this uses logic in the program through an “if” statement or some sort of switch. When the “if” triggers on a floating-point value, the impact on the solution can be subtle and creates a host of practical issues.

Ability to find the answers is more important than ability to know the answers.

― Amit Kalantri

As computer program development becomes more rigorous testing of various sorts becomes important and valuable to quality of the work. One form of the testing is regression testing. Here the program is run through a series of usually simple problems with well-defined answers. If the program’s solution changes in some way that is unexpected, the testing should pick it up and alert the development team. In addition this testing often runs across a bunch of different computers to make sure the answers are the same, and the program works properly on all of them. For basic quality assessment and control, regression testing is essential. It is one of the hallmarks of serious, professional code development. The logic and if statements introducing discontinuous behavior into the code based on floating point numbers can wreck havoc with the testing! We can end up with a situation where the tests produce much different results because of infinitesimal changes in the numbers at some point in the calculation.

You might be asking how this can happen? This all seems rather disturbing, and it is. It is simple matter whenever the logical decision is made on the basis of a floating-point number. Consider a simple, but common bit of (pseudo) computer code,

if (value > 0.0) then

newvalue = value1

else

newvalue = value2

endif

which is a very simple test that one might see with upwind finite differences. In some cases a logical switch like this might trigger an elaborate mathematical expression, or even call a very different function or subroutine. Consider the case where the “value” is very near zero, but not exactly. In this case small differences in the quantity “zero” will trigger completely different evaluations of the logic. For special values (especially zero) this happens all the time. If the solution is dependent upon a certain sequence any regression test depending on this test could differ based on inconsequential numerical differences. As programs become more complex these branches and differences explode. Code development teams relying upon regression testing end up chasing this sort of problem over and over. It becomes a huge drain on the productivity and quality.

These problems can be replicated with a set of standard functions. The logic above can be replaced by a single statement using a “sign” function,

newvalue = sign(value) * value1 + 0.5*(1.0 – sign(value)) * value2

which gives exactly the same result as the if test in the previous paragraph. It is also prone to exactly the same problems in practical testing. These issues are the tip of a proverbial iceberg. It isn’t just regression testing that suffers from these issues, if the method for solution involves solving a nonlinear equation that goes through the above logic, the solution can stall and stagnate causing solution accuracy to suffer. The same switches can produce breaks in symmetry or bifurcation of solutions near critical points. Next, I will describe ways of implementing the sign function to alleviate these problems. It turns out that there is a whole family of functions that can replace the discontinuous behavior with something continuous, and the sign function can be used to construct other functions with the same switching behavior built in. I’ve written about some of these functions before in a different context where discontinuous logical functions were replaced by differentiable functions for the purpose of conducting modified equation analysis that rely upon valid Taylor series expansions.

Here are a couple of times I’ve hit upon this topic before: https://williamjrider.wordpress.com/2016/06/07/the-marvelous-magical-median/, https://williamjrider.wordpress.com/2015/08/17/evolution-equations-for-developing-improved-high-resolution-schemes-part-2/, https://williamjrider.wordpress.com/2016/06/22/a-path-to-better-limiters/. This post might be a good postscript to these because the techniques here can cure some of the practical ills remaining for these rather powerful methods. We see issues with solving nonlinear equations where limiters are used in discretizations, various symmetry breaking effects, and extreme sensitivity to initial conditions. As Iwill touch upon at the very end of this post, Riemann solvers-numerical flux functions can also benefit from this, but some technicalities must be proactively dealt with.

Slow is smooth; smooth is fast.

― Jack Coughlin

Using the sign function we can systematically remove the switching behavior that plagues regression testing, nonlinear solutions, symmetry preservation, and extreme sensitivity to initial conditions.

For me, the first function to start this was the hyperbolic tangent function. The basic behavior of this function acts to create a switch between two states based on the argument and steepness of the transition, as becomes larger, the function approaches the idealized step function. It turns out that there are a number of smooth functional representations of the sign function including , , and . There are many others that can be derived as well as other more exotic functions. These functions are used in other fields to remove the discontinuity from a switching function (https://en.wikipedia.org/wiki/Sigmoid_function ).

These functions provide a flexible foundation to build upon. As an initial example take the definition of the absolute value, (https://en.wikipedia.org/wiki/Sign_function ). This can be rearranged in a number of useful forms, . We can see that a simple smoothed version of the absolute value is . We can now build an entire family of softened or smoothed functions that can be differentiated (they are ). Each of the classical versions of these functions cannot be differentiated everywhere and create a host of problems in practical programs. Another common switching function are “min” and “max”. We can rewrite both functions as , and . The modified smooth versions are relatively obvious, , and . From this basic set of function we can build the backbone of the limiters, the minmod function and the marvelous magical median function. What we have removed in the process is the discontinuous switching process that can wreck havoc with finite precision arithmetic.

We note that there is a separate version of the softmin and softmax functions used in some optimization solutions (https://www.johndcook.com/blog/2010/01/13/soft-maximum/, https://en.wikipedia.org/wiki/Softmax_function, ). This uses a combination of exponentials and logrithmns to provide a continuously differentiable way to take the maximum (or minimum) of a set of arguments. My naming convention “soft” comes from being introduced to the ideas in this blog post. This separates the idea from a “hard” max where the arguments switch based on the precision of the floating-point numbers as opposed to being continuous. For completeness the softmax uses the following expression , which may be expanded to additional arguments without complications. By the same token we can define a “softmin” as that can be similarly be expanded to more arguments. In both cases the parameter controls the sharpness of the smoothed version of the standard function, the larger the value the closer the function is to the standard function.

Using our previous definitions of “softmax” we can derive a new version of “softabs”.   We rearrange to derive a . We start with the observation that therefore . We find that this version of the absolute value has much different properties than the previous softened version in some important ways. The key thing about this version of absolute value is always greater in value than the classical absolute value function. This would turn out be useful for use with Riemann solvers by not violating the entropy condition. With appropriate wavespeed estimates the entropy condition will be satisfied (wavespeed estimates are out of scope for this post). By the same token this absolute value is not valuable for limiters because of the same property!

Ultimately we want to understand whether these functions alter the basic accuracy-consistency or stability properties of the numerical methods based on using the classical functions. The answer to this question is subtle, but can be answered via analysis and numerical experiment. Not to belabor the details, but we use series expansions and discover that with appropriate regularization of the smoothed functions, we can use them to replace the classical functions and not undermine the accuracy of the discretization. This has been confirmed for the softened version of the “minmod” limiter. A downside of the softened limiters are small deviations from idealistic monotonticity preserving behavior.

Finally as eluded to early in the post, we can also use these functions to modify Riemann solvers. The first code example can form the logical basis for upwind bias with a finite difference by choosing a one-sided difference based upon the sign of the characteristic velocity. When Riemann solvers are examined we see that either “if” statements are used or when full flux functions are used absolute values (the flux is in a general sense a characteristic quantity multiplied by the characteristic velocity), the absolute value of the characteristic velocity introduces the sign convention the “if” statement provides.

The lingering problem with this approach is the concept of entropy violating approximations. This issue can easily be explained by looking at the smooth sign function compared with the standard form. Since the dissipation in the Riemann solver is proportional to the characteristic velocity, we can see that the smoothed sign function is everywhere less than the standard function resulting in less dissipation. This is a stability issue analogous to concerns around limiters where these smoothed functions are slightly more permissive. Using the exponential version of “softabs” where the value is always greater than the standard absolute value can modulate this permissive nature.

Let us study things that are no more. It is necessary to understand them, if only to avoid them.

― Victor Hugo

 

It doesn’t matter how beautiful your theory is … If it doesn’t agree with experiment, it’s wrong.

― Richard Feynman

Bill’s corollary: It doesn’t matter how massive your calculation is … If it doesn’t agree with experiment, it’s wrong.

The real world is complex, dangerous and expensive. It is also where mystery lives and the source of knowledge. There seems to be some desire to use computers, modeling an
d simulation to replace our need for dealing with the real world. This is untenable from many different perspectives and misplaces the proper role of everything possible via computing. Worse yet, computing can not be a replacement for reality, but rather is simply a tool for dealing with it better. In the final analysis the real world still needs to be in the center of the frame. Computing needs to be viewed in the proper context and this perspective should guide our actions in its proper use.

Experiment is the sole source of truth. It alone can teach us something new; it alone can give us certainty.

― Henri Poincaré

We see the confluence of many things in our attitudes toward computing. It is a new thing constantly unveiling new power and possible ways of changing our lives. In many ways computing is driving enormous change societally and creating very real stress in the real world. These stresses are stoking fears and lots of irrational desire to control dangers and risks. All of this control is expensive, and drives an economy of fear. Fear is very expensive. Trust, confidence and surety are cheap and fast. One totally irrational way to control fear is ignore it, allowing reality to be replaced. For people who don’t deal with reality well, the online world can be a boon. Still the relief from a painful reality ultimately needs to translate to something tangible physically. We see this in an over-reliance on modeling and simulation in technical fields. We falsely believe that experiments and observations can be replaced. The needs of the human endeavor of communication can be done away with through electronic means. In the end reality must be respected, and people must be engaged in conversation. Computing only augments, but never replaces the real world, or real people, or real experience. This perspective is a key realization in making the best use of technology.

The real world is where the monsters are.

― Rick Riordan

In science we must always remember that understanding reality is the fundamental objective. Theory acts to explain what we see, but observation always rules supreme in defining the validity of knowledge and understanding. We must always remember that computing is a tool that augments theory. It never replaces theory, nor can it replace experiments or observation. A computational simulation can never be better than the model that theory has provided it. If the theory is lacking (and it always is), more computing cannot rescue it. No amount of computing can fill in the gap between what is and isn’t known. It is a new and powerful tool to be wielded with care and skill, but a tool. These perspectives seem to be lost on so many people who see computing as some sort of silver bullet that transcends these simple truths.

It is sometimes an appropriate response to reality to go insane.

― Philip K. Dick

While computing isn’t a silver bullet for making painful elements of reality go away, it is a powerful tool if wielded properly. Modeling and simulation serves as a powerful means of testing our knowledge and general capability to understand the world around us. When simulations are tested against reality and produce good results (that is they are validated), we feel that our grasp of the how’s and why’s of the real world are at hand. If we are grounded in this understanding, the modeling and simulation can aid our ability to examine the World around us. We can optimize our observations or design experiments to more effectively examine and measure various things. A successful model can serve a wonderful role in focusing our attention toward the most important aspects of reality, or ignoring what is not essential.

More than simply assisting the design of better experiments and observations of reality, the use of modeling and simulation can provide a significant flywheel effect. All the models of reality we use are flawed at some level. In a similar vein, our observations of reality are always limited and flawed. In very good models these flaws are subtle and hard to expose. Good experiments need to be designed to expose and improve these models. We can achieve some stunning synergies if we utilize the models to design the most stringent tests of them. This is exactly the thing we can do with a well-designed program that collaborates effectively. If we examine the models we can find the parts of a physical system most sensitive to the impact of parts of the model. One way of proactively improving models is to identify where to make measurements, and what to measure to maximize the ability to prove, disprove or improve a given model. The key point is the oft-missed point that the models are always imperfect.

Reality is that which, when you stop believing in it, doesn’t go away.

― Philip K. Dick

These imperfections are rarely acknowledged in the current National dialog on high performance computing. Rather than state this rather powerful truth, we see a focus on computer power coupled to an unchanging model as the recipe for progress. Focus and attention to improving modeling is almost completely absent in the modeling and simulation world. This ignores one of the greatest truths in computing that no amount of computer power can rescue an incorrect model. These truths do little to alter the approach although we can be sure that we will ultimately pay for the lack of attention to these basics. Reality cannot be ignored forever; it will make itself felt in the end. We could make it more important now to our great benefit, but eventually our lack of consideration will demand more attention.

A more profitable a proactive strategy would benefit everyone. Without attention many end up accommodating the model’s imperfections through heavy use of calibration. Ultimately the calibration hammer is lowered on imperfect models to render them useful and capable of influencing reality. In the wake of heavy-handed calibration we can achieve a great focus on localizing the modeling issues. In a deep sense the areas for crude calibration (often crude and very effective) are exactly the places for the greatest modeling improvement. Typically the calibration ends up merging multiple issues together. As a result one needs to carefully deconstruct the whole of the effects being accounted for in calibration. For example one may find a single calibration knob accounting for the effects of turbulence, inadequate constitutive relations and mesh resolution. To make progress these effects need to be separated and dealt with independently. The proper decomposition of error allows the improvement of modeling in a principled manner.

The key to utilizing simulation effectively is the recognition of what it can and cannot do. While one can experiment with computations, these experiments can only unveil secrets of the models or computations themselves. The capacity of such unveiled secrets to be meaningful in reality always involves direct comparison with observations of the real world. If the secret seen computationally is also seen in reality then a true discovery can be made. In the process the model gains credibility and validity as well. In these cases simulation and modeling can tell us where to look, and if the secret is found, we know the model is valuable and correct. If it is not found, we know the model is deficient and must be improved. The observations may or may not be sufficient for improving the model in such a way that its predictions are validated by reality.

Successful modeling and simulation implies a level of understanding that empowers humanity. The implication of understanding goes to our ability to control reality effectively through human action. If reality can be modeled its effects can be affected or accommodated through design or mitigation. The definition of success is always through validation of the model’s results against observations of the world (including carefully designed experiments). If the model can be demonstrated via verification to be solving the model we believe we are using, the validation is powerful evidence. One must recognize that the degree of understanding is always relative to the precision of the questions being asked. The more precise the question being asked is, the more precise the model needs to be. This useful tension can help to drive science forward. Specifically the improving precision of observations can spur model improvement, and the improving precision of modeling can drive observation improvements, or at least the necessity of improvement. In this creative tension the accuracy of solution of models and computer power plays but a small role.

Any physical theory is always provisional, in the sense that it is only a hypothesis: you can never prove it. No matter how many times the results of experiments agree with some theory, you can never be sure that the next time the result will not contradict the theory.

― Stephen Hawking

 

Sometimes, these tribal affiliations push us to become better versions of ourselves. We take a long-term view, check our selfish impulses and work hard to meet the high standards of those around us.

– Seth Godin

Sometimes you read something that hits you hard. Yesterday was one of those moments while reading Seth Godin’s daily blog post (http://sethgodin.typepad.com/seths_blog/2017/03/the-best-of-us-the-worst-of-us.html). I’ve enjoyed Seth’s books and ideas finding them easy to grasp and connect to. Like a lot of things. The point of the post was simple. Our associations impact us. They can bring out the best or worst in us. When I reflected on this point, the quote above came into sharp focus. Looking at my current work the quote seemed almost cruel. It was completely opposite of everything driving me today. Such a circumstance is ultimately untenable.

Writers like Godin often speak of aspirations for better World, a better workplace that makes all of us better. My managers read these books all the time (Daniel Pink’s book “Drive” comes to mind). I’ve opined that the distance between the workplace espoused in these books and where I work is vast. The management seems to be actively working to make things worse and worse. On the other hand they are always reading these books or going to the Harvard Business Review for advise. Do they really think that they are applying anything to their actual work? It would seem to me that they are completely delusional if they think their actions follow from any of this advise.

I once worked somewhere that pushed me to be better. It was a wonderful place where I grew professionally every day. The people there were generous with their knowledge. Collaboration was encouraged. It was also a rough and tough place to work. The culture was aggressive and combative. There was plenty of bad behavior and conflict. Nonetheless it was an incubator for me. It changed me for the better and filled me with desire to improve. It was also a place that had run out of time so we systematically destroyed it. Perhaps it was a place that can’t exist in today’s world, but it would be good to create places like it that can. We should all aspire to create places that make us better, that help us grow into the best versions of ourselves.

I rewrote Godin’s quote to reflect how work is changing me (at the bottom of the post). It really says something needs to give. I worry about how many of us feel the same thing. Right now the workplace is making me a shittier version of myself. I feel that self-improvement is a constant struggle against my baser instincts. I’m thankful for a writer like Seth Godin who can push me to into a vital and much needed self-reflective “what the fuck” !

Sometimes, these tribal affiliations push us to become worse versions of ourselves. We take a short-term view, give into our selfish impulses and become lazy to meet the low standards of those around us.

 

Management is doing things right; leadership is doing the right things.

― Peter F. Drucker

It’s a really incredible time to be alive. The world is going through tremendous changes in many respects. Much of the change is driven by technology and scientific breakthroughs of the past century. One might reasonably argue that the upheavals we are witnessing today are the most important since the Renaissance and the Reformation. We are seeing cultural, economic, and political changes of epic proportions across the human world. With the Internet forming a backbone of immense interconnection, and globalization, the transformations to our society are stressing people resulting in fearful reactions. These are combining with genuine threats to humanity in the form of weapons of mass destruction, environmental damage, mass extinctions and climate change to form the basis of existential danger. We are not living on the cusp of history; we are living through the tidal wave of change. There are massive opportunities available, but the path is never clear or safe. As the news every day testifies, the present mostly kind of sucks. While I’d like to focus on the possibilities of making things better, the scales are tipped toward the negative backlash to all this change. The forces trying to stop the change in its tracks are strong and appear to be growing stronger.

People in any organization are always attached to the obsolete – the things that should have worked but did not, the things that once were productive and no longer are.

― Peter F. Drucker

Many of our institutions are under continual assault by the realities of today. The changes we are experiencing are incompatible with many of our institutional structures such as the places I work. Increasingly this assault is met with fear. The evidence of the overwhelming fear is all around us. It finds its clearest articulation within the political world where fear-based policies abound with the rise of Nationalist anti-Globalization candidates everywhere. We see the rise of racism, religious tensions and protectionist attitudes all over the World. The religious tensions arise from an increased tendency to embrace traditional values as a hedge against change and the avalanche of social change accompanying technology, globalization and openness. Many embrace restrictions and prejudice as a solution to changes that make them fundamentally uncomfortable. This produces a backlash of racist, sexist, homophobic hatred that counters everything about modernity. In the workplace this mostly translates to a genuinely awful situation of virtual paralysis and creeping bureaucratic over-reach resulting in a workplace that is basically going no where fast. For someone like me who prizes true progress above all else, the workplace has become a continually disappointing experience.

All organizations are perfectly designed to get the results they are now getting. If we want different results, we must change the way we do things.

― Tom Northup

One of the most prevalent aspects of today’s world is the focus on appearances as opposed to substance. As we embrace online life and social media, we have gotten supremely fixated on superficial appearances and lost the ability to focus on substance. The way things look has become far more important than the actuality of anything. Having a reality show celebrity as the President seems like a rather emphatic exemplar of this trend. Someone who looks like a leader, but lacks most of the basic qualifications is acceptable to many people. People with actual qualifications are viewed as suspicious. The elite are rejected because they don’t relate to the common man. While this is obvious on a global scale through political upheaval, the same trends are impacting work. The superficial has become a dominant element in managing because the system demands lots of superficial input while losing any taste for anything of enduring depth. Basically, the system as a whole is mirroring society at large.

Management cares about only one thing. Paperwork. They will forgive almost anything else – cost overruns, gross incompetence, criminal indictments – as long as the paperwork’s filled out properly. And in on time.

― Connie Willis

There is nothing so useless as doing efficiently that which should not be done at all.

― Peter F. Drucker

Working within one of our “prized” National institutions has been an interesting, magical and initially empowering experience. Over the past decade or two, these institutions have been dragged down by the broader societal trends into the muck. It is no exaggeration to say that we are being slowly and surely strangled by overwhelming management oversight. The basic recipe for management of the Labs I’ve worked at is making lots and lots of rules to keep people from “fucking up”. The bottom line is that it’s fine if we really don’t accomplish anything as long as people just don’t ever fuck up. The maxim at work is don’t ever fuck up, which is the result of fear being the core motivation for everything. All of our most important institutions are withering under society-wide loss of faith and mistrust. This creates an environment where any scandal can be a direct threat to the future of the institution. This direct threat means that achievement and the very reason for the institution’s existence are lost.

The goal of management is to remove obstacles.

― Paul Orfalea

The prime institutional directive is survival and survival means no fuck ups, ever. We don’t have to do anything as long as no fuck ups happen. We are ruled completely by fear. There is no balance at all between fear-based motivations and the needs for innovation and progress. As a result our core operational principle for is compliance above all else. Productivity, innovation, progress and quality all fall by the wayside to empower compliance. Time and time again decisions are made to prize compliance over productivity, innovation, progress, quality, or efficiency. Basically the fear of fuck ups will engender a management action to remove that possibility. No risk is ever allowed. Without risk there can be no reward. Today no reward is sufficient to blunt the destructive power of fear.

Our management has become all about no fuck ups, and appearances. The actual, good, productive management work that should be done is virtually entirely absent. We don’t see managers trying to figure out how to clear barriers or enable people to get work done. We see enforced compliance. We hear lots of things about formality of operations and assurance of results. This all comes down to pervasive lack of trust and fear of failure. Increasingly we can fake progress and results. Increasingly bullshit has taken the place of actual results. Even better, bullshit results are safe and entail far less risk of fuck ups. They are mostly upside without the downside, plus bullshit is in vogue! It has the benefit of sounding better than anything we are likely to achieve, and doesn’t carry the risks of real work. The end result is deep-seated corrosive forces unleashed within our institutions that are eating away at them from the inside.

The over-management is joined at the hip with a lack of leadership and direction. It is the twin force for professional drift and institutional destruction. Working at an under-led institution is like sleepwalking. Every day you go to work basically making great progress at accomplishing absolutely nothing of substance. Everything is make-work and nothing is really substantive you have lots to do because of management oversight and the no fuck up rules. You make up results and produce lots of spin to market the illusion of success, but there is damn little actual success or progress. The utter and complete lack of leadership and vision is understandable if you recognize the prime motivation of fear. To show leadership and vision requires risk, and risk cannot take place without failure and failure courts scandal. Risk requires trust and trust is one of the things in shortest supply today. Without the trust that allows a fuck up without dire consequences, risks are not taken. Management is now set up to completely control and remove the possibility of failure from the system.

Leadership and learning are indispensable to each other.

― John F. Kennedy

The capacity to achieve rewards and achievement without risk is incompatible with experience. Everyday I go to work with the very explicit mandate to do what I’m told. The clear message every day is never ever fuck up. Any fuck ups are punished. The real key is don’t fuck up, don’t point out fuckups and help produce lots of “alternative results” or “fake breakthroughs” to help sell our success. We all have lots of training to do so that we make sure that everyone thinks we are serious about all this shit. The one thing that is absolutely crystal clear is that getting our management stuff correct is far more important than every doing any real work. As long as this climate of fear and oversight is in place, the achievements and breakthroughs that made our institutions famous (or great) will be a thing of the past. Our institutions are all about survival and not about achievement. This trend is replicated across society as a whole; progress is something to be feared because it unleashes unknown forces potentially scaring everyone. The fear resulting in being scared undermines trust and without trust the whole cycle re-enforces itself.

Leaders must be close enough to relate to others, but far enough ahead to motivate them.

― John C. Maxwell

Along with progress, leadership is also sacrificed at the altar of fear. Anything out of the ordinary is completely suppressed in the current environment. The ordinary can be managed and controlled; it is a known quantity. Progress and innovation produces unusual things that might have unintended consequences making its management difficult. Something unusual is more likely to produce a fuck up and therefore it must be stopped to assure the survival imperative. Of course, innovation, progress, and the unusual can also be wonderful and produce the breakthroughs all of us celebrate. The problem is that this cannot take place without risk and the potential for things to get fucked up. This also holds for people, who also must be ordinary, the unusual that might lead us in new directions are to be feared and controlled. The unusual is dangerous and feared. Leaders are unusual, so they too are reviled.

Start with the end in mind.

― Stephen R. Covey

A big piece of the puzzle is the role of money in perceived success. Instead of other measures of success, quality and achievement, money has become the one-size fits all measure of the goodness of everything. Money serves to provide the driving tool for management to execute its control and achieve broad-based compliance. You only work on exactly what you are supposed to be working on. There is no time to think or act on ideas, learn, or produce anything outside the contract you’ve made with you customers. Money acts like a straightjacket for everyone and serves to constrict any freedom of action. The money serves to control and constrain all efforts. A core truth of the modern environment is that all other principles are ruled by money. Duty to money subjugates all other responsibilities. No amount of commitment to professional duties, excellence, learning, and your fellow man can withstand the pull of money. If push comes to shove, money wins. The peer review issues I’ve written about are testimony to this problem; excellence is always trumped by money.

One of the things that are mostly acutely impacted by all of this is the ability for strategic thought, work or action. In the wake of the lack of trust and degree of control, the ability to do big things is almost completely lost. All work becomes unremittingly tactical in nature. Big ideas are lost and people can only envision committing to small things. Big ideas require a level of trust that cannot be summoned or supported. An element in this lack of trust is an obsession with reporting and careful attention to progress by the management. We see rather extensive draws of information from the depths of organizations to check on whether money is being spent properly. The entire management apparatus is engaged in getting information, but nothing is done with it. It is only used to check up on things, the whole of the management is devoted to attending to the trustworthiness of those working. The good that management might do is scarified, and leadership is completely absent. Without some symmetry of trust, the whole idea of leadership is vacant.

What the hell is to be done about all of this? How do we recapture progress and reject fear? How do we embrace leadership and harness management as a force for good rather than decline and decay?

I really don’t know the answer to any of these questions, but I can propose a few things that might resist these issues. Hopefully a large number of people will join together in prizing progress enough to reject fear as a prime societal motivator. The desire to live and grow will overthrow the fear of change. The forces of fear have the potential to undo so much of the good of the modern World. Those who prize modernity and the benefits of freedom and progress will reject fear as a motivator. Realizing that fear emboldens hatred and reduces the potential for good is a first step. We must recognize and reject our so-called leaders who utilize fear as a prime motivation. Every time a leader uses fear to further their agenda, we take a step backward. One the biggest elements in this backwards march is thinking that fear and danger can be managed. Danger can only be pushed back, but never defeated. By controlling it in the explicit manner we attempt today, we only create a darker more fearsome danger in the future that will eventually overwhelm us. Instead we should face our normal fears as a requirement of the risk progress brings. If we want the benefits of modern life, we must accept risk and reject fear. We need actual leaders who encourage us to be bold and brave instead of using fear to control the masses. We need to quit falling for fear-based pitches and hold to our principles. Ultimately our principles need to act as a barrier to fear becoming the prevalent force in our decision-making.

People who don’t take risks generally make about two big mistakes a year. People who do take risks generally make about two big mistakes a year.

― Peter F. Drucker

 

 

Excellence does not come from believing in excellence, but from constant change, challenge, and improvement.

― Jeffrey Fry

Everyone wants his or her work or work they pay for to be high quality. The rub comes when you start to pay for the quality you want. Everyone seems to want high quality for free, and too often believes that low cost quality is a real thing. Time and time again it becomes crystal clear that high quality is extremely expensive to obtain. Quality is full of tedious detail oriented work that is very expensive to conduct. More importantly when quality is aggressively pursued, it will expose problems that need to be solved to reach quality. For quality to be achieved these problems must be addressed and rectified. This ends up being the rub, as people often need to stop adding capability or producing results, and focus on fixing the problems. People, customer and those paying for things tend to not want to pay for fixing problems, which is necessary for quality. As a result, it’s quite tempting to not look so hard at quality and simply do more superficial work where quality is largely asserted by fiat or authority.

Trouble cannot be avoided, you either go looking for it or it will come looking for you.

― Constance Friday

The entirety of this issue is manifested in the conduct of verification and validation in modeling and simulation. Doing verification and validation is a means of high quality work for modeling and simulation. Like other forms of quality work, it can be done well engaging in details and running problems to ground. Thus V&V is expensive and time consuming. These quality measures take time and effort away from results, and worse yet produce doubt in the results. As a consequence the quality mindset and efforts need to have significant focus and commitment, or they will fall by the wayside. For many customers the results are all that matters, they aren’t willing to pay for more. This becomes particularly true if those doing the work are willing to assert quality without doing the work to actually assure it. In other words the customer will take work that is asserted to be high quality based on the word of those doing the work. If those doing the work are trying to do this on the cheap, we produce low or indeterminate quality work, sold as high quality work masking the actual costs.

The reality is that we do have that problem. Work of unknown quality is being asserted as high quality without the evidence of the high quality ever being produced. Generally the evidence is simply the authority of a “trusted” code user.

The largest part of the issue is the confluence of two terrible trends: increasingly naïve customers for modeling and simulation and decreasing commitment for paying for modeling and simulation quality. Part of this comes from customers who believe in modeling and simulation, which is a good thing. The “quality on the cheap” simulations create a false sense of security because it provides them financial resources. Basically we have customers who increasingly have no ability to tell the difference between low and high quality work. The work’s quality is completely dependent upon those doing the work. This is dangerous in the extreme. This is especially dangerous when the modeling and simulation work is not emphasizing quality or paying for its expensive acquisition. We have become too comfortable with the tempting quick and dirty quality. The (color) viewgraph norm that used to be the quality standard for computational work that had faded in use is making a come back. A viewgraph norm version of quality is orders of magnitude cheaper than detailed quantitative work needed to accumulate evidence. Many customers are perfectly happy with the viewgraph norm and naïvely accept results that simply look good and asserted as high quality.

Instead of avoiding criticisms, make criticisms work for you.

― Aniekee Tochukwu Ezekiel

Perhaps an even bigger issue is the misguided notion that the pursuit of high quality won’t derail plans. We have gotten into the habit of accepting highly delusional plans for developing capability that do not factor in the cost of quality. We have allowed ourselves to bullshit the customer to believing that quality is simple to achieve. Instead the pursuit of quality will uncover issues that must be dealt with and ultimately change schedules. We can take the practice of verification as an object lesson in how this works out. If done properly verification will uncover numerous and subtle errors in codes such as bugs, incorrect implementations, boundary conditions, or error accumulation mechanisms. Sometimes the issues uncovered are deeply mysterious and solving them requires great effort. Sometimes the problems exposed require research with uncertain or indeterminate outcomes. Other times the issues overthrow basic presumptions about your capability that require significant corrections in large-scale objectives. We increasingly live in a world that cannot tolerate these realities. The current belief is that we can apply project management to the work, and produce high quality results that ignore all of this.

The way that the trip down to “quality hell” starts is the impact of digging into quality. Most customers are paying for capability rather than quality. When we allow quick and dirty means of assuring quality to be used, the door is open for the illusion of quality. For the most part the verification and validation done by most scientists and engineers is the quick, dirty and incomplete variety. We see the use of eyeball or viewgraph norm pervasively in comparing results in both verification and validation. We see no real attempt to grapple with the uncertainties in calculations or measurements to put comparisons in quantitative context. Usually we see people create graphics that have the illusion of good results, and use authority to dictate that these results indicate mastery and quality. For the most part the scientific and engineering community simply gives in to the authoritative claims despite a lack of evidence. The deeper issue with the quick and dirty verification is the mindset of those conducting it; they are working from the presumption that the code is correct instead of assuming there are problems, and collecting evidence to disprove this.

The core issue in quality is the accumulation of evidence pro and con with the courage to accept what that evidence shows. As usual, the greater the evidence the better the work is. Graphical work is useful and powerful only when it is backed up by quantitative work. The quantitative work is the remedy for the qualitative, eyeball, viewgraph, and color video metric so often used today. Deep quantitative studies show the sort of evidence that cannot be ignored. If the results are good, the evidence of quality is strong. If a problem is found, the need for remedy is equally strong. In validation or verification the creation of an error bar goes a long way to putting any quality discussion in context. The lack of an error bar casts any result adrift and lacking in context. A secondary issue would be the incomplete work where full error bars are not pursued, or results that are not favorable are not pursued or worse yet, suppressed.

One key aspect of verification testing is the high degree of technical knowledge needed to conduct it in a fully professional and complete manner. The gap between quick and dirty verification using plot overlays, and various forms of the viewgraph norm, and fully well executed quantitative verification is vast. Orders of magnitude level of effort stands between the quick check producing a plot overlay and a fully quantitative study including rates of convergence. The plot overlay can be produced without consideration for deep fields of technical knowledge that verification taps into. Many of the code development communities at our leading institutions are quite accepting of the quick and dirty approach as adequate. These communities produce little or no push back and the people paying for the code development are too naïve to know the difference. The quick and dirty approach has the virtue of providing a basic qualitative check on the code and is quite capable of unveiling catastrophic problems in the code. One has checked the ability of the code to get some semblance of a solution to a problem with an analytical solution. The subtle issues that must be solved for high quality have lots of room to hide, and sometimes hide for decades.

At the same time the full benefit of the analytical solution and verification is clouded by the lack of quantitative rigor. Moreover the full benefit of verification is lost in the sloppy standard of acceptance for quick and dirty verification. Serious bugs in a code can go completely unnoticed where the full quantitative approach would unmask the bugs for removal by the development teams. If acceptance standards are weak, the quick and dirty approach is accepted as giving the appearance of verification in the absence of its professional level of practice. Validation practices tap into similar differences in execution cost and sophistication in terms of effort and cost.

Nothing in this world is harder than speaking the truth, nothing easier than flattery.

― Fyodor Dostoyevsky

What really happens when the feedback from our customers does not accept the steps necessary for quality? We are starting to see this unfold around us. The forces behind “alternative facts” are driven by this lack of willingness to deal with reality. Why deal with truth and the reality of real problems when we can just define them away with more convenient facts. In today’s world we are seeing a rise of lies, bullshit and delusion all around us. As a result, we are systematically over-promising and under-delivering on our work. We over-promise to get the money, and then under-deliver because of the realities of doing work one cannot get maximum capability with maximum quality for discount prices. Increasingly bullshit (propaganda) fills in the space between what we promise and what we deliver. Pairing with this deep dysfunction is a systematic failure of peer review within programs. Peer review has been installed as backs stop again the tendencies outlined above. The problem is that too often peer review does not have a free reign. Too often with have conflicts of interest, or control that provide an explicit message that the peer review had better be positive, or else.

We bring in external peer reviews filled with experts who have the mantle of legitimacy. The problem is that these experts are hired or drafted by the organizations being reviewed. Being too honest or frank in a peer review is the quickest route to losing that gig and the professional kudos that goes along with it. One bad or negative review will assure that the reviewer is never invited back. I’ve seen it over and over again. Anyone who provides an honest critique is never seen again. A big part of the issue is that the reviews are viewed as pass-fail tests and problems uncovered are dealt with punitively. Internal peer reviews are even worse. Again any negative review is met with distain. The person having the audacity and stupidity to be critical is punished. This punishment is meted out with the clear message, “only positive reviews are tolerated.” Positive reviews are thus mandated by threat and retribution. We have created the recipe for systemic failure.

Putting the blame on systematic wishful thinking is far too kind. High quality for a discount price is wishful thinking at best. If the drivers for this weren’t naïve customers and dishonest programs, it might be forgivable. The problem is that everyone who is competent knows better. The real key to seeing where we are going is the peer review issue. By squashing negative peer review, the truth is exposed. Those doing all this substandard work know the work is poor, and simply want a system that does not expose the truth. We have created a system with rewards and punishments that allows this. Reward is all monetary, and very little positive happens based on quality. We can assert excellence without doing the hard things necessary to achieve it. As long as we allow people to simply declare their excellence without producing evidence of said excellence quality will languish.

Do Not Lie to Yourself

We have to be honest about what we want and take risks rather than lie to ourselves and make excuses to stay in our comfort zone.

― Roy T. Bennett

 

 

Science is not about making predictions or performing experiments. Science is about explaining.

― Bill Gaede

In the modern dogmatic view of high performance computing, the dominant theme of utility revolves around being predictive. This narrative theme is both appropriate and important, but often fails to recognize the essential prerequisites for predictive science, the need to understand and explain. In scientific computing the ability to predict with confidence is always preceded by the use of simulations to aid and enable understanding and assist in explanation. A powerful use of models is the explanation of the mechanisms leading to what is observed. In some cases simulations allow exquisite testing of models of reality, and when a model matches reality we infer that we understand the mechanisms at work in the World. In other cases we have observations of reality that cannot be explained. With simulations we can test our models or experiment with mechanisms that can explain what we see. In both cases the confidence of the traditional science and engineering community is gained through the process of simulation-based understanding.

Leadership and learning are indispensable to each other.

― John F. Kennedy

Too often in today’s world we see a desire to leap over this step and move directly to prediction. This is a foolish thing to attempt. Like fools, this is exactly where we are leaping! The role of understanding in the utility of simulation is vital in building the foundation upon which prediction is laid. This has important technical and cultural imprints that should never be overlooked. The role of building understanding is deep and effective in providing a healthy culture of modeling and simulation excellence. Most essentially it builds deep bonds of curiosity satisfaction within the domain science and engineering community. The experimental and test community is absolutely vital to a healthy approach, and needs a collaborative spirit to thrive. When prediction becomes the mantra without first building understanding, simulations often get put into an adversarial position. For example we see simulation touted as a replacement to experiment and observation. Instead of collaboration simulation becomes an outright threat. This can lead to completely and utterly counter-productive competition where collaboration would serve everyone far better in almost every case.

Understanding as the object of modeling and simulation also works keenly to provide a culture of technical depth necessary for prediction. I see simulation leaping into the predictive fray without the understanding stage as arrogant and naïve. This is ultimately highly counter-productive. Rather than building on the deep trust that the explanatory process provides, any failure on the part of simulation becomes proof of the negative. In the artificially competitive environment we too often produce, the result is destructive rather than constructive. Prediction without first establishing understanding is an act of hubris, and plants the seeds of distrust. In essence by sidestepping the understanding phase of simulation use makes failures absolutely fatal to success instead of stepping-stones to excellence. This is because the understanding phase is far more forgiving. Understanding is learning and can be engaged in with a playful abandon that yields real progress and breakthroughs. It works through a joint investigation of things no one knows and any missteps are easily and quickly forgiven. This allows the competence and knowledge to be built through the acceptance of failure. Without allowing these failures, success in the long run cannot happen.

Raise your quality standards as high as you can live with, avoid wasting your time on routine problems, and always try to work as closely as possible at the boundary of your abilities. Do this, because it is the only way of discovering how that boundary should be moved forward.

― Edsger W. Dijkstra

The essence of the discussion revolves around the sort of incubator that can be created by a collaborative, learning environment focused on understanding. When the focus is understanding of something the dynamic is forgiving and open. No one knows the answer and people are eager to accept failure as long as it is an honest attempt. More importantly when success comes it has the flavor of discovery and serendipity. The discovery takes the role of an epic win by heroic forces. After the collaboration has worked to provide new understanding and guided the true advance of knowledge, simulation sits in a place where it can be a trusted partner in the scientific or engineering enterprise. Too often in today’s world we disallow the sort of organic mode of capability development in favor of an artificial project based approach.

Our current stockpile stewardship program is a perfect example of how we have systematically screwed all this up. Over time we have created a project management structure with lots of planning, lots of milestone, lots of fear of failure and managed to completely undermine the natural flow of collaborative science. The accounting structure and funding has grown into a noose that is destroying the ability to build a sustainable success. We divide the simulation work from the experimental or application work in ways that completely undermine any collaborative opportunity. Collaborations become forced and teaming with negative context instead of natural and spontaneous. In fact anything spontaneous or serendipitous is completely antithetical to the entire management approach. Worse yet, the newer programs have all the issues hurting the success of stockpile stewardship and have added a lot additional program management formality. The biggest inhibition to success is the artificial barriers to multi-disciplinary simulation-experimental collaborations, and the pervasive fear of failure permeating the entire management construct. By leaping over the understanding and learning phase of modeling and simulation we are short-circuiting the very mechanisms for the most glorious successes. We are addicted to managing programs not to ever fail, which ironically sew the seeds of abject failure.

The problem with the current project milieu is the predetermination of what success looks like. This is then encoded into the project plans and enforced via our prevalent compliance culture. In the process we almost completely destroy the potential for serendipitous discovery. Good discovery science is driven by having rough and loosely defined goals with an acceptance of outcomes that are unknown beforehand, but generally speaking provide immense value at the end of the projects. Today we have instituted project management that attempts to guide our science toward scheduled breakthroughs and avoid any chance at failure. The bottom line is that breakthroughs are grounded on numerous failures and course corrections that power enhanced understanding and a truly learning environment. Our current risk aversion and fear of failure is paving the road to a less prosperous and knowledgeable future.

A specific area where this dynamic is playing out with maximal dysfunctionality is climate science. Climate modeling codes are not predictive and tend to be highly calibrated to the mesh used. The overall modeling paradigm involves a vast number submodels to include a plethora of physical processes important within the Earth’s climate. In a very real sense the numerical solution of the equations describing the climate are forever to be under-resolved with significant numerical error. The system of Earth’s climate also involves very intricate and detailed balances between physical processes. The numerical error is generally quite a bit larger than the balance effects determining the climate, so the overall model must be calibrated to be useful.

You couldn’t predict what was going to happen for one simple reason: people.

― Sara Sheridan

In the modern modeling and simulation world this calibration then provides the basis of very large uncertainties. The combination of numerical error and modeling error means that the true simulation uncertainty is relatively massive. The calibration assures that the actual simulation is quite close to the behavior of the true climate. The models can then be used to study the impact of various factors on climate and aid the level of understanding of climate science. This entire enterprise is highly model-driven and the level of uncertainty is quite large. When we transition over to predictive climate science, the issues become profound. We live in a world where people believe that computing should help to provide quantitative assistance for vexing problems. The magnitude of uncertainty from all sources should provide people with significant pause and provide a pushback from putting simulations in the wrong role. It should also not prevent simulation from providing a key tool in understanding this incredibly complex problem.

The premier program for high performance computing simply takes all of these issues and amplifies them to an almost ridiculous degree. The entire narrative around the need for exascale computing is entirely predicated on the computers providing predictive calculations. This is counter to the true role of computation as a modeling, learning, explanation, and understanding partner with scientific and engineering domain expertise. While it is wrong at an intrinsic level the secondary element in the program’s spiel is the simplicity of moving existing codes to new, faster computers for better science. Nothing could be further from the truth on either account. Most codes are woefully inadequate for predictive science first and foremost because of their models. All the things that the exascale program ignores are the very things that are necessary for predictivity. At the end of the day this program is likely to only produce more accurately solved wrong models and do little for predictive science. To exacerbate these issues, the exascale program generally does not support the understanding role of simulation in science.

The long-term impact of this lack of support for understanding is profound. It will produce a significant issue with the ability for simulation to elevate itself to a predictive role in science and engineering. The use of computation to help with understanding difficult problems paves the way for a mature predictive future. Removing the understanding is akin to putting someone into an adult role in life without going through a complete childhood. This is a recipe for disaster. The understanding portion of computational collaboration with engineering and science is the incubator for prediction. It allows the modeling and simulation to be very unsuccessful with prediction and still succeed. The success can arise through learning things scientifically through trial and error. These trials, errors and response over time provide a foundation for predictive computation. In a very real way this spirit should always be present in computation. When it is absent, the computational efficacy will become stagnant.

In summary we have yet another case of marketing of science overwhelming the true narrative. In the search for funding to support computing, the sale’s pitch has been arranged around prediction as a product. Increasingly, we are told that a faster computer is all that we really need to do. The implied message in this sale’s pitch is a lack of necessity to support and pursue other aspects of modeling and simulation for predictive success. These issues are plaguing our scientific computing programs. Long-term success of high performance computing is going to be sacrificed, based on this funding-motivated approach. We can add the failure to recognize understanding, explaining and learning as a key products for science and engineering from computation.

Any fool can know. The point is to understand.

― Albert Einstein