Here is the answer in a nutshell; people use an inappropriate approach and test problems to characterize their methods.

The reason I started doing work in verification was directly related to my work in developing better numerical methods.  In the early 90’s I was developing an improved implementation of a volume tracking method (or VOF where an interface is tracked through volumes, and the interface is reconstructed geometrically).   The standard implementation of the method resulted in horrific spaghetti code, and made subsequent development and debugging a nightmare.  I had developed a better, more object-oriented implementation and was eager to share it.  In the course of doing this implementation work, I engaged in extensive testing to put the method through its paces.  I felt that the standard tests were insufficiently taxing and provided poor code coverage, so I came up with several alternative test problems.  Existing tests involved translating objects on a grid, or engaging in solid body rotation.  Difficulty was defined by the complexity of the shape being moved.  My new problems used time dependent flowfields that had non-zero vorticity and could be reversed in time allowing exact error assessment.  These new problems were a hit and have largely replaced the earlier (too) simple tests.

Now fifteen years on the paper is very highly cited, but mostly because of the better tests, not because of the reason the paper was actually written.  I’d say about 75-90% of the citations are primarily for the tests, not the methodology.  There is a lesson in this that is positive; people want to test their methods rigorously.  For myself, I was a verification devotee thereafter largely because of the experience.

In terms of numerical methods and physics my greatest interest is the solution of hyperbolic PDEs.  I find the mathematics and physics of these equations both engaging on multiple levels, and the numerical methods to solve these equations spell-bindingly interesting.  It is also an area of numerical methods where verification is not the standard practice; at least for the application of the equations to full nonlinear PDEs that characterize the application of the method.  This isn’t entirely fair; for simple linear equations with smooth solutions or nonlinear equations with discontinuities verification is commonplace and expected.  Under these limited conditions authors regularly verify their methods and present results in virtually every paper.  When a discontinuity forms accuracy in the sense of high-order convergence is lost; solutions are limited to first-order convergence or lower.  Under these conditions verification is not done as a matter of course, in fact, it is exceedingly rare to the point of almost being unheard of.

This is extremely unfortunate.  Verification is not only about order of accuracy; it is also about estimating the magnitude of numerical error.

The standard approach when smoothness is not present is to plot results at some resolution against an exact solution and show these results graphically.  The main reason to show the results is to demonstrate a lack of oscillatory wiggles near the discontinuity.  The error itself is easily (almost trivial) computable, but almost never presented. The basis of this reasoning is that all results are basically first-order accurate, and the error isn’t important.  Implicit in this judgment is the belief that order of accuracy matter, and by virtue of this numerical error magnitude is unimportant.   Unfortunately, this point of view is not entirely correct, and misses a key aspect of method character.

When all solutions are first-order accurate there are actually significant differences in the level of numerical error.  Under mesh refinement these differences can become quite significant in terms of the quality of solution.   Consider the following example with regard to the quality of solution, for first-order a factor of two increases in mesh resolution results in a halving of numerical error.  If instead you changed the numerical method to halve the error on the original grid, the savings in compute time could be large.  A halving of the mesh results in the computational work increasing by a factor of four in one dimension (assuming the Courant number is held constant).  In two dimensions the increase in effort is a factor of eight and a factor of sixteen in three dimensions.  If the more accurate numerical method does not require four or eight or sixteen times the effort, it will be a win.    More accurate methods are most computationally intensive, but rarely so much that they aren’t more efficient than lower order methods. The cases where these dynamics play out are the closest to actual applications where the problems are discontinuous and never well behaved mathematically. Indeed it is this character that explains the rapid adoption of the second-order MUSCL (Van Leer differencing for you weapons’  lab folks) over the first-order methods.

The differences are not geometric as they would be with high-order accuracy and smooth solutions, but quite frankly no one really gives a damn about smooth solutions where high-order accuracy is achievable.  They are literally only of academic interest.   Seriously.  So why, isn’t more rigorous error estimation on more difficult problems the status quo?  Fundamentally people simply haven’t come to terms with the inevitable loss of accuracy for real problems, and the consequences this has on how methods should be designed and evaluated.

Despite this, the community solving hyperbolic PDEs believes that verification is being routinely done.  It is.  It is being done on cases that matter little, and only reflect on the order of accuracy of the method in cases no one cares about.  For people doing applications work, it makes verification seem like a pitiful and useless activity.  My suggestions that verification is done for error characterization and estimation is not considered useful.  I looked back through three and a half decades of the literature at results for the venerable Sod’s shock tube problem (en.wikipedia.org/wiki/Sod_shock_tube), and could find no examples of accuracy being quoted with computed results.  Sod showed runtime in his 1978 paper.  This implies that the accuracy of the results is comparable and the computational cost at fixed resolution only matters.  This seems to be an article of faith by the community, and it is wrong!

The fact is that both speed and accuracy matter, and results should be presented thusly.

There are large differences in the accuracy between methods.  If one takes the measure of accuracy of computational effort required to achieve constant accuracy one can find a factor of 30 difference in the efficiency of different approaches (http://www.sciencedirect.com/science/article/pii/S0021999107000897).  This difference in efficiency is in one dimension with the potential for dramatically larger differences in two or three dimensions.  Yet this community almost systematically ignores this issue, and verification of methods for practical problems.

This is a long-winded way of saying that there are a lot of cases where people prefer to use methods that are worse in performance.  Part of this preference is driven by the acceptance of the sort of verification practice discussed above.  An example would be WENO (http://www.sciencedirect.com/science/article/pii/S0021999196901308) where elegance and the promise of high-order would seem to drive people, despite its low resolution.  WENO stands for weighted essentially non-oscillatory, which blends order of accuracy preservation with lack of Gibbs oscillations in an elegant almost algebraic algorithm.  Most of the development of WENO since its inception has revolved around accuracy, which is compounded by computing results for these linear problems.   The focus of order-of-accuracy results in the method being quite poor at resolving discontinuities with performance at those points of the solution being on par with the minmod second-order method (the worst and most dissipative of the second-order TVD methods!).  Instead of improving WENO for practical problems and focusing on its efficiency, the mathematical community has focused on its ability to achieve formally high-order accuracy in situations no one really cares about.

This might explain the lack of penetration of this class of method into practical computations in spite of thousands of citations. It is an utterly absurd state of affairs.

MUSCL (http://en.wikipedia.org/wiki/MUSCL_scheme) is an example of much more successful method, and as I showed with Jeff Greenough a well-written MUSCL code kills a WENO code in actual efficiency (http://www.sciencedirect.com/science/article/pii/S0021999103005965).  This was just for one dimension and the gains for MUSCL over WENO in multiple dimensions are as great as expected.  It isn’t a surprise that WENO has not been as successful for applications as MUSCL.  A change in focus by the community doing WENO development might serve the method well.

PPM (http://crd.lbl.gov/assets/pubs_presos/AMCS/ANAG/A141984.pdf) is much better than either WENO or MUSCL.  The gains in efficiency are real (about a factor of two over MUSCL in 1-D).  Part of this resolution is due to the difference between a linear and parabolic local representation of the solution, and the parabolic profile’s capacity to represent local extrema.  PPM can also exhibit bona fide high-order accuracy through the selection of high-order values for the edge values used in determining the parabola.  The bottom line is that the PPM method has some intrinsic flexibility that can be ruthlessly exploited.  PPM is widely used by astrophysicists for many applications.  Beyond astrophysics it has been less successful and perhaps that should be studied.

WENO is elegant.  It is a beautiful method, and I loved coding it up.  I would have to say that my WENO code is extremely attractive, just as the mathematics underpinning these methods are appealing.  This appealing veneer can be superficial and hides the true performance of the method.  It is sort of like a beautiful person with a horrible personality or weak intellect; the attractive package hides the less than attractive interior.  This isn’t to say that the method can’t be improved in terms of efficiency. Indeed through taking various lessons learned from MUSCL and PPM and applying them systematically to WENO, the method could be dramatically improved.  In doing this some of the properties of WENO for smooth hyperbolic PDEs are undermined, while the properties for discontinuous problems are improved.  It could be done quite easily.

Ultimately the community has to decide where it wants impact.  Are hyperbolic PDEs important for applications?

Yes, and completely beyond a shadow of doubt.

Why does the published literature act the way it does? How has the community evolved to where the actual performance on more application-relevant testing matters less than beauty and elegance?

I think this whole issue says a great deal about how the applied mathematics community has drifted away from relevance.  That is a topic for another day, but keenly associated with everything discussed above.

Thanks for listening, happy New Year.

 One of the big problems that the entire V&V enterprise has is the sense of imposition on others.  Every simulation worth discussing does “V&V” at some level, and almost without exception they have weaknesses.  Doing V&V “right” or “well” is not easy or simple.  Usually, the proper conduct of V&V will expose numerous problems with a code, and/or simulation.  It’s kind of like exposing yourself to an annual physical; its good for you, but you might have to face some unpleasant realities.  In addition, the activity of V&V is quite broad and something almost always slips between the cracks (or chasms in many cases). 

To deal with this breadth, the V&V community has developed some frameworks to hold all the details together.  Sometimes these frameworks are approached as prescriptions for all the things you must do.  Instead I’ll suggest that these frameworks are not recipes, nor should they be thought of as prescriptions.  They are “thou should,” not “thou shalt,” or even “you might.“

 Several frameworks exist today and none of them is fit for all purposes, but all of them are instructive on the full range of activities that should be at least considered, if not engaged in.

 CSAU – Code Scaling Assessment and Uncertainty developed by the Nuclear Regulatory Committee to manage the quality of analyses done for power plant accidents.  It is principally applied to thermal-fluid (i.e. thermal-hydraulic) phenomena that could potentially threaten the ability of nuclear fuel to contain radioactive products.  This process led the way, but has failed in many respects to keep up to date.  Nonetheless it includes processes and perspectives that have not been fully replicated in subsequent work.  PCMM is attempting to utilize these lessons in improving its completeness.

 PCMM – Predictive Capability Maturity Model developed at Sandia National Laboratories for the stockpile stewardship program in the last 10 years.  As such it reflects the goals and objectives of this program and Sandia’s particular mission space.  It was inspired by the CMMI developed by Carnegie Mellon University to measure software process maturity.    PCMM was Sandia’s response to calls for greater attention to detail in defining the computational input into quantitative margins and uncertainty (QMU), the process for nuclear weapons’ certification completed annually.

 CAS – Credibility Assessment Scale developed by NASA.  They created a similar framework to PCMM for simulation quality in the wake of the shuttle accidents and specifically after Columbia where simulation quality played an unfortunate role.  In the process that unfolded with that accident, the practices and approach to modeling and simulation was found to be unsatisfactory.  The NASA approach has been adopted by the agency, but does not seem to be enforced.  This is a clear problem and potentially important lesson.  There is a difference between an enforced standard (i.e., CSAU) and one that comes across as well intentioned, but powerless directives.  Analysis should be done with substantial rigor when lives are on the line.  Ironically, formally demanding this rigor may not be the most productive way to achieve this end.

 PMI, Predictive Maturity Index developed at Los Alamos.  This framework is substantially more focused upon validation and uncertainty, and IMHO it gives a bit lax with respect to the code’s software and numerical issues.  In my view, these aspects are necessary to focus upon given advances in the past 25 years since CSAU came into us in the nuclear industry.

Computational simulations are increasingly used in our modern society to replace some degree of expensive or dangerous experiments and tests.  Computational fluid and solid mechanics are ever more commonplace in modern engineering practice.  The challenge of climate change may be another avenue where simulation quality is scrutinized and could benefit from a structured, disciplined approach to quality. Ultimately, these frameworks serve the role of providing greater confidence (faith) in the simulation results and their place in decision-making.   Climate modeling is a place where simulation and modeling plays a large role, and the decisions being made are huge.

The question lingers in the mind, “what can these frameworks do for me?”  My answer follows:

1. V&V and UQ are both deep fields with numerous deep subfields.  Keeping all of this straight is a massive undertaking beyond the capacity of most professional scientists or engineers.
2. Everyone will default to focusing on where they are strong and comfortable, or interested.  For some people it is mesh generation, for others it is modeling, and for yet others it is analysis of results.  Such deep focus may not lead (or is not likely to lead) to the right sort of quality.  Where quality is needed is dependent upon the problem itself and how the problem’s solution is used.
3. These are useful outlines for all of the activities that a modeling and simulation project might consider.  Project planning can use the frameworks to develop objectives and subtasks, prioritize and review.
4. These are menus of all the sort of things you might do, not all the things you must do. 
5. They provide a sequenced set of activities, prepared in a sequenced rational manner with an eye toward what the modeling and simulation is used for.
6. They help keep your activities in balance.  They will help keep you honest.
7. You will understand what is fit for purpose, when you have put too much effort into a single aspect of quality.
8. V&V and UQ are developing quickly and the frameworks provide a “cheat sheet” for all of the different aspects.
9. The frameworks flexibility is key, not every application necessarily should focus on every quality aspect, or apply every quality approach in equal measure.

10. Validation itself is incredibly hard in both breadth and depth.  It should be engaged in a structured, thoughtful manner with a strong focus on the end application.  Validation is easy to do poorly.

11. The computational science community largely ignores verification of code and calculations.  Even when it is done, it is usually done poorly.

12. Error estimation and uncertainty too rarely include the impact of numerical error, and estimate uncertainty primarily through parametric changes in models.

13. Numerical error is usually much larger than acknowledged.  Lots of calibration is actually accounting for the numerical error, or providing numerical stability rather than physical modeling.

The thoughts associuated with last week’s post continued to haunt me.  It hung over my thoughts as I grappled with the whole concept of thankfulness for last week’s holiday.  I know implicitly that I have a tremendous amont to be thankful for at both a personal and professional level.  By all but the most generous standards, I am very lucky, certainly far better off than the vast, vast majority of humanity.  Nonetheless, it is hard to shake the sense of missed opportunity and squandered potential.  The USA is not a healthy country right now, and that fills me with a sadness.

With a system that is so dysfunctional most of what we can be thankful for is close to home.  Things we used to take for granted, like freedom, and that your children had a better chance at a good life than you do are not secure any more.  We have transformed into a society where corporate rights exceed personal or human rights, and there is little trust in any institution public or private.   I wrote about the lack of trust in science last week and how it makes a scientists work both overly expensive and underly effective.  It is a tragic situation.

Of course the Internet has a lot to do with it.  Or at the very least the Internet has amplified many of the trends that were already apparent preceding its rise to societal prominence.  In a sense the Internet is a giant communication magnifier, but also allows anonymous communication at a scale unimaginable.  I am reminded of what Clay Shirky noted that we may be in the midst of huge reorganization of society similar to what happened after Gutenberg invented the printing press.  Its a big deal, but not much fun to be part of.  Anonymity can be a horrible thing if misused.  We are seeing this in spades.  You really can have too much information, or have it too unfiltered.  Worse yet than a filter is the propaganda that spins almost everything we read toward some world-view.  The money and talent is largely in the corporate camp, and we are being sold ideas they favor with a marketers talent for shaping our views.

Part of my inability to shake these ideas came in the form of a “longread” recommendation about the 40 year decline in the prospects of the American worker (http://prospect.org/article/40-year-slump) starting in 1974.  After massive growth in equality in the years following World War II, it ended in 1974, and began the steady march toward today’s perverse levels of inequality.  This was the same year as I had posited the beginning of the decline in trust for public institutions such as science.  While some institutions such as science, research and government have lost the public’s trust, the corporation has become the centerpiece for society. 

Stockholder return has become the benchmark for success.  This has become a stark contrast to the actual health or long-term prospects for a company.  Likewise, the government-funded research has become starkly risk adverse and commenserately short term focused.  Society as a whole will suffer the effects of these twin pathologies.  Neither public, nor private interests are investing in the future.  Our infrastructure crumbles and no one even thinks of creating a 21^st Century infrastructure because we can’t keep our 20^th Century infrastructure healthy.  R&D has become all development, and has to be attached to the bottom line preferably the next quarter.   This progress adverse corporate principle is adopted whole-cloth by the government because of a faith-driven belief that the “market knows best” and business practice is inherently superior to other approaches.  One might consider that such research would be application-mission oriented, but no it simply risk adverse and pursued because “success,” such as it is defined, is almost a sure thing.

In the process, the “1%” has risen to dizzying heights of wealth, largely through the decision making process where corporations invest far less capital in their futures and instead to pay massive dividends to their stockholders.  The implications for science, and society are far reaching.  Is there a connection?  I think it is all part of the same thing.  Any science that makes life difficult or complicated for a corporation is attacked because the profit motive and margin has become the sole measure of societal success.  The human toll is massive and our society shows no mercy or thought for the suffering all this wealth accumulation is unleashing.

I am reminded about the problems of correlation being linked to causation, but the seeming coincidence in timing might be more meaningful.  How are these two issues connected?  I’m not sure, but I do think there is one.  Perhaps science being an endeavour for the common good has lost favor because its benefits can be shared by all rather than a few.  Everything that produces a shared benefit seems to be in decline and the entire society is being structured to serve the very top of the food chain.  The wealthy have become apex predators in the human ecosystem, and seem have no qualm about “feeding” on the ever-expanding masses in poverty.  The loss of trust powers the problem because of the unique nature of the American phyche and its devotion to the individual above all.  Now we are transitioning to a view where the devotion is to the corporation (but really the rich who profit from the corporate “success”).

So these thems stretch across the expanse of years that define nearly my entire life, the mystery of how to leave something better for my children is unsolved.  This is what really haunts me, and why I’m not as thankful as I would like to be.