What’s measured improves

– Peter Drucker

I’ve been doing a fair amount of thinking about metrics for computer performance. The question is of supreme importance moving forward with supercomputing. The Drucker quote fairly well captures both the importance of answering the metrics question moving forward, but also consideration of what happened in the recent path. I’ve come to the conclusion that the devotion to “weak scaling” is probably doing a lot of damage to high performance computing.

What is weak scaling? From http://en.wikipedia.org/wiki/Scalability

• The first is strong scaling, which is defined as how the solution time varies with the number of processors for a fixed total problem size.

• The second is weak scaling, which is defined as how the solution time varies with the number of processors for a fixed problem size per processor.

The problem is that weak scaling has allowed a number of issues to be papered over. The biggest of these issues is the continuing diminishing performance on node, or the ability to efficiently use the full computing capacity of the machines. We get “better” by using more nodes, but the nodes are used very poorly. A big issue is the memory bandwidth and cost. Basically the processors are starved for data and useful work to do.

The other thing that we are missing is algorithmic development. The success given to weak scaling has starved the development of better algorithms. We are spending a lot of effort implementing old algorithms to achieve weak scaling. Algorithmic scaling is actually much harder, and given the success with which a weak scaling is greeted, the emphasis is chosen by fiat. The fact is that implementing an old algorithm on a new computer is low risk compared to developing new algorithms. Given our propensity toward low risk (i.e., risk management), we get today’s situation; computational science in the doldrums and a lack of real innovation and progress.

It all comes down to what we measure. As long as weak scaling is deemed a success we can continue to ignore the issue. We can hope that the issue will be ultimately put squarely in the frame by the issues of power consumption, which is creating issues that can’t be ignored (the electric bill).

I think the real metric should be something like time or cost to produce a solution of a given, credible quality. It should include wall clock time and the cost of labor. Finally we should consider the costs associated with the lack of basic vitality of the community. We could stand to have some significantly greater inspiration.

In preparing for battle I have always found that plans are useless, but planning is indispensable.

― Dwight D. Eisenhower

Many of you might be familiar with the PCMM (Predictive Capability Maturity Model). Most commonly it is thought of as structured way to assess (i.e., judge) the quality of a computational investigation. Since no one likes to be judged, it isn’t greeted with the greatest warmth. I’d like to provide another way of looking at PCMM, as a rubric for doing computational science.

In this sense it provides a catalog of things that you should worry about, decide on your intent and shape the work to meet specific goals and intended uses. The concept of broad categories of competence with graded levels of quality allows one to make sure all the necessary activities are being done. Too often a program will be executed almost solely within a single category and the other aspects of computation would be cobbled together. The intended quality of the work is an afterthought.

With the traditional structure of the PCMM as a starting point, there is no problem in removing or adding new categories to suit the purposes of the project. The usual set of categories: geometry/representation, physical modeling, code/software verification, solution verification/error estimation, validation, and uncertainty quantification are a good place to start, but inadequate for many projects. An example might be weather or climate modeling where data assimilation is important enough to warrant its own category. In computation of social science, the geometry is irrelevant and needs to be replaced with an appropriate description of the environment things like agents are placed in. In other cases the experimental work is sufficiently complex and focused that it should be expanded into far greater detail included a data focus. The point is that PCMM is not a fixed framework, but an idea of how to organize your activity as to not leave important things out.

An idea can only become a reality once it is broken down into organized, actionable elements.

― Scott Belsky

How might I use PCMM to do something that isn’t V&V related first? Say, like writing a new code?

I would consider what the application of the code is intended to be and how much further than the original intent might be supported? How essential is the geometric fidelity to the quality of simulations? How well are the basic physical models, and supporting constitutive relations established? Is the numerical method and the equations supported by mathematical rigor? Are numerical errors well understood where the equations and method are to be applied? What experiment exist for validation, and will new validation experiments be conducted? What sort of quantities of interest are needed and how will their uncertainty be assessed? For every question how critical is the quality of the answer, and what is the level of decision to be made with the results? Might any of this change over time, and can those changes be accommodated in the desired code?

Notice that this is the application of PCMM classic in the form of a set of questions to be asked at the outset of the work. If these questions are proactively addressed the code will be able to be assessed for V&V quality with great ease. Ultimately this should lead to an appropriate statement of the computational credibility of the code and its suitability for intended use.

The key is that PCMM isn’t just for V&V any more, and if it is used proactively V&V can be so much easier.

The essence of community, its heart and soul, is the non-monetary exchange of value; things we do and share because we care for others, and for the good of the place.

― Dee Hock

In the councils of government, we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military-industrial complex. The potential for the disastrous rise of misplaced power exists and will persist. We must never let the weight of this combination endanger our liberties or democratic processes. We should take nothing for granted. Only an alert and knowledgeable citizenry can compel the proper meshing of the huge industrial and military machinery of defense with our peaceful methods and goals, so that security and liberty may prosper together.

-President Eisenhower

Our republic is evolving quickly into some sort of nationalist-kleptocratic union of industrial interests who have the political apparatus in their pocket. They have all the tools needed to market (dupe) the public into voting for their interests. Their interests are siphoning as much money as possible into as few pockets as possible. They have created a system where the Nation’s laws are being written for the sole purpose of enabling their greed. The nation is increasingly being organized for sole benefit of a vastly rich ruling class served by a vast underclass whose good is ignored. The underclass is brainwashed through mass media propaganda into supporting the very policies that are hurting them. All of this is viewed as being utterly patriotic. This is a shining example of American Exceptionalism.

So what are these complexes?

The most effective way to destroy people is to deny and obliterate their own understanding of their history.

― George Orwell

      The military-industrial complex. The classic, the original or at least the one you were warned about. It is still around and still draining a huge amount of money from the economy. Our defense budget is still the size of the next 10 largest in the World, yet it still isn’t enough. Any cut in spending sends people into a frenzy of fear. We have to be the World’s policemen and go around killing various enemies who really offer no threat to us at all. This leads to war after was that simply offer the defense industry the opportunity to profit on other’s misery. We could really do with a budget of half to a third of its current size. The only risk it would pose is to the bottom line and profit margin of defense contractors.

They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.

― Benjamin Franklin

The national security-industrial complex. This is the new and improved version of the original complex. Where the military-industrial complex is sold through nationalism and the ability to exercise raw power, this complex has the selling point of irrational fear. Make no mistake; this one is coupled to the other. The activities of this complex are cloaked in secrecy and a vast unspecified (black) budget that is completely justified in the eyes of the cowering-terrified masses of voters. Americans have agreed in overwhelming numbers to part with liberty and a great deal of money in service of this complex. The true aim of this complex is control by the powerful as well as profit. Again, the threats we are facing are vastly overblown.

I would argue that the security state we are creating is more of a threat to our way of life than any of the enemies they are “protecting” us from. We have allowed ourselves to become fearful as a Nation and we will suffer a fate befitting this trend. Recently and increasingly this complex is supporting the militarized and aggressive police.

 Propaganda is to a democracy what the bludgeon is to a totalitarian state.

― Noam Chomsky,

The financial-investment-industrial complex. You know who these guys are. This is the industry that almost melted down the economy five years ago. Bailouts happened and the incomes of most of the guilty were saved (if temporarily diminished slightly). Reform was promised and never delivered upon. Almost none of the shady, sleazy or corrupt practices that enriched the financial managers and preyed upon common citizens resulted in any legal action. New financial vehicles have been created to prey on the citizenry (e.g., student loans). More than nuclear weapons, this vehicle of mass destruction looms over the economy waiting for the next bubble to burst. The national-security-police apparatus was used to crust any resistence and treated the Occupy Wall Street movement like domestic terrorists. So much for democracy.

In a society governed passively by free markets and free elections, organized greed always defeats disorganized democracy.

― Matt Taibbi

This complex has the money to buy immense favor, does almost nothing for society, and unlike the others produces only rich people. Somehow they have conned to electorate into believing that they create economic opportunity. Once finance did this, now all the opportunity is in the hands of the “shareholders” who are mostly financial managers. Increasingly business interests is just code for ability for executives to be compensated in repugnantly extravagant ways. Much of the deplorable situation with education can be traced to student loans as investment vehicles along with the concentration of wealth cascading to enhanced support of a cadre of private universities that serve the rich. No other complex is so singularly responsible for the erosion of the middle class as this one.

Health makes good propaganda.

― Naomi Wolf

The medical-industrial complex. This complex is the two-edged sword of overwhelming good on one side and horrific greed on the other. Medicine does enormous good and healing people is a worthy profession. Too bad it has become the cover for a massive plundering operation. Medicine in the USA consumes nearly to over twice the amount of GDP than it does in other comparable Nations. In return for this massive cost we get substandard care. Our health outcomes and quality of care is actually lower than comparable nations. We get a lousy deal and the medical industry makes even more money.

Marketing is what you do when your product is no good.

― Edwin H. Land

Are Americans just incompetent? No. We simply have adopted a system that is immensely wasteful and has massive opportunities for profit taking by corporate interests who only care about how much money they make. These interests have managed to sabotage any effort to introduce a better system. Even a relatively favorable to industry, modest and small change is marketed to the electorate as being socialism. Yes I mean Obamacare, which is largely a free-market based plan that only modestly changes the dynamic. It only diminishes their returns a bit and does little to change the underlying problems.

The criminal-justice-prison-industrial complex. The embarrassment we should be feeling over the incarnation rate of our citizens is absent. Do we really feel that the American public is so awful that they should be imprisoned at a rate two to ten times larger than comparable countries? Are Americans inherently violent and criminal in our conduct? Is that the way we are exceptional? The answer to all these questions is no. The problem is that power and money flow from locking up Americans. It has also become a convenient way of reinstituting “Jim Crow” laws.

This all began with the failed war on drugs. Ultimately this produced a bunch of people to imprison. Eventually the prisons became privatized and a way to make money. Prisons had a lobby. At the same time law enforcement found ways of increasing their power through the same war. Increased weaponry, increased surveillance, increased powers of seizure. All of this coupled together to produce an orgy of imprisonment, and a deluge of cash to industry. This has been amplified by the rise of the national security industrial complex that provides an additional source of enhanced power.

These industrial interests have absolutely no interest of consciousness about the damage done to society. The good of society as a whole is not of consequence. All of these complexes are unremittingly self-interested and self-serving, only concerning themselves with their bottom line. This is the core of the problem. These complexes are monuments to shortsighted and internally focused narcissism without conscious or morality that considers the good of others. It is a derelict philosophy that the United States appears to have adopted as its core operating principle.

The election this week will be an enormous boon to the collection of “complexes” that run our lives. It will not be good for the economic, security or personal destinies of most Americans. On the other hand it will help propel the fortunes of the leaders of these complexes to new heights on the backs of the labor and suffering of most of the citizens. I’m not convinced that an alternate end to the election would have been bad for the complexes, just not as good. Complexes have broad bipartisan support. In other words they have enough money to legally buy both parties.

Education is the most powerful weapon which you can use to change the world.

― Nelson Mandela
Today and tomorrow I am visiting the University of Notre Dame for the purpose of reviewing and advising a DOE sponsored research center (CSWARM). The project combines material science, computation and computer science as the scientific world struggles toward the next generation of computing. Given today’s work, my
background and my son’s entry into science and engineering education next year, the topic of college and its health is keenly in my focus. Universities are perhaps one of the most important institutions in our society. They should be emblematic of the best we have to offer and a shining example of what we aspire to be. So what do university’s have to say about us today?

 Anyone who stops learning is old, whether at twenty or eighty. Anyone who keeps learning stays young.

― Henry Ford

In all respects the visit didn’t disappoint me. The research project while centered at Notre Dame spans two other universities (Indiana and Purdue) and involves computer science, computational science, experiment, and theory. Each topic was present and the work of melding the usually independent university research agendas into a single multidisciplinary center has begun. This is hard work that is firmly against the grain of university dynamics. We were presented with a plethora of results, wowed by their progress, but saw many areas where suggestions could profitably be made. All-in-all a good showing and a great start at a great research project.

We are all failures- at least the best of us are.

― J.M. Barrie

Before going on to the things that bothered me about the visit, I will freely admit that some aspects of interaction with the universities give me pause. The current style of program management for research is not congruent with educational objectives. These projects do not get the full brunt of government management, but enough of their guidance is the sort of “do not ever fail” category to run afoul of educational optimality. Research if done properly will fail, and fail a lot. It is research because we don’t know exactly what we are doing. If it isn’t failing a great deal it probably isn’t good research. Too much of what I work on is under a no-failure edict. This creates an environment that largely runs counter to the capability to create of good educational setting. Given that the current nature of the educational setting is itself a problem, we may not be doing too much extra harm.

Children must be taught how to think, not what to think.

― Margaret Mead

The less favorable side of college is also evident. Notre Dame oozes with money, in an almost overwhelming way. Given the tuition it demands, it must, but clearly wealthy donors are at work too. The university is also taken with their pride and joy, the football team and its effects are profound. Events must be scheduled around the home games, which impact everything in town making anything else impossible. The stadium looms over the campus like the shrine to modern gladiatorial combat that football is. It is a semi-pro team in every respect except paying the players and name. Its impact and importance at an educational institution is both unsettling and all too common. The opulence of the setting is somewhat out of character with ought to be the priorities.

Like so much with higher education there is a lot to be said about these trappings and their seeming importance, none of it good. We are not emphasizing the educational aspects, nor celebrating its achievements like the sporting face of the university. Similarly and complementary, money is being celebrated as important in equal measure. This too is a problem because all of this is being done under the watchful eye of the generation being educated. What message are we projecting through this? What impact do the obvious priorities have on the perceptions and thoughts of the youth?

Tell me and I forget, teach me and I may remember, involve me and I learn.

― Benjamin Franklin

All of this is merely a backdrop to an educational setting where research is also happening. Research in and of itself has become a similar obsession for universities because of its place as a source of money. This has in turn driven resources and focus away from education as the principal focus for universities. At least research can serve a proper place in education. If done with the proper balance research can be a powerful for educating the participants and continuing the development of the senior people involved. I wonder whether in some places the focus is even further removed from education by rich alumni, and massive athletics. Is this really what we need from these institutions? Is this really in service of the best interests of society as a whole, or are these institutions being highjacked to entertain the wealthy while giving substandard educations to the less fortunate?

 Wisdom is not a product of schooling but of the lifelong attempt to acquire it.

― Albert Einstein

Vulnerability is the birthplace of innovation, creativity and change.

― Brené Brown

Right now, I’m visiting North Carolina and speaking with plasma physicists about V&V (the American Physical Society’s Gaseous Electronics Conference-GEC in Raleigh). I gave a talk on my experiences and any wisdom gained through 15 years of effort in the largest scientific software project ever. Afterward, I participated in a panel discussion of the topic and the prospects for crafting a path forward.

    Towards the end of the questions about my presentation, one of the other speakers asked me how large the code development efforts I worked on were. In answering it, I realized to my shame that am enormously lucky to be working with resources I have at my disposal. The people I was speaking to are working with vastly less. Who was I to be giving them advice? It was like the rich guy rolling down the window of his limo to encourage the homeless person to “get a job, you bum!”

It’s mind-boggling how many different worlds people live in on this one planet.

― Richelle E. Goodrich

As a result, at the end of the panel discussion I was bombarded by a couple of feelings that were odd. The first was gratitude for the opportunities I’ve been granted professionally. I have been extraordinarily fortunate and lucky to work on projects that have been so incredibly well funded and supported. Fifteen years of (semi-) coherent effort towards some goal is amazing in today’s attention-deficit World.

The second part was some degree of shame related to telling anyone without the resources I’ve had access to about how to do anything. The third part was some degree of dismay in seeing how little progress we have actually made with the resources given to us where I work. Those with precious little to apply to their problems (like those who invited me to this meeting) don’t seem to have a chance of slaying the dragons confronting them. Saying this has nothing to do with their capability or capacity for good work; it has everything to do with the level of effort that can be mustered toward singular goals. The GEC-community is dealing with a host of different physics in a plethora of different regimes applied to a phalanx of different purposes. Without a common axis to orient their efforts, the efforts are horribly fragmented, and naturally incoherent. I think about my situation and think, “holy shit! I do have it easy!”

There is the honest issue that parts of the plasma physics community who have resources have failed at improving modeling and simulation in a balanced way. Too much effort has been placed in creating physics models, and too little effort has been placed in better algorithms and better practices to solve problems. These projects are almost as well off as the ones I work on, but have left their community with too little to build on. In many respects the plasma physics community has lagged behind in some significant ways. It felt good to admit to the audience yesterday that in many respects plasma physics is the whipping boy in some V&V circles for how NOT to do things. I asked this in providing the audience with a compliment that they were doing something to change this. It was also a question to them about how the broader plasma physics community feels about V&V and what their collective concerns are.

There’s class warfare, all right, but it’s my class, the rich class, that’s making war, and we’re winning.

― Warren Buffett

Much is being made of inequality today. The reasons are plain to see, the levels of inequality of opportunity, wealth and quality of life are stunning. They imperil our ability to advance as a society because so many are left behind without any capacity to fully contribute to the success of the whole. A large part of the reason for the deficit of capacity for contribution is the lack of meaningful outlet when society as a whole is organized to funnel the product of societal efforts into the hands of precious few. We are not organized to maximize the efforts of our citizens, but rather maximize the wealth of a select few.

Why bring this up? What could this possibly have to do with science?

Everything. Scientists are part of the overall trend where our efforts are thwarted by the overall tendency toward massive redistribution of resources.

What separates privilege from entitlement is gratitude.

― Brené Brown

The inequality is society is impacting science massively. Money has become the barometer by which all things are measured. It is seemingly the one size fits all metric for the value of all things. It is driving the short-term thinking permeating business thinking today. The reasoning for the short-term thinking is its ability to maximize the flow of resources into the hands of the already rich in the credit and investment markets; it is not because it maximizes the aggregate benefit to society or even the long-term benefit to business. The policies governing our World today are for the benefit of the few, not the many.

In a nearly Pavlovian sense, science has adopted this philosophy of governance because business knows how to run everything. Science is part of the overall political environment and subject to its trends. The results of this approach are wrecking havoc across the board. It is ruining academia by shifting the core focus of institutions toward short-term maximization of grants (i.e., basic cast flow) over the core principled approach of education. This is driven by and justified by the lack of societal investment in higher education. The research focus of universities is being driven toward the “publish or perish” philosophy that hollows out the depth of the research while draining away the critical training of the next generation of researchers. The consequences are grim. We are producing much more work, with much less quality and impact.

While the government research Labs are relatively better in terms of available resources, the short-term thinking is driving away real innovation and ruining this precious resoruce. It has been impacting the Labs like a hostile takeover; the stores of knowledge are being sold off to the highest bidder, and nothing is being done to replenish them. The sort of deep multi-disciplinary science that made these Labs great is being traded away for survival. The impact on quality is similar to what the university system is creating.

All told the impact of these forces means that big thorny problems will go unsolved. It doesn’t mean that they can’t be solved; it means that it is much less likely to see great work done. It isn’t merely a resource issue, but how those resources are arrayed. Those with more resources, like the Labs, will make breakthroughs more often. Overall, the current risk-adverse, short-term-focused, business-inspired approach is tailor made for assuring no progress on anything difficult.

We will continue along the path where mediocrity will be sold as success until we change our attitudes to value the long-term and things beyond money. This will happen because our current trajectory is unsustainable.

Nonetheless, I learned an important lesson yesterday, I’m more fortunate than many. I still hold to the core principle that quality of circumstance is not improved by the misfortune of others. Just because I’m better off and relatively lucky doesn’t make it good; instead it just makes it better than awful.

 A people that values its privileges above its principles soon loses both.

― Dwight D. Eisenhower

The first principle is that you must not fool yourself — and you are the easiest person to fool.

― Richard Feynmann

As computing becomes more and more important in science and engineering their credibility becomes essential. We are increasingly relying upon modeling and simulation to provide key input to a variety of endeavors many of which are high consequence. A good question to ask is why should I believe a computation? Why should I trust it? Why do I believe it? Do I know where the calculation breaks down? Do I understand the errors that are incurred in simulation? What could go wrong and what has been done to minimize that possibility?

Belief can be manipulated. Only knowledge is dangerous.

― Frank Herbert

This is where the entire practice of verification and validation comes in. The entirety of what V&V does (or should do) is developing an evidentiary basis for answering the above questions in an affirmative manner. Doing V&V is a way of establishing the credibility of a calculation or calculational capability. A key is the lack of definitive proof in computations; you only build the case that the computation is credible. In a deep sense V&V is the due diligence aspect of computing and as due diligence it is both incredibly important and terribly unsexy.

Science is the belief in the ignorance of experts.

― Richard Feynmann

Once the basic credibility has been established using V&V one can start to work toward enhancing this state. A key aspect of V&V regards a two-sided relationship for credibility. Often it is the positive credibility affirming aspect where the correctness of a computation is demonstrated which is thought of. The flip side of the picture is the negative aspects of V&V. These are equally important where V&V finds the limits of computational capability. In this sense the bounds of knowledge and capability are mapped through V&V. This defines useful and important work that can expand the capability toward new vistas.

All along the path toward increased credibility whenever you are doing useful work with computation uncertainty should be quantified. This experience should be the stock and trade with every verification and validation exercise. Often it is not. With verification, the error should be estimated, and the order of accuracy quantified. Far too often it is not conducted. The usual credibility is the establishment of mesh sensitivity, which is dangerous because it often gives a false or wrong sense of confidence. Uncertainty quantification should always accompany the validation exercise so that the simulated results can be compared directly with the experimental including the uncertainty.

The easier it is to quantify, the less it’s worth.

— Seth Godin

All of these provide a leveling of the credibility of the simulation for the intended use. In addition to providing the credibility basis, the V&V with associated UQ provides the methodology for defining details of any use of computation in a serious manner.

Doubt is an uncomfortable condition, but certainty is a ridiculous one.

― Voltaire

To avoid criticism say nothing, do nothing, be nothing.

― Aristotle

Whenever I get a review of a paper or proposal, I feel some deep sense of trepidation upon receiving it. The first time I read one; the sense of dread permeates my thinking. Why? I like most others don’t like being criticized, and I’ve had some really awful reviews over the years. The worst have been the product of a single anonymous reviewer, who I happen to know the identity of.

If we had no faults we should not take so much pleasure in noting those of others.

― François de La Rochefoucauld

My very first “real” research article was never published. I had inadvertently stepped into the middle of an extremely contentious topic (almost a religious war from what I found out later). Three famous scientists reviewed the paper; one of them identified himself to me via letter (the late Ami Harten). His review was quite cursory, simply “accept, and publish immediately.” I was beside myself with joy. That emotion was short-lived.

It’s easy to attack and destroy an act of creation. It’s a lot more difficult to perform one.

― Chuck Palahniuk

The second review was the complete opposite. It was technically supremely well accomplished. As professional as the technical content was, the writing was beautiful, but horribly unprofessional. It was venomous and full of ad hominem attacks. Due to an editorial mistake I found the reviewer’s identity (perhaps it was even semi-intentional).

The pleasure of criticizing takes away from us the pleasure of being moved by some very fine things.

― Jean de La Bruyère

A few years later I got another review from the same person. It was stunning in exactly the same way as the first review. I figured out who it was because the figures included in the review were identical to those in an article I was reading. Given this evidence and the style of the review, I knew who it was. I’ve met this person and he is brilliant, and quite friendly and kind in person, yet under the cloak of anonymity he is an awful person. His venom tipped writing blunts the extraordinary technical quality of the work he puts into the review.

I should keep this experience in mind because I review papers all the time. I turned in one review last Thursday and have four more in my queue. To be honest I rarely return reviews of the technical quality of the ones I mention above. I hope and pray that I never match the above-stated level of unprofessional and personal venom toward any author. It should be something that I keep in the back of my mind. A couple of events in the last week put this front and center in my thoughts.

Critics are our friends, they show us our faults.

― Benjamin Franklin

Yesterday I was doing a different sort of refereeing, soccer games. I had a couple of very competitive youth games. I also work with a gifted an extremely successful young referee who is half my age, but much more highly ranked. At the end of my match as the main, center referee, he gave me a critique of my performance. I certainly didn’t have my best game, but it wasn’t bad either. His feedback to me felt extraordinarily harsh, and even a bit personal. He has been reviewed by the very top referees in the United States (i.e., assessed). I took his comments with the utmost seriousness, but still his style made me feel awful, and undermined the effectiveness of the comments.

This young man has as you might expect a very serious style, which is keenly reflected in his success and personal style. Plus critiquing someone twice your age can’t be easy. So, I will take his feedback to heart, and make efforts to improve my performance (work on positioning, signaling advantage and style in man-management). In addition I need to reflect upon the dynamic of critique in every part of my life.

A coach is someone who can give correction without causing resentment.

― John Wooden

Work provided yet another situation that connects with each of these instances. It also reflects deeply on why V&V so often generates such negativity. I reviewed some very good work in analysis by a couple of younger staff. The issue is that the standard practice in analysis is lacking in a rather critical way in one respect. The key to the problem is that the young staff followed, standard accepted practice, yet failed to examine and recognize a critical source of error. In a sense the real core of the problem lies with their seniors, who have mentored the younger staff and set the standard they follow.

Often those that criticize others reveal what he himself lacks.

― Shannon L. Alder

I won’t get into the specifics, but it goes directly at uncertainty quantification and setting quantities of interest. Both issues set the stage for the problem that arose. In doing analysis the quantity of interest is often set by the application and defined by worst-case issues. Unfortunately, these worst cases quantities are horribly behaved numerically. They put the numerical methods under enormous pressure, and show them at their very worst. Uncertainty quantification is a big deal these days, and expected from first-class analysis. The combination of the awful quantity of interest and requisite poor numerical behavior, the analysts have shied away from examining numerical convergence and estimating numerical error.

The trouble with most of us is that we’d rather be ruined by praise than saved by criticism.

― Norman Vincent Peale

Instead the standard practice is conducting mesh sensitivity, but not verification analysis. Mesh sensitivity usually looks at the change in a quantity of interest across several meshes. If it is small, then the mesh is assumed to be adequate. This is the heart of the problem. If the convergence rate is low, the mesh sensitivity can be very small, and the numerical error can be very large. In the case I examined this was the case. In the final analysis the numerical error that had been assumed to be small was almost as large as all the other uncertainties. The convergence rate was so low that the numerical uncertainty was about 10 times larger than the mesh sensitivity would have indicated.

I communicated these results to the younger staff, who thanked me, but I fear did nothing else. We have to provide highlights of our work each quarter, and this work was written up as my contribution. Some of these reports get kicked up the chain, and this one was chosen. Ultimately a few of these reports get sent to Washington, and this one made the cut.

Getting chosen is a two-edged sword, it is good to be recognized, but given the critical nature of the work, it hurt feelings too. This is an unintended consequence of being critical. This is the heart of why V&V is so often reviled.

A critic must be knowledgeable in several fields, practices, and mediums. Brushing off art that they personally don’t understand, is not a critique.

― Justin K. McFarlane Beau