I’m a progressive. In almost every way that I can imagine, I favor progress over the status quo. This is true for science, music, art, and literature, among other things. The one place where I tend to be status quo are work and personal relationships that form the foundation for my progressive attitudes. These foundations are formed by several “social contracts” that serve to define the roles and expectations. Without this foundation, the progress I so cherish is threatened because people naturally retreat to conservatism for stability.

Without deviation from the norm, progress is not possible.

― Frank Zappa

What I’ve come to realize is that the shortsighted, short is demolishing many of these social contracts –term thinking dominating our governance. Our social contracts are the basis of trust and faith in our institutions whether they are the rule of government, or the place we work. In each case we are left with a severe corrosion of the intrinsic faith once granted these cornerstones of public life. The cost is enormous, and may have created a self-perpetuating cycle of loss of trust precipitating more acts that undermine trust.

 Stagnation is self-abdication.

― Ryan Talbot

Take the Labs where I’ve worked. At one time the Lab’s were trusted with the (nuclear) defense of the Nation. This happened in a time of immense threat and danger, yet the oversight was minimal. A substantial resource was given to the Labs to pursue the mission, and the Labs performed marvelously. The Labs fulfilled their social contract with the Nation, and similarly the Labs created a social contract with its employees. Serve here, and we will take care of you. You will be given engaging work, paid well, and ultimately allowed to retire comfortably. Shape your scientific explorations in service of the National security mission, and you will be provided resources. Beyond the direct success in the nuclear work, the scientific work was part of the Nation’s preeminence internationally and produced much of the foundation for great economic success. We have almost systematically destroyed everything good about these Labs.

It takes strength and courage to admit the truth.

―Rick Riordan

Even before the Cold War ended, this social contract began to unravel. The trust eroded and the money came with increasing strings attached. Similarly, the social contract with the employees became too “expensive” to fulfill. Over time the lack of trust and the associated “accountability” has spiraled out of control (we will spend ten dollars to save one). This has precipitated the no risk, no failure allowed environment that is choking innovation and progress out of our work. Increasingly the support for a career at the Labs is being removed, and it’s turning into just another job (not a bad one, but nothing special either).

These developments are paralleled by changes across the economy. They are manifestations of the short-term quarterly return mentality ruling industry. Research and development without immediate impact on the bottom line are increasingly missing from industrial research (missing from government research too). Employees are commodities whose life and career prospects is none of the business concern. The Labs benchmark themselves to these industries and share these attitudes because it benefits the short-term balance sheet.

What gets lost? Almost everything. Progress, quality, security, you name it. Our short-term balance sheet looks better, but our long-term prospects look dismal. The scary thing is that these developments help drive conservative thinking, which in turn drives these developments. As much as anything this could explain our Nation’s 50-year march to the right. We have taken the virtuous cycle we were granted, and developed a viscous cycle. It is a cycle that we need to get out of before it crushes our future.

Any defensiveness is a sign of failure. You can’t move forward if you are defensive.

― Bryant McGill,

We got here through overconfidence and loss of trust can we get out of it by combining realism with trust in each other. Right now, the signs are particularly bad with nothing looking like realism, or trust being part of the current public discourse on anything.  

The people I work with like to think of them as being smart, and they are. Most of all they like believing that they are right, and that they have solved important problems. The truth is a bit more complicated than that. The truth is that we know a lot less than we would like to admit. Actually we know hell of a lot less. Admitting this to us is scary.

Denial can be beautiful

But only when you’re a fantastic liar

― Kim Holden,

Still, many of us would like to convince ourselves of the opposite for the comfort it provides the soul. This is a selfish and self-serving mentality, which most clearly comes from the desire to have accomplished more than we have. Often the honesty in admitting our flawed knowledge and capability for understanding is too much to bear and we submerge it in falsehoods. This is basic human nature and it is inescapable.

Humankind cannot bear too much reality.

― T.S. Eliot

It seems to be a lot easier to metaphorically put our heads in the sand. A lot of the time we go to great lengths to convince ourselves of the opposite of the truth, to convince ourselves that we are the master’s of the universe. Instead we can only achieve the mastery we crave though the opposite. We should never consider our knowledge and capability to be flawless, but flawed and incomplete.

Integrity is telling myself the truth. And honesty is telling the truth to other people.

― Spencer Johnson

This comes up all the time when you’re doing V&V. When V&V gets into the assessment mode we constantly butt our heads into the people putting their heads in the sand. They want to think that everything is OK and mastery is at hand, and the problems that exist are under control. The reaction to finding problems is often full of emotion and anger because the truth is so unpleasant. It confronts the control of reality they have worked hard to build for themselves.

It takes strength and courage to admit the truth.

― Rick Riordan

Often the control they have is local and it is really OK. For a lot of modeling the results are calibrated heavily and as long as the analysis is done close to where the calibrating data was taken there is credibility. In the long run this is a difficult circumstance that is unsustainable. The problem is that the calibrated modeling is often quite successful. The people applying calibrated models are often lauded as the models of success. The problems with this are deep and pernicious. We want to do much more than calibrate results, we want to understand and explore the unknown. The only way to do that is systematically uncover our failings, and shortcomings with a ken focus on exposing the limits we have. The practical success of calibrated modeling stands squarely in the way of pushing the bounds of knowledge.

A program that produces incorrect results twice as fast is infinitely slower.

–John Osterhout

A close analog to this process exists in code development. In debugging code finding a bug is to be celebrated, but one should never believe that all the bugs have been located. A more healthy and reasonable philosophy is to assume that more bugs are hiding waiting to be discovered. Anyone who thinks their code is bug-free is delusional. Such attitudes are rightly greeted with skepticism. The same skepticism should be offered to those who think their modeling is similarly “bug-free”.

Most men would rather deny a hard truth than face it.

― George R.R. Martin

It is so very difficult to achieve because of the human element; it is because people would rather think that things are better than they actually are. It takes leadership to overcome these issues. An environment that provides the impetus for improving modeling quality means confronting issues that make people uncomfortable. Specific measures need to be taken that reward people for finding “bugs” in the modeling capability. Finding problems in the modeling needs to be expected, instead of confirmations of mastery.

Quality means doing it right when no one is looking.

― Henry Ford

This is your life and its ending one moment at a time.
― Chuck Palahniuk,

We are encouraged by everything around us to work on things that are important. Given the intrinsic differences between the messaging we are given explicitly and implicitly, its hard to really decide what’s important. Of course, if you work on what’s important you will personally make a bit more money.  You really make a lot of money if you work specifically in the money making industry…

Where I’ve worked, the explicit message is that nuclear weapons are important. Science for nuclear weapons should be the sweet spot, but its not really. Part of the problem is that science sells itself to scientists. They actually have to pay us less to get us to do science. On the other hand, the implicit message for all of society is money is important. Nuclear weapon’ labs are no different, money is the most important things there too. Its become pretty clear that the same superficial value system present society-wide is at work there too.

At a personal level you also like to do interesting work. Sometimes interesting and challenging are equated. You get into science because it is both interesting and you’re good at it.  Someone will pay you to do something you’re good at and is interesting. You convince yourself that it is good for society, good for your Nation, and the right thing to do. As you get older, some of these things start to come into question.

The messages you begin to get implicitly start to undermine all those things you convinced yourself of. Interesting work keeps you sane; important work keeps you paid. Maybe there isn’t much more that you can hope for.

Nothing of me is original. I am the combined effort of everyone I’ve ever known.
― Chuck Palahniuk

These words are spoken whenever we go into planning “reportable” milestones in virtually every project I know about. If we are getting a certain chunk of money, we are expected to provide milestones that report our progress. It is a reasoned and reasonable thing, but the execution is horribly botched by the expectations that are grafted onto the milestone. Along with the guidance in the title of this post, we are told, “these milestones must always be successful, so choose your completion criteria carefully.” Along with this we make sure that these milestones don’t contain too much risk.

That would be dangerous.

People who don’t take risks generally make about two big mistakes a year. People who do take risks generally make about two big mistakes a year.

― Peter F. Drucker

Dangerous for whom, I wonder? Dangerous for what reason I ask?

 The desire for safety stands against every great and noble enterprise.

― Tacitus

The real danger in the philosophy we have adopted is the creeping intrusion of mediocrity into everything we do. Nothing is important enough to take risks with. The thoughts expressed through these words are driving a mindless march toward mediocrity, once great research institutions are being thrust headfirst into the realm of milquetoast also-rans. The scientific and engineering superiority of the United States is leaving in lockstep with every successfully completed milestone built this way.

There is no discovery without risk and what you risk reveals what you value.

― Jeanette Winterson

Science depends on venturing bravely into the unknown, a task of inherent risk, and massive potential reward. The reward and risk are linked intimately; with nothing risked, nothing is gained. By making milestones both important and free of risk, we sap vitality from our work. Instead of wisely and competently stewarding the resources we are trusted with, they are squandered on work that is shallow and uninspired. Rather than being the best we can do, it becomes the thing we can surely do.

Never was anything great achieved without danger.

― Niccolò Machiavelli

When push comes to shove, these milestones are always done, and always first in line for resource allocation. At the same time we have neutered them from the outset. The strategy (if you can call it that!) is self-defeating, and only yields the short-term benefit of the appearance of success. This appearance of success is believed to be necessary for continuing the supply of resources.

The long-term cost is systematic atrophy of the core identity of the Labs as centers of excellence. Today the Labs are being transformed into centers of compliance. No one can solely take the blame for this, it is the handiwork of the entire system.

You cannot swim for new horizons until you have courage to lose sight of the shore.

― William Faulkner

If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.

― Albert Einstein

If you haven’t heard of “wicked problems” before it’s a concept that you should familiarize your self with. Simply put, a wicked problem is a problem that can’t be stated or defined without attempting to solve it. Even then your definition will be woefully incomplete. Wicked problems are recursive. Every attempt to solve the problem yields a better definition of the problem. They are the metaphorical onion where peeling back every layer produces another layer.

Wicked problems were introduced as a way of defining the sorts of issues dealt with in the social sciences. There the human element provides a source of infinite complexity. Issues such a poverty and race relations are the archetype of wicked problems. Virtually every response to solving the original problem creates more problems itself as well as informing future approaches. The concept defines the difficulty in providing well-defined, finite solutions that are simple and easy to apply.

The formulation of the problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill.

― Albert Einstein

There are ten classical characteristics of wicked problems to consider in deciding if your problem is indeed “wicked”. Defining the wicked problem is difficult to impossible.

2. You don’t know when to stop solving the wicked problem (because its never solved, just solved well enough, or you simply run out of time).
3. No proposed solution can be complete, but can be defined in terms of quality (good or bad, good enough).
4. Testing solutions to the wicked problem is inconclusive and limited.
5. Every solution to the wicked problem to a problem has consequences and implications. These are unpredictable.
6. The set of potential solutions to the wicked problem do not form a well-defined or bounded list.
7. Each wicked problem is different.
8. Each wicked problem is connected to other problems.
9. There are multiple explanations for the causes of wicked problems. Some of these explanations are conflicting.
10. Wicked problems have consequences if things are wrong, thus solutions are expected to be right the first time. Failures are punished due to consequences.

Over time we have come to realize that science and technology has its own cadre of wicked problems. I first encountered the concept in code development. Steve McConnell made the observation that code development is wicked in his classic book “Code Complete”. You can’t even anticipate all the problems you’ll have to solve in developing a code until you start to code. Deep aspects of the code will unveil themselves in the process of implementation and testing.

In code development this often takes the form of refactoring where the original design of part of the software is redone based on the experience gained through its earlier implementation. You understand the use of and form that the software should take once you’ve tried to write it (or twice or thrice or…). The point is that the implementation is better the second or third time based on the experience of the earlier work. In essence this is embracing failure in its proper role as a learning experience. A working, but ultimately failed form of the software is the best experience for producing a better piece of software.

We can not solve our problems with the same level of thinking that created them

― Albert Einstein

This principle applies far more broadly to scientific endeavors. An archetypical scientific wicked problem is climate change not simply because the complexity of the scientific aspects of the problem, but also the political and cultural dynamics stirred up. In this way climate change connects back to the traditional wicked problems from the social sciences. A more purely scientific problem that is wicked is turbulence because of its enormous depth in terms of physics, engineering and math with an almost metaphysical level of intractability arising naturally. Turbulence is also connected to a wealth of engineering endeavors with immense economic consequences.

This brings me to the practice of verification and validation with uncertainty quantification. This is most certainly wicked as well. One of the clear signs of wickedness is the PCMM (Predictive Capability Maturity Model). PCMM has evolved continuously toward ever-greater complexity as experience grows without any limits in sight. Other frameworks for wrangling the complexity of V&V exist with entirely different structures.

We easily start to see the entire engagement with V&V is inherently wicked. The more you do V&V and solve problems using its principles, the more complex and deep the topic seems. This makes V&V appear to be more daunting than it actually is. Experts in V&V with a deep knowledge have difficulty in pulling themselves back to a novice’s perspective. Ultimately this is counter-productive as it makes the topic difficult to approach once it matures.

Another aspect of wicked problems that manifests with V&V is cultural and political consequences. V&V intersects with the culture of engineering and science in aggressive ways. V&V pushes the cultural norms of conducting science and engineering. Ultimately V&V collides with the political side of things with decision-making or resource allocation. Once this occurs you’re down the proverbial rabbit’s hole, and complexity explodes. You will then understand intuitively that you have found a wicked problem.

Maintaining the perspective of wickedness as being fundamental is useful as it drives home the belief that your deep knowledge is intrinsically limited. The way that experts look at V&V (or any other wicked problem) is based on their own experience, but is not “right” or “correct” in and of itself. It is simply a workable structure that fits the way they have attacked the problem over time.

 I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

― Abraham Maslow

It is more shameful to distrust our friends than to be deceived by them.

― Confucius

I’ve continued to think about that planning session a couple of days ago. It was utterly stunning that a group of people could simultaneously come up with the same idea of what needed to be done, and be totally convinced that it was impossible. Not technically impossible, but culturally, and practically impossible due to political constraints. Let’s be clear, the solution we came up with would be transformative to how we execute our mission. It would drive meaningful use and need for the cutting edge (exascale) computing, we are blamed for not having. Despite all of this, it seems unthinkable to even attempt it.

Success is most often achieved by those who don’t know that failure is inevitable.

― Coco Chanel,

How can this be? I think it boils down to trust, or more properly the lack of it. We have become a low trust society and the current trends seem to be pushing us to even lower levels of confidence and faith in our fellow citizens. Democrats don’t trust Republicans and vice versa, white doesn’t trust black, young and old lack trust for each other. The rich don’t trust the poor, and even trustworthy professions like medicine, science and engineering aren’t trusted. We imprison a massive portion of our population to a point that exceeds the rate everywhere else on Earth. Everywhere we look the evidence points to little confidence or faith in the best intentions of our fellow citizens.

The implicit costs of all of this lack of trust are immense. I’m only seeing a small amount of it in the lack of high-minded risk-taking encouraged in science. People won’t engage in anything risky or overly challenging because there isn’t any faith that they won’t be skewered if they don’t deliver on promises. Ultimately everyone is afraid of appearing in some congressional hearing and being made into a political football to be kicked around.

The fear-based politics and shame baiting does everything to make things worse. The ever-growing evidence that our democracy is devolving into oligarchical, kleptocratic system of privilege over substance only drives a deep-seated feeling of untrustworthiness. Given the Orwellian nightmare unfolding with surveillance of every moment of our lives coupled with the capacity of drones to kill people at will makes every untrusting thought seem more rational.

At one time, we trusted massive resources to people to accomplish big things. They were accountable for doing the best job possible to achieve the goals. The difference with today is that we hadn’t broken down every step, every month-week-day of effort into an accounting superstructure designed to track every single action. Just because we can easily create a system to account for something doesn’t make it a good idea. The impact of this is to be a straightjacket on every effort, and assure that we accomplish much less rather than just breakeven. That’s right, I would be happy if we just stopped losing ground. Every year we become less productive.

If we could muster the courage to trust each other, we would be able to craft a much better, much happier, much richer future.

 The best way to find out if you can trust somebody is to trust them.

― Ernest Hemingway

The difference between hope and despair is a different way of telling stories from the same facts.

― Alain de Botton

At work yesterday we had a planning meeting. We did an exercise where we were asked to envision an optimal future. What could we work on, if we could? We split up into small four or five person groups to discuss and jot down ideas. Remarkably, almost everyone came up with the same thing. Not exactly the same, but the same big conceptual idea. I found it one of the most hopeful things to happen at work in a great while.

In all honesty I’d always felt there was a surprising lack of vision of the future at work. Things just seemed to be terribly pedestrian and practical with very little inspiration. Instead, among my peers, I found a kinship with similar dreams of what we should be doing. It was incredible to see and I left with a new respect and admiration for all of them. Everyone knew going in that the reality of today’s world made these dreams impossible to realize. All of the participants had enough hope to allow themselves to see something more. The downside is that reality is swallowing the capacity to hope and dream for something better.

There is only one thing that makes a dream impossible to achieve: the fear of failure.

― Paulo Coelho

This is truly sad considering the transformative potential bound up in those hopeful, unrealistic, dreams we allowed ourselves to express. We could be doing things that are magnificent; instead we withdraw to the world of the possible and bureaucratically controlled, politically viable reality. The projects we hopefully envisioned would be transformative and create a far greater future than the path we are currently on. We are told that the people in Washington can’t envision anything greater either. Perhaps they are just like us, simply unwilling to honestly voice anything greater than our currently pedestrian path.

Everyone must dream. We dream to give ourselves hope. To stop dreaming – well, that’s like saying you can never change your fate. Isn’t that true?

― Amy Tan

This is why the future is so bleak; the dreams are there, but no one has faith that these dreams can be realized. Support for working on the dream is missing, why start something that will never be finished? People have recently realized that the future was supposed to bring flying cars and instead we got mini-supercomputers in our pockets (that do very little computing). Of course it doesn’t quite look like “Blade Runner” or “Minority Report” either. The problem is that it looks like the dystopian parts of those movies have a greater chance of reality than the cool parts.

The thing that binds us together is that we have both lowered our expectations of life

― Orhan Pamuk

This is the point of despair. We could be so much more than we are; instead we are so much less than we could be.

When a man is in despair, it means that he still believes in something.

― Dmitri Shostakovich

Whatever the status quo is, changing it gives you the opportunity to be remarkable.

— Seth Godin

The concepts of cost and value should share a relationship of symmetry. It seems that more often than not, such symmetry is missing. Quite often the value doesn’t seem to be aligned with the cost (that watch cost how much!?). The key is to look deeper, the costs and the value are aligned, but the truth is often hidden by shame. The shame is often the explicit admission that we value things that are so inherently superficial and ultimately damaging to our future.

Anything that just costs money is cheap.

― John Steinbeck

Much of the actual value we take in items is hidden. It also matters greatly who does the evaluation. This is clearest with luxury items where people are willing pay a great deal for a brand name because of its cachet. This is a how top designer and brands make their money; people are willing to pay significant premium for having the product with a name. Examples can be found with shoes, cars, watches, and handbags among many items. It becomes a sign of distinction just to own certain items.

The same dynamic is playing out with supercomputers. In this field there is a great value placed in being #1. For this reason we are quite willing to build virtually unusable computers created solely for the purpose of being the fastest using the standard. The standard is the Linpack benchmark, basically the LU decomposition of a dense system of linear equations. As a measure of scientific computing performance this benchmark should have lost all relevance decades ago, yet it lives on shaping supercomputers and driving decisions.

It all revolves around having a supercomputer that is defined as fast being a luxury item of great value. The damage done by this state of affairs is tremendous although it remains unremunerated because most of the damage is tied up in the alternatives not explored because of the pursuit of this luxury item. The irony is that computer speed was valued because of a computers utility as a problem-solving tool. Ironically we have taken these marvelous tools and rendered them less useful to make them faster. We have taken a tool and created a fetish. It isn’t so amazing or awful because people do this so often. A gun or car or any of the luxury items mentioned about is a similar fetish. The awful part is the lack of recognition that this is what we’ve done!

No human endeavour can ever be wholly good… it must always have a cost.

― William Golding

The other side of the dynamic is also in evidence when the value of something is gained without paying the true cost. A chief example is the acquisition and use of natural resources. The long-term costs of mining are often not carried by those who gained profit through the acquisition of a mineral, but pay nothing for the environmental carnage unleashed. These costs are seemingly hidden because they only manifest themselves downstream although a rational examination of the situation would have yielded a determination of the costs up front.

But nothing’s really free, is it? People always make you pay one way or another.

― Jane Lotter

Opportunity cost is similar. The cost of doing one thing over another is not readily evident. The reality is that we don’t know the outcome of roads not taken, but we can reasonably assess it. In the scientific world, the balance between basic and applied research can define this. Applied research can be viewed as short-term profit taking whereas basic research can be viewed as the long-term, high-risk investments.

In the area of computing other opportunity costs are in evidence. We have made significant effort to make high performance computing about big computers solving meaningless benchmarks. We have defined weak scaling as a success. These have come at significant costs such as the diminishment of the efficiency of doing most of our computational work because we have failed to focus on performance at the basic node level. Our pursuit of mistaken goals in high performance computing has also driven a divestment from many important areas of algorithmic research. Perhaps the most powerful tool for effectively using computers, the algorithm, has been stripped of its vitality in the process.
This has been merged with the wholesale change in the computing market, which as transitioned from mainframes to personal computers to hand-held. In doing so the entire focus of commercial computing has changed as well as grown to a scale unimaginable a generation ago. With this change the opportunity costs are snowballing into something that should start getting attention.
Consumptive spending compared to investment is a keen societal example where near term value is taken by using resources now. Investments such as infrastructure or research yield less immediate benefits, but usually provide greater returns over the long run. We have made
serious marketing choices by masking our short-term business profiteering and general lack of investment by merely relabeling such uses of resources as “investment” when it is nothing of the sort. The true cost is implicit and defines much of the economic trouble we find ourselves in today.

The riskiest thing we can do is just maintain the status quo.

― Bob Iger

What’s measured improves

– Peter Drucker

I’ve been doing a fair amount of thinking about metrics for computer performance. The question is of supreme importance moving forward with supercomputing. The Drucker quote fairly well captures both the importance of answering the metrics question moving forward, but also consideration of what happened in the recent path. I’ve come to the conclusion that the devotion to “weak scaling” is probably doing a lot of damage to high performance computing.

What is weak scaling? From http://en.wikipedia.org/wiki/Scalability

• The first is strong scaling, which is defined as how the solution time varies with the number of processors for a fixed total problem size.

• The second is weak scaling, which is defined as how the solution time varies with the number of processors for a fixed problem size per processor.

The problem is that weak scaling has allowed a number of issues to be papered over. The biggest of these issues is the continuing diminishing performance on node, or the ability to efficiently use the full computing capacity of the machines. We get “better” by using more nodes, but the nodes are used very poorly. A big issue is the memory bandwidth and cost. Basically the processors are starved for data and useful work to do.

The other thing that we are missing is algorithmic development. The success given to weak scaling has starved the development of better algorithms. We are spending a lot of effort implementing old algorithms to achieve weak scaling. Algorithmic scaling is actually much harder, and given the success with which a weak scaling is greeted, the emphasis is chosen by fiat. The fact is that implementing an old algorithm on a new computer is low risk compared to developing new algorithms. Given our propensity toward low risk (i.e., risk management), we get today’s situation; computational science in the doldrums and a lack of real innovation and progress.

It all comes down to what we measure. As long as weak scaling is deemed a success we can continue to ignore the issue. We can hope that the issue will be ultimately put squarely in the frame by the issues of power consumption, which is creating issues that can’t be ignored (the electric bill).

I think the real metric should be something like time or cost to produce a solution of a given, credible quality. It should include wall clock time and the cost of labor. Finally we should consider the costs associated with the lack of basic vitality of the community. We could stand to have some significantly greater inspiration.

In preparing for battle I have always found that plans are useless, but planning is indispensable.

― Dwight D. Eisenhower

Many of you might be familiar with the PCMM (Predictive Capability Maturity Model). Most commonly it is thought of as structured way to assess (i.e., judge) the quality of a computational investigation. Since no one likes to be judged, it isn’t greeted with the greatest warmth. I’d like to provide another way of looking at PCMM, as a rubric for doing computational science.

In this sense it provides a catalog of things that you should worry about, decide on your intent and shape the work to meet specific goals and intended uses. The concept of broad categories of competence with graded levels of quality allows one to make sure all the necessary activities are being done. Too often a program will be executed almost solely within a single category and the other aspects of computation would be cobbled together. The intended quality of the work is an afterthought.

With the traditional structure of the PCMM as a starting point, there is no problem in removing or adding new categories to suit the purposes of the project. The usual set of categories: geometry/representation, physical modeling, code/software verification, solution verification/error estimation, validation, and uncertainty quantification are a good place to start, but inadequate for many projects. An example might be weather or climate modeling where data assimilation is important enough to warrant its own category. In computation of social science, the geometry is irrelevant and needs to be replaced with an appropriate description of the environment things like agents are placed in. In other cases the experimental work is sufficiently complex and focused that it should be expanded into far greater detail included a data focus. The point is that PCMM is not a fixed framework, but an idea of how to organize your activity as to not leave important things out.

An idea can only become a reality once it is broken down into organized, actionable elements.

― Scott Belsky

How might I use PCMM to do something that isn’t V&V related first? Say, like writing a new code?

I would consider what the application of the code is intended to be and how much further than the original intent might be supported? How essential is the geometric fidelity to the quality of simulations? How well are the basic physical models, and supporting constitutive relations established? Is the numerical method and the equations supported by mathematical rigor? Are numerical errors well understood where the equations and method are to be applied? What experiment exist for validation, and will new validation experiments be conducted? What sort of quantities of interest are needed and how will their uncertainty be assessed? For every question how critical is the quality of the answer, and what is the level of decision to be made with the results? Might any of this change over time, and can those changes be accommodated in the desired code?

Notice that this is the application of PCMM classic in the form of a set of questions to be asked at the outset of the work. If these questions are proactively addressed the code will be able to be assessed for V&V quality with great ease. Ultimately this should lead to an appropriate statement of the computational credibility of the code and its suitability for intended use.

The key is that PCMM isn’t just for V&V any more, and if it is used proactively V&V can be so much easier.

The essence of community, its heart and soul, is the non-monetary exchange of value; things we do and share because we care for others, and for the good of the place.

― Dee Hock