Your number-one mission as a speaker is to take something that matters deeply to you and to rebuild it inside the minds of your listeners. We’ll call that something an idea.
― Chris J. Anderson

Every September my wife and I attend the local TeDx event here in Albuquerque. It is a marvelous way to spend the day, and leaves a lasting impression on us. We immerse ourselves in inspiring, fresh ideas surrounded by like-minded people. It is empowering and wonderful to see the local community of progressive people together at once listening, interacting and absorbing a selection of some of the best ideas in our community. This year’s event was great and as always several talks stood out particularly including Jannell MacAulay (Lt.Col USAF) talking about applying mindfulness to work and life, or Olivia Gatwood inspiring poetry about the seeming mundane aspects of life that speaks to far deeper issues in society. The smallest details are illustrative of the biggest concerns. Both of these talks made me want to think deeply about applying these lessons in some fashion to myself and improving my life consequentially.

That’s part of the point of TeD, the talks are part of the gospel of progress, part marketing of great ideas and part performance art. All of these things have a great use to society in lifting up and celebrating a drive to be better and progress toward a better future. Humanity has immense power to change the world around them for the better. We can look across the globe and witness the collective power of humanity to change their environment. A great deal of this change is harmful or thoughtless, but much of it is a source of wonder. Our understanding of the World around us and the worlds within us has changed our biological destiny.

We have transitioned from an animal fighting for survival during brief violent lives, to beings capable of higher thought and aspiration during unnaturally long and productive lives. We can think and invent new things instead of simply fighting to feed us and reproduce a new generation of humans to struggle in an identical manner. We also can produce work whose only value is beauty and wonder. TeD provides a beacon for human’s best characteristics along with a hopeful forward-looking community committed to positive common values. It is a powerful message that I’d like to take with me every day. I’d like to live out this promise with my actions, but the reality of work and life comes up short.

There was a speaker from my employer this year, and there always is. There wasn’t anyone from my former employer, the other major scientific Lab in our state (what was once one of the premier scientific institutions in the World, but that’s a thing of the past). Also noticeable is the lack of support for the local TeD organization by either employer. I’ll grant you that Los Alamos has supported it in the past, but no longer. There’s probably some petty and idiotic reason for the withdrawal of support. My employer, Sandia, doesn’t support it, and hasn’t ever. It looks like our local University doesn’t support it either. I know that Los Alamos did their own local TeD conference and perhaps they thought that was enough TeD for them. That’s the sad best-case scenario, and I don’t know what the full story is.

For Sandia it’s not particularly surprising as it’s not exactly a progressive, idea-centered place, and these days no place is anyway. The University should be, but the lack of financial support from the state could explain it (its a common characteristic of GOP governance to eviscerate universities). It is quite hard for me to express my level of disappointment in these institutions’ lack of civic support for progressive thought. It is stark testimony on the current state of affairs where two National Laboratories and a University cannot be supportive of a major source of progressive thought in the community they are embedded within. An active progressive and intellectual community in the areas where these institutions are located should be beneficial for recruiting and retention of progressive and intellectual staff. It is one sign that this sort of long view isn’t at work. It is a sign of the times.

TeD talks are often the focus of criticism for their approach and general marketing nature strongly associated with the performance art nature. These critiques are valid and worth considering including the often-superficial nature of how difficult topics are covered. In many ways where research papers can be criticized increasingly as merely being the marketing of the actual work, TeD talks are simply the 30-second mass market advertisement of big ideas for big problems. Still the talks provide a deeply inspiring pitch for big ideas that one can follow up on and provide the entry to something much better. I find the talk is a perfect opening to learning or thinking more about a topic, or merely being exposed to something new.

Control leads to compliance; autonomy leads to engagement.

– Daniel H. Pink

One prime example is one of my favorite talks of all time by Daniel Pink (https://www.ted.com/talks/dan_pink_on_motivation). This talk is basically a pitch for the book “Drive” and touches only superficially on the topic. The book itself is a distillation of very complex topics. All of this is true, but none of this undermines the value in the ideas. TeD provides a platform to inspire people to do more and get closer to the actual application of the ideas to their lives (not just buy Pink’s book, the true cynics take on the purpose). Interestingly, the managers at work were also reading Pink’s book and discussing the ideas therein. The rub was the observation that I could not identify a single thing recommended in Pink’s book that made it to the workplace. It seemed to me that the book simply inspired the management to a set of ideals that could not be realized. The managers aren’t really in charge; they are simply managing the corporate compliance instead of managing in a way that maximizes the performance of its people. The Lab isn’t about progress any more; it is about everything, but progress. Compliance and subservience has become the raison d’etre.

For artists, scientists, inventors, schoolchildren, and the rest of us, intrinsic motivation the drive do something because it is interesting, challenging, and absorbing is essential for high levels of creativity.

– Daniel H. Pink

Intrinsic motivation is conducive to creativity; controlling extrinsic motivation is detrimental to creativity.

–Daniel H. Pink

This deep frustration isn’t limited to TeD talks; it is almost every source of great advice or inspiration available. Almost every manager I know reads the Harvard Business Review. I read it too. It is full of wonderful ideas and approaches to improving the way we work. It is impossible to see anything ever done with all the great advice or inspiration. My workplace looks like all the “before” cases studies in HBR and more like it every day, not less. Nothing ever recommended happens at work, nothing is tried, nothing changes in the positive direction; its like we are committed to moving backwards. HBR is progressive in terms of the business world. The problem is that the status quo and central organizing principle today is anti-progressive. Progress is something everyone is afraid of, and the future appears to be terrifying and worth putting off for as long as possible. We see genuinely horrible lurch toward an embrace of the past along with all its anger, bigotry, violence and fear. Fear is the driving force for avoiding anything that looks progressive.

Management isn’t about walking around and seeing if people are in their offices, he told me. It’s about creating conditions for people to do their best work.

– Daniel H. Pink

Now that I’ve firmly established the lack of relevance of TeD and progressive thought in my workplace, I can at least appreciate and apply it at a personal level. I’d love for work to reflect a place for genuine progress, but this seems a bridge too far today. Work is a big part of life and these observations are rather dismaying. Ideally, I’d like a workplace that reflects my own values. The truth of the matter is that this is nearly impossible for a progressive-minded person in America today. Even the bastions of progressive thought like Universities are not working well. Society at large seems to be at war with elites and progressive thought far more under siege than whites, or Christians. I can ask the serious question, how many atheists are in Congress? How much well proven and accepted science does our government reject already? Don’t get me started on our judicial system, or the war on drugs both of which focus far more on oppressing minorities than crime or drug abuse. The bottom line is the sense that we are in a societal backlash against change; so more progress seems to be impossible. We will be fighting to hold onto the progress we’ve already made.

Still I can offer a set of TeD talks that have both inspired me and impacted my life for the better. They have either encouraged me to learn more, or make a change, or simply change perspective. I’ll start with a recent one where David Baron gave us an incredibly inspiring call to see the total eclipse in its totality (https://www.ted.com/talks/david_baron_you_owe_it_to_yourself_to_experience_a_total_solar_eclipse). I saw the talk concluding that I simply had to go, and then I showed to my wife to convince her. It did! We hopped into the car at midnight the day of eclipse and drove eight hours to get from Northern Idaho to Eastern Oregon. We got off I-82 atDurkee finding a wonderful community center with a lawn and watched it with 50 people from all over the local area plus a couple from Berlin! The totality of the eclipse lasted only two minutes. It was part of a 22-hour day of driving over 800 miles, and it was totally and completely worth every second! Seeing the totality was one of the greatest experiences I can remember. My life was better for it, and my life was better for watching that TeD talk.

Another recent talk really provoked me to think about my priorities. It is a deep consideration of what your priorities are in terms of your health. Are you better off going to the gym or going to party, or the bar? Conventional wisdom says the gym will extend your life the most, but perhaps not. Susan Pinker provides a compelling case that social connection is the key to longer life (https://www.ted.com/talks/susan_pinker_the_secret_to_living_longer_may_be_your_social_life ). This gets at the disparity between men and women since women tend to connect in long life affirming friendships with greater ease than men. The talk is backed up by data, and by visiting places where people live long lives. These people live in communities where they are entangled in each other’s lives almost by design. It gets to the priorities associated with health care and self care along with the benefit of actions. Focusing on your social life is a genuinely beneficial act to prolonging your life.

Our modern computing world is a marvel, but it also has some rather pronounced downsides. In many ways our cell phones are making us far unhappier people. The phones and their apps are designed to grab, demand our attention. They can become sources of deep and pervasive anxiety. This is exactly what they are designed to do. As Adam Alter explains, an entire industry is set up to get as much of our attention as possible because our attention equals money, big money (https://www.ted.com/talks/adam_alter_why_our_screens_make_us_less_happy). He also explains that it doesn’t have to be like this. The same social engineering that has gone into making the phones so demanding could be harnessed to help us be better. If we balanced the naked profit motive with some measure of social responsibility, we might turn this problem into a benefit. This is a wonderfully inspiring idea; it is also terribly progressive and dangerous to the unfettered capitalism fueling this growing societal crisis.

Love rests on two pillars: surrender and autonomy. Our need for togetherness exists alongside our need for separateness.

– Esther Perel

The power of TeD extends to far deeper personal matters as well. A couple of talks by Esther Perel speak to reframing our love lives (https://www.ted.com/talks/esther_perel_the_secret_to_desire_in_a_long_term_relationship, https://www.ted.com/talks/esther_perel_rethinking_infidelity_a_talk_for_anyone_who_has_ever_loved ). Perel defies conventional thought on love, marriage and infidelity providing a counter theory to all these matters. Her first talk is an accompaniment to her first book and tackles the thorny issue of keeping your long-term relationship hot and steamy. It is a challenge many of us have tackled, and no doubt struggled with. This struggle is for good reasons, and knowing the reasons provides insight to solutions. Perel powerfully explains the problem and speaks to working toward solutions.

The thornier issue of infidelity is the second talk (and her brand new book). Like before, she tackles the topic from a totally different perspective. Her approach is unconventional and utterly refreshing. The new perspectives provide an alternative narrative to handling this all too common human failing. Explaining and understanding the complex root of this all-to-common relationship problem can improve our lives. It is an alternative to the moral perspective that has failed to provide any solutions. Among the threads to concentrate on is the relatively new character of modern marriage in the history of humanity, and the consequences of the deep changes in the institution. One of the beauties of TeD is the exposure to fresh perspective on old ideas along side completely new ideas.

The very ingredients that nurture love mutuality, reciprocity, protection, worry, and responsibility for the other are sometimes the very ingredients that stifle desire.

– Esther Perel

Truth and courage aren’t always comfortable, but they’re never weakness.

– Brene Brown

The last talk I’ll highlight today is truly challenging to most of us. Brene Brown is a gifted and utterly approachable speaker presenting a topic that genuinely terrifies most of us, vulnerability (https://www.ted.com/talks/brene_brown_on_vulnerability). Begin vulnerable is an immensely valuable characteristic that almost everyone struggles with. Vulnerable often equates with being weak, but also open and honest. That openness and honesty is the key to being a better person and developing better relationships. In many cases the weakness and honesty is shared only with yourself. In either case vulnerability provides an avenue to connection and an embrace of humanity that both frees you and allows deeper relationships to flourish. The freedom you give yourself allows you to grow, learn and overcome bad experiences.

What would you be glad you did–even if you failed?

– Brene Brown

I always wish that I could focus on most of what I hear at a local TeD event, but one must make choices, time and effort are limited. While I do plan to more mindfully apply mindfulness to my life, right now I’ll hedge toward the artistic side of things, if for no other reason that I usually don’t. I will close by honoring the inspirational gift of Olivia Gatwood’s talk on poetry about seeking beauty and meaning in the mundane. I’ll write a narrative of a moment in my life that touched me deeply.

The Best Gift

A night of enchanting companionship was drawing to a close,

and I was longing for one last kiss before parting

Those early autumn nights are so welcoming,

 

the crisp nights promised, but not yet arrived,

summer still alive, but fading

I hadn’t even bothered to fully dress for the goodbye,

 

Conventions and neighbors be damned

It was a warm evening and my skin wanted to drink it in,

drink her in too, one last time

 

We slowly made our way out to my driveway

talking, still flirting, our banter unabated

The moon full, bright, and peeking between the gaps in the single cloud

 

adorning the sky as it illuminates the night

It will light her way home as a warm beacon

“Good,” I think, “you’ll be safe” on your long drive home

 

We draw close to each other, pressing hard while

savoring the time spent together fun and friendship

with a depth that was unexpected, but welcome

 

You ask, “What would you like for your birthday?”

My mind goes to my elaborate tattoo to adorn me soon,

“I’m already getting what I want for myself”

 

“I always ask for more time,” she said longingly

Her words cut me to the core,

of course, what else would she want?

 

My head spins with the truth revealed by her breathtaking honesty,

with words failing me for a breath or two, … or three

My mind opens with the realization of her precious offering

 

“I just want good memories”

Realization washes over me, she just gave me the best gift I could have hoped for

We kiss deeply and parted until we next renew making good memories

 

 

You are not special. You’re not a beautiful and unique snowflake. You’re the same decaying organic matter as everything else. We’re all part of the same compost heap. We’re all singing, all dancing crap of the world.

– Chuck Palahniuk

This post was inspired by twin events: a comment from a dear friend, and watching the movie “Fight Club” again. This is my 300^th blog post here. Its been an amazing experience thanks for reading.

If you consider the prospect of retirement and you feel that your place of work does not need you and would not suffer from you departure, you aren’t alone. This is an increasing trend for work today. You are an imminently replaceable cog in the machine, which can be interchanged with another person without any loss to the workplace. Your personal imprint on the products of work is not essential and someone else could do exactly what you do. If you work in one of the many service industry jobs, or provide the basic execution of tasks, the work is highly prescribed and you versus someone else doesn’t matter much. If you are reliable, show up and work hard, you are a good worker, but someone else with all the same characteristics is just as good.

What’s measured improves

–Peter F. Drucker

I didn’t used to feel this way, but times have changed. I felt this way when I worked at McDonalds for my first job. I was a hard worker, and a kick ass grill man, opener, closer, and whatever else I did. I became a manager and ultimately the #2 man at a store. Still I was 100% replaceable and in no way essential, the store worked just fine without me. I was interchangeable with another hard working person. It isn’t really the best feeling; you’d like to be a person whose imprint on the World means something. This is an aspiration worth having, and when your work is truly creative, you add value in a way that no one else can replicate.

When I started working almost 30 years ago at Los Alamos, this dynamic felt a lot different. People mattered a lot, and an individual was important. Every individual was important, unique and worth the effort. As a person you felt the warm embrace of an incubator for aspiring scientists. You were encouraged to think of the big picture, and the long term while learning and growing. The Lab was a warm and welcoming place where people were generous with knowledge, expertise and time. It was still hard work and incredibly demanding, but all in the spirit of service and work with value. I repaid the generosity through learning and growing as a professional. It was an amazing place to work, an incredible place to be, an environment to be treasured, and made me who I am today.

Never attribute to malevolence what is merely due to incompetence

–Arthur C. Clark

It was also a place that was out of time. It was a relic. The modern World came to Los Alamos and destroyed it, creating a shadow of its former greatness. The sort of values that made it such a National treasure and one of the greatest institutions could not coexist with today’s culture. The individuals so treasured and empowered by the scientific culture there were relabeled as “butthead cowboys,” troublemakers, and failures. The culture that was generous, long term in thought, viewing the big picture and focused on National service was haphazardly dismantled. Empowerment was ripped away from the scientists and replaced with control. Caution replaced boldness, management removed generosity, all in the name of formality of operations that removes anything unforeseen in outcomes. The modern world wants assured performance. Today Los Alamos is mere shadow of itself, stumbling forward toward the abyss of mediocrity. Witnessing this happen was one of the greatest tragedies of my life.

People who don’t take risks generally make about two big mistakes a year. People who do take risks generally make about two big mistakes a year.

–Peter F. Drucker

Along with assured performance we lose serendipity and discovery. We lose learning and surprises, good and bad. We lose the value in the individual, and the ability to have one person make a positive difference. All of this is to keep one person from making a negative difference or to avoid mistakes and failures. The removal of mistakes and failures removes the engine of learning and real scientific discovery from table as well. Each and every one of these steps is directly related to the fear of the bad things happening. Every good is a flip side of a bad thing, and when you can’t accept the bad, you can’t have the good either. In the process the individual has been removed from importance. Everything is process today and anything bad can be managed out of existence. No one looks at the downside to this, and the downside is sinister to the quality of the workplace.

Let’s be clear about what I’m talking about. This isn’t about being cavalier and careless. It isn’t an invitation to be dangerous or thoughtless. This is about making a best earnest effort at something, and still failing. This is about doing difficult things that may not succeed, putting your best effort forward even if it falls short. In many ways we have lost the ability to distinguish between the good and bad failure with all failure viewed as bad, and punished. We have made the workplace an obsessively cautious and risk adverse place that lacks the soul it once embraced. We have lost the wonder and power of the supremely talented person in the prime of their creative powers to create game changing things or knowledge.

The core problem is the willingness to deal with the inevitable risks and failures with empowering people. Instead of seeing the risks and failures and a necessary element in enabling success, we have fallen victim to the fiction that we can manage the risk and failure out of existence, all while assuring productivity. This is utterly foolish and antithetical to reality. The risks are necessary to strive to achieve difficult and potentially great things. If one is working at the limit of their capability the result is frequently failure, and the ensemble of failures paves the way for success. It tells us clearly what does not work, and provides the hard lessons that educate us. Somehow we have allowed the delusion that achievement can be had without risk and failure to creep into our collective consciousness.

Instead of encouraging and empowering our people to take risks while tolerating and learning from failure, we do the opposite. We steer people away from doing risky work, punish failure and discourage lesson learning. It is as if we had suddenly become believers in the “free lunch”. True achievement is extremely difficult, and true achievement is powered by the ability to try to do risky almost impossible things. If failure is not used as an opportunity to learn, people will become disempowered and avoid the risks. This in turn will kill achievement before it can even be thought of. The entire system would seem to be designed to disempower people, and lower their potential for achievement.

The other aspect of this truly viscous cycle is the dismantling of expertise. Expertise is built on the back of years and years of failure. Of course this happens only if the failures are actively engaged as educational opportunities that empower the expert to engage in more thoughtful risks. These thoughtfully engaged in risks still need to fail and perhaps fail most of the time. Gradually the failures of today begin to look like the achievements of yesterday. What we see as a failure today would be a monumental achievement a decade ago. This is completely built on the back of seeing the failures of yesterday in the right light, and learning the lessons available from the experience.

When we empower people to take risks and grow them into experts, they also provide the knowledge necessary to mentor others. This was a key aspect of my early career experience at Los Alamos. At that time the Lab was teeming with experts who were generous with their time and knowledge. All you had to do was reach out and ask, and people helped you. The experts were eager to share their experience and knowledge with others in a spirit of collective generosity. Today we are managed to completely avoid this with managed time and managed focus. We are trained to not be generous because that generosity would rob our “customers” of our effort and time. The flywheel of the experts of today helping to create the experts of tomorrow is being undone. People are trained to neither ask, nor provide expertise freely.

What we are moving toward is a system that is less than the sum of its parts. What I started with was a system that added great value to every person, and effectively was far greater than the sum of its parts. The generosity that characterized my early career added immense value to every hour spent at work. Today this entire way of working is being torn apart by how we are managed. People can’t be generous if they have to account for all their time and charge it to a specific customer. The room for serendipity, discovery and the addition of personal value to activities is being removed to satisfy bean counters and small-mined people. We have allowed an irrational fear of one misspent dollar to waste billions of dollars and the productive potential of people’s lives. Worse yet, the whole apparatus erected to produce formal operations are ripping the creative force from the workplace and replacing it with soulless conformity. It matters less and less who we are each day; we are simply replaceable parts in a mindless machine.

I might be temped to simply end the discussion here, but this conclusion is rather dismal. It is where we find ourselves today. We also know that the state of affairs can be significantly better. How can we get there from here? The first step would be some sort of collective decision that the current system isn’t working. From my perspective, the malaise and lack effectiveness of our current system is so pervasive and evident that action to correct it is overdue. On the other hand, the current system serves the purposes of those in control quite well, and they are not predisposed to be agents of change. As such, the impetus for change is almost invariably external. It is usually extremely painful because the status quo does not want to be rooted out unless it is forced to. The circumstances need to demand performance that current system cannot produce, and as systems degrade this becomes ever more likely.

At the time, my life just seemed too complete, and maybe we have to break everything to make something better out of ourselves.

–Chuck Palahniuk

The current system is thoroughly disempowering and oriented toward explicit control of people’s actions. Keeping order and people in line while avoiding risk and failure are the core principles. The key to any change is enabling trust for the individual to move to centrality in the system. The upside to the trust is the degree of efficiency and effectiveness that is born from trust; the downside is the possibility of failure, poor performance and various human failings. The system needs to be resilient to these inevitable problems with people. The negative impact of trying to control and manage these failings results in destroying most of the positive things individuals can provide. Empowerment needs to trump control and allow people’s natural inclination toward success to be central to organizational design.

In most cases being a good boss means hiring talented people and then getting out of their way.

–Tina Fey

We need to completely let go of the belief that we can manage all the bad things away and not lose all the good things in the process. Bad things, bad outcomes and bad behavior happen, and perhaps need to happen to have all the good (in other words “shit happens”). Today we are gripped with a belief that negative outcomes can be managed away. In the process of managing away bad outcomes, we destroy the foundation of everything good. To put it differently we need to value the good and accept the bad as a necessary condition for enabling good outcomes. If one looks at failure as the engine of learning, we begin to realize that the bad is the foundation of the good. If we do not allow the bad things to happen, let people fuck things up, we can’t have really good things either. One requires the other and our attempts to control bad outcomes, removes a lot of good or even great outcomes at the same time.

An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them.

– Werner Heisenberg

So to sum up, let’s trust people again. Let’s let them fail, fuck up and do bad things. Let’s let people learn from these failures, fuck up’s and painful experiences. These people will learn a lot, including very painful lessons and get hurt deeply in the process. They will become wise, strong, and truly experts at things. People who are entrusted are empowered and love their work. They are efficient, productive and effective. They have passion for what they do, and give their work great loyalty. They will take risks in a fearless manner. They will be allowed to fail spectacularly because spectacular success and breakthroughs only come from these fearlessly taken risks.

May I never be complete. May I never be content. May I never be perfect.

–Chuck Palahniuk

 

 

Sometimes the hardest thing any of us can hope for is finding the courage to be honest with ourselves.

― Kira Saito

Today I’m writing about dealing with the unfortunate practice of failing to address uncertainty, and implicitly setting its value to zero, the smallest possible value. This approach is pernicious, and ubiquitous in computational science (and a lot of other science). It is a direct threat to progress and far too acceptable in practice.  I wrote about this at length decrying this standard practice, but it remains the most common practice in uncertainty quantification (https://williamjrider.wordpress.com/2016/04/22/the-default-uncertainty-is-always-zero/). In a nutshell when someone doesn’t know what the uncertainty is they simply assign a value of zero to it. We can do something better, but first this needs to be recognized for what it is, systematic and accepted ignorance.

The reasons for not estimating uncertainties are legion. Sometimes it is just too hard (or people are lazy). Sometimes the way of examining a problem is constructed to ignore the uncertainty by construction (a common route to ignore experimental variability and numerical error). In other cases the uncertainty is large and it is far more comfortable to be delusional about its size. Smaller uncertainty is comforting and implies a level of mastery that exudes confidence. Large uncertainty is worrying and implies a lack of control. For this reason getting away with choosing a zero uncertainty is a source of false confidence and unfounded comfort, but a deeply common human trait.

If we can manage to overcome the multitude of human failings underpinning the choice of the default zero uncertainty, we are still left with the task of doing something better. To be clear, the major impediment is recognizing that the zero estimate of uncertainty is not acceptable (most “customers” like the zero estimate because it seems better even though its assuredly not!). Most of the time we have a complete absence of information to base uncertainty estimates upon. In some cases we can avoid zero uncertainty estimates by being more disciplined and industrious, in other cases we can think about the estimation from the beginning of the study and build the estimation into the work. In many cases we only have expert judgment to rely upon for estimation. In this case we need to employ a very simple and well-defined technique to providing an estimate.

Learning is not the accumulation of knowledge, but rather, one thing only: understanding

― Donna Jo Napoli

The best way to explore estimates is using the time-honored approach of bounding the uncertainty. One should be able to provide clear evidence that the uncertainty is both smaller and larger than certain known values. This provides bounds for the magnitude of uncertainty.  Depending on the purpose of the study, these magnitudes can be used to more appropriately use the results. This can then be used to provide some sort of reasonable and evidence based uncertainty to energize progress and underpin credibility. If the estimate of the smallest possible uncertainty is that ubiquitous zero, the estimate should be rejected out of hand. The uncertainty is never ZERO, not ever. Nothing is known with absolute certainty. If the uncertainty is very small there should be very strong evidence to support the bold assertion. We do know some things extremely well like Planck’s constant, but it still has an uncertainty of a finite size.

The flip side to the lower bound is the upper bound for the uncertainty. Generally speaking, there will be a worst case to consider or something more severe than the scenario at hand. Such large uncertainties are likely to be quite uncomfortable to those engaging in the work. This should be uncomfortable if we are doing things right. The goal of this exercise is not to minimize uncertainties, but get things right. If such bounding uncertainties are unavailable, one does not have the right to do high consequence decision-making with results. This is the unpleasant aspect of the process; this needs to be the delivery of the worst case. To be more concrete in the need for this part of the bounding exercise, if you don’t know how bad the uncertainty is you have no business using the results for anything serious. As stated before the bounding process needs to be evidence based, the assignment of lower and upper bounds for uncertainty should have a specific and defensible basis.

Belief can be manipulated. Only knowledge is dangerous.

― Frank Herbert

Once the bounds for the uncertainty are established along with associated evidence, some choices need to be made to use the information. To a large extent the most conservative choice is the easiest to defend meaning that the upper bound for uncertainty should be used. If the work is being engaged in an honest sense this would be a pessimistic perhaps in the extreme. If one thinks about things in a probabilistic sense, the bounds should establish an interval for the potential uncertainty. This interval is most likely to be defensibly treated with a uniform distribution. For most cases using a midpoint averaging the lower and upper bounds is a reasonable choice. If the application associated with the decision-making is extremely important, the upper bound or something skewed in that direction is probably advisable.

To some extent this is a rather easy lift intellectually. Cultural difficulty is another thing altogether. The indefensible optimism associated with the default zero uncertainty is extremely appealing.  It provides the user with a feeling that the results are good. People tend to feel that there is a single correct answer. The smaller the uncertainty is the better they feel about the answer. Large uncertainty is associated with lack of knowledge and associated with low achievement. The precision usually communicated with the default, standard approach is highly seductive. It takes a great deal of courage to take on the full depth of uncertainty along with the honest admission of how much is not known. It is far easier to simply do nothing and assert far greater knowledge while providing no evidence for the assertion.

Uncertainty is a discomforting concept for people. Certainty is easy and comfortable while uncertainty is difficult and carries doubt. It is problematic to consider the role of chance in events, and the fickle nature of reality. A great many important events occur largely by chance and could have quite easily turned out quite differently. Consider how often you encounter a near miss in life, something where danger seemed far to close and just missed you. When these events turn out disastrously they can be tragedies. How often have similar tragedies been barely averted? This same dynamic plays out in experiments that are repeated. An attempt is made to make the experiment reproducible. Occasionally something completely different unfolds. The repeated effects are never exactly the same; there is a small variation. These variations are the uncertainty and depending on the experiment, they have a magnitude.

What happens when you do the experiment exactly once? The simplest thing to do is consider this experiment to be a completely determined event with no uncertainty at all. This is the knee jerk response of people is the consideration of this single event as being utterly and completely deterministic with no variation at all. If the experiment were repeated with every attempt to make it as perfect as possible, it would turn out slightly differently. This comes from the myriad of details associated with the experiment that determine the outcome. Generally the more complex and energetic the phenomenon of being examined is, the greater the variation (unless there are powerful forces attracting a very specific solution). There is always a variation, the only question is how large it is; it is never, ever identically zero. The choice to view the experiment as perfectly repeatable is usually an unconscious choice that has no credible basis. It is an incorrect and unjustified assumption that is usually made without a second thought. As such the choice is unquestionably bad for science or engineering. In many cases this unconscious choice is dangerous, and represents nothing more than wishful thinking.

to hope was to expect

― Jane Austen

 

 

 

The purpose of life is not to be happy. It is to be useful, to be honorable, to be compassionate, to have it make some difference that you have lived and lived well.

― Ralph Waldo Emerson

As one gets older and enters into the heart of mid-life, it is natural to contemplate ones place in the World. I’m deep into such contemplation. I’ve been blessed with work with meaning for much of my adult life, but that meaning seems to have leaked away recently. Part of my thinking is the decision of whether this is a local or global condition. Are things worse where I am, or better than the average? For most of my adult life, I’ve had far better conditions than average, and been able to find great meaning in my work. Is the steady erosion of the quality of the work environment a consequence of issues local to my institution or organization? Or is it part of the massive systemic dysfunction our society is experiencing?

If the problem is local, I could leave for another organization, or another institution that is functioning better. If it’s a global issue, then its not something I can likely influence (much), and its time to ride the storm out the best I can. Right now my money is on the issue being global, and we ought to all be ready for the shit to hit the fan. My guess it is already happened, the shit storm is in effect and we are headed into deep trouble as a Nation and the World. We have repugnantly dysfunctional National government, led by an incompetent narcissistic conman without a perceptible moral compass. Racial tensions, and a variety of white supremacist/right wing ultra-Nationalists are walking the streets. Left wing and anarchist groups are waking up as well. Open warfare may soon be upon us making us long for the days where sporadic terrorist attacks were our biggest worry. A shit storm is actually a severe understatement; this is a fucking waking nightmare. I hope this is wrong and I could simply find a better place to work at and feel value in my labors. I wish the problem was simple and local with a simple job change fixing things.

Work is an important part of life for a variety of reasons. It is how we spend a substantial portion of our time, and much of our efforts go into it. In work we contribute to society and assist in the collective efforts of mankind. As I noted earlier, I’ve been fortunate for most of my life, but things have changed. Part of the issue is a relative change in the degree of self-determination in work. The degree of self-determination has decreased over time. An aspect of this is the natural growth in scope of work as a person matures. As a person grows in work and is promoted, the scope of the work increases, and the degree of freedom in work decreases. Again this is only a part of the problem as the system is working to strangle the self-determination out of people. This is control, fear of failure and generic lack of trust in people. In this environment work isn’t satisfying because the system is falling apart, and the easiest way to resist this is controlling the little guy. My work becomes more of a job and a route to a paycheck every day. Earning a living and supporting your family is a noble achievement these days, and aspiring to more simply a waking dream contracting in the rear view mirror of life.

Creative autonomy is essential for the work I do to be satisfying. It is essential to the work being effective. I can’t be an effective problem solver if most of my best options are off the table. We exist in a system where the solutions are dictated in advance. No one is trusted to solve any real problems, just work toward the solutions that have been pre-ordained. Autonomy is threatening to the system because the trust in people is so intrinsically low. The result is the leaking away of meaning in work. The control that exists only calms the deep fears of a system that is failing. Inside the model where we are teetering on the edge of a societal shit storm, the attempt to control makes sense. The system is desperately trying to hold onto whatever control it has, fearing the unraveling about to unfold. Fear makes us do stupid things, and the fear is simply throwing fuel onto the fire by making everyone simply hate life.

The purpose of life is a life of purpose.

― Robert Bryne

I’m trying to grapple with what is happening in my own experience through the lens of the bigger picture. We see a contraction of the trust and autonomy necessary for me to enjoy work. This is in direct reaction to the fears unleashed by the changes in society, and the terror these changes have induced in much of the population. The old world is coming to an end, but not without a fight. People are genuinely frightened by change, and for most people the most comfortable place is the past. They are holding onto the past with a fervent passion, but the future is unstoppable. In between the two is conflict and pain. For someone like myself who demands work that makes progress, I might have to take a break and simply resign myself to defending the progress that has already been made. No new progress can happen without trust in an environment dominated by fear. We are simply trying to maintain the progress that has already been won.

Imagine what our story would look like if, rather than succumbing to the insistent voices of family or culture, we determined that our vocation was to be a better human.

― James Hollis, Ph.D.

The domination of fear has an extremely large impact on the appetite for risk; there isn’t any. Part of the fearful environment is the inability to accept anything that looks, smells or even hints at failure. Without failure you don’t have learning or achievement. Research depends on failure because research is basically learning in its rawest form. Let me be clear that I’m talking about good failure where you try your best, making a best effort and coming up short. Most of the time a failure leads to learning something new. You tweak your approach or knowledge on the basis of the experience and grow. Without failure you short-circuit expertise. We need to energize failure in many small things to engage success in big things. All of this requires the sort of deep trust that our current World is almost devoid of. A combination of courage and trust can unleash people’s full potential through allowing them to fail spectacularly and then fully support the next step forward. Today, cowardice and mistrust dominate and even marginal failure results in punishment. It is corroding the foundation of achievement. It makes work simply a job and life more survival than living.

Since our current World is so deeply arrayed against personal success and growth, it might be wise to seek other avenues of fulfillment. Perhaps work is most healthily viewed as simply a task of mere survival. The current environment is so rife with fear, and patently incompetent that no one can really reach their potential. This isn’t a conclusion I like reaching, but the evidence seems overwhelming. Fear and mistrust have led to overarching control issues that remove any degree of personal control for achievement, or at least control while staying inside the rules. If one is willing to completely ignore the rules, success can be had. If one plays by the rules, success is absolutely impossible. The rules of the game are written to avoid all of the acts necessary for success because these involve risk and danger. Fundamental to success is trust, and trusting someone is beyond our collective ken.

The purpose of life is to contribute in some way to making things better.

― Robert F. Kennedy

 

Leadership is fundamentally about credibility.

― Rick Crossland

Under the best of circumstances we would like to confidently project credibility for the modeling and simulation we do. Under the worst of circumstances we would have confidence in modeling and simulation without credibility. This is common. Quite often the confidence is the product of arrogance or ignorance instead of humility and knowledge. This always manifests itself with a lack of questioning in the execution of work. Both of these issues are profoundly difficult to deal with and potentially fatal to meaningful impact of modeling and simulation. These issues are seen quite frequently. Environments with weak peer review contribute to allowing confidence with credibility to persist. The biggest part of the problem is a lack of pragmatic acceptance of modeling and simulation’s intrinsic limitations. Instead we have inflated promises and expectations delivered by over confidence and personality rather than hard nosed technical work.

When confidence and credibility are both in evidence, modeling and simulation is empowered to be impactful. It will be used appropriately with deference to what is and is not possible and known. When modeling and simulation is executed with excellence and professionalism along with hard-nosed assessment of uncertainties, using comprehensive verification and validation, the confidence is well grounded in evidence. If someone questions a simulations result, answers can be provided with well-vetted evidence. This produces confidence in the results because questions are engaged actively. In addition the limitations of the credibility are well established, and confidently be explained. Ultimately, credibility is a deeply evidence-based exercise. Properly executed and delivered, the degree of credibility depends on honest assessment and complete articulation of the basis and limits of the modeling.

When you distort the truth, you weaken your credibility.

― Frank Sonnenberg

One of the dangers of hard-nosed assessment is the tendency for those engaged in it to lose confidence in the work. Those who aggressively pursue credibility assessment tend to be cynics and doubters. They are prone to pessimism. They usually project doubt and focus on limitations of the modeling instead of confidence where it may be used. One of the hardest tricks of credibility assessment is pairing excellence in the execution of the work with an appropriate projection of confidence. The result is a mixed message where confidence is projected without credibility, and credibility is projected without confidence. Neither serves the purpose of progress in the impact of modeling and simulation.

One of the major sins of over-confidence is flawed or unexamined assumptions. This can be articulated as “unknown knowns” in the famously incomplete taxonomy forwarded by Donald Rumsfeld in his infamous quote. He didn’t state this part of the issue even though it was the fatal flaw in the logic of the Iraqi war in the aftermath of 9/11. There were basic assumptions about Hussein’s regime in Iraq that were utterly false, and these skewed the intelligence assessment leading to war. They only looked at information that supported the conclusions they had already drawn or wanted to be true. The same faulty assumptions are always present in modeling. Far too many simulation professionals ignore the foundational and unfounded assumptions in their work. In many cases assumptions are employed without thought or question. They are assumptions that the community has made for as long as anyone can remember and simply cannot be questioned. This can include anything from the equations solved, to the various modeling paradigms applied as a matter of course. Usually these are unquestioned and completely unexamined for validity in most credibility assessments.

This is an immensely tricky thing to execute. The standard assumptions are essential to managing complexity and making progress. That said, it is a remarkably difficult and important task to detect when the assumptions become limiting. More succinctly put, the limitations of the standard assumptions need to be thought-through and tested. Usually these assumptions can only be tested through removing everything else from the field and doing very hard work. It is so much easier to simply stay the course and make standard assumptions. In many cases the models have been significantly calibrated to match existing data, and new experiments or significantly more accurate measurements are needed to overturn or expose modeling limitations. Moreover the standard assumptions are usually unquestioned by peers. Questions are often met with ridicule. A deeply questioning assessment requires bravery and fortitude usually completely lacking from working scientists and utterly unsupported by our institutions.

Another manner for all of this to unfold is unwarranted confidence. Often this is couched in the form of arrogant perspectives where the proof of credibility is driven by personality. This proof by authority is incredibly common and troubling to dislodge. In many cases personal relationships to consumers of simulations are used to provide confidence. People are entrusted with the credibility and learn how to give their customer what they want. Credibility by personality is cheap and requires so much less work plus it doesn’t raise any pesky doubts. This circumstance creates an equilibrium that is often immune to scientific examination. It is easier to bullshit the consumers of modeling and simulation results than level with them about the true quality of the work.

The credibility of the teller is the ultimate test of the truth of a proposition.

― Neil Postman

More often than not honest and technically deep peer review is avoided like a plague. If it is imposed on those practicing this form of credibility, the defense of simulations takes the personal form of attacking the peer reviewers themselves. This sort of confidence is a cancer on quality and undermines any progress. It is a systematic threat to excellence in simulation, and must be controlled. It is dangerous because it is effective in providing support for modeling and simulation along with the appearance of real World impact.

One of the biggest threats to credibility is the generation of the lack of confidence honesty has. Engaging deeply and honestly in assessment of credibility is excellent at undermining confidence. Almost invariably the accumulation of evidence regarding credibility endows the recipients of this knowledge with doubt. These doubts are healthy and often the most confident people are utterly ignorant of the shortcomings. The accumulation of evidence regarding the credibility should have a benefit for the confidence in how simulation is used. This is a problem when those selling simulation oversell what it can do. The promise of simulation has been touted widely as transformative. The problem with modeling and simulation is its tangency to reality. The credibility of simulations is grounded by reality, but the uncertainty comes from both modeling, but also the measured and sensed uncertainty with our knowledge of reality.

The dynamic and tension with confidence and credibility should be deeply examined. When confidence is present without evidence, people should be deeply suspicious. A strong culture of (independent) peer review is an antidote to this. Too often these days the peer review is heavily polluted by implicit conflicts of interest. The honesty of peer review is hampered by an unwillingness to deal with problems particularly with respect to modification of the expectations. Invariably modeling and simulation has been oversold and any assessment will provide bad news. In today’s World we see a lot of bad news rejected, or repackaged (spun) to sound like good news. We are in the midst of a broader crisis of credibility with respect to information (i.e. fake news), so the issues with modeling and simulation shouldn’t be too surprising. We would all be well served by a different perspective and approach to this. The starting point is a re-centering of expectations, but so much money has been spent using grossly inflated claims.

Belief gives knowledge credibility.

― Steven Redhead

So what should we expect from modeling and simulation?

Modeling and simulation is a part of the scientific process and subject to its limits and rules. There is nothing magic about simulation that unleashes modeling from its normal limitations. The difference that simulation makes is the ability to remove the limitations of analytical model solution. Far more elaborate and accurate modeling decisions are available, but carry other difficulties due to the approximate nature of numerical solutions. The tug-of-war intellectually is the balance between modeling flexibility, nonlinearity and generality with effects of numerical solution. The bottom line is the necessity of assessing the uncertainties that arise from these realities. Nothing releases the modeling from its fundamental connection to validity grounded in real world observations. One of the key things to recognize is that models are limited and approximate in and of themselves. Models are wrong, and under a sufficiently resolved examination will be invalid. For this reason an infinitely powerful computer will ultimately be useless because the model will become invalid at some resolution. Ultimately progress in modeling and simulation is based on improving the model. This fact is ignored by computational science today and will result wasting valuable time, effort and money chasing quality that is impossible to achieve.

Bullshit is a greater enemy of the truth than lies are.

—Harry Frankfurt

In principle the issue of credibility and confidence in modeling and simulation should be based on evidence. Ideally this evidence should be quantitative with key indicators of its quality included. Ideally, the presence of the evidence should bolster credibility. Instead, paradoxically, evidence associated with the credibility of modeling and simulation seems to undermine credibility. This is a strong indicator that claims about the predictive power of modeling and simulation has been over-stated. This is a nice way of saying this is usually a sign that the quality is actually complete bullshit! We can move a long way toward better practice by simply recalibrating our expectations about what we can and can’t predict. We should be in a state where greater knowledge about the quality, errors and uncertainty in modeling and simulation work improves our confidence.

If you can’t dazzle them with brilliance, baffle them with bullshit!

– W.C. Fields

Part of the issue is the tendency for the consumers of modeling and simulation work to not demand evidence to support confidence. This evidence should always be present and available for scrutiny. If claims of predictive power are made without evidence, the default condition should be suspicion. The various sources of error and uncertainty should be drawn out, and quantified. There should be estimates based on concrete evidence for the value of uncertainty for all sources. Any uncertainty that is declared to be zero or negligible must have very specific evidence to support this assertion. Even more important any claims of this nature should receive focused and heavy scrutiny because they are likely to be based on wishful thinking, and often lack any evidentiary basis.

One of the issues of increasing gravity in this entire enterprise is the consumption of results using modeling and simulation by people unqualified to judge the quality of the work. The whole enterprise is judged to be extremely technical and complex. This inhibits those using the results from asking key questions regarding the quality of the work. With the people producing modeling and simulation results largely driven by money rather than technical excellence, we have the recipe for disaster. Increasingly, false confidence accompanies results and snows the naïve consumers into accepting the work. Often the consumers of computational results don’t know what questions to ask. We are left with quality being determined more by flashy graphics and claims about massive computer use than any evidence of prediction. This whole cycle perpetuates an attitude that starts to allow viewing reality as more of a video game and less like a valid scientific enterprise. Over inflated claims of capability are met with money to provide more flashy graphics and quality without evidence. We are left with a field that has vastly over-promised and provided the recipe for disaster.

We now live in a world where counter-intuitive bullshitting is valorized, where the pose of argument is more important than the actual pursuit of truth, where clever answers take precedence over profound questions.

― Ta-Nahisi Coates

Judge a man by his questions rather than by his answers.

― Voltaire

In thinking about what makes work good for me, I explored an element of the creative process for me revolving around answering questions. If one doesn’t have the right question, the work isn’t framed correctly and progress will stall. A thing to consider in this frame of reference is what makes a good question? This itself is an excellent question! The quality of the question makes a great difference in framing the whole scientific enterprise, and can either lead to bad places of “knowledge cul-de-sacs” or open stunning vistas of understanding. Where you end up depends on the quality of the question you answer. Success depends far more on asking the right question than answering the question originally put to you (or you put to yourself).

truth, like gold, is to be obtained not by its growth, but by washing away from it all that is not gold.

― Leo Tolstoy

A great question is an achievement in itself although rarely viewed as such. More often than not little of the process of work goes into asking the right question. Often the questions we ask are highly dependent upon foundational assumptions that are never questioned. While assumptions about existing knowledge are essential, finding the weak or invalid assumptions is often the key to progress. These assumptions are wonderful for simplifying work, but also inhibit progress. Challenging assumptions is one of the most valuable things to do. Heretical ideas are fundamental to progress; all orthodoxy began as heresy. If the existing assumptions hold up under the fire of intense scrutiny they gain greater credibility and value. If they fall, new horizons are opened up to active exploration.

If we have no heretics we must invent them, for heresy is essential to health and growth.

― Yevgeny Zamyatin

It goes without saying that important questions are good ones. Defining importance is tricky business. There are plenty of important questions that lead nowhere “what’s the meaning of life?” or we simply can’t answer using existing knowledge, “is faster than light travel possible?” On the other hand we might do well to break these questions down to something more manageable that might be attacked, “is the second law of thermodynamics responsible for life?” or “what do subatomic particles tell us about the speed of light?” Part of the key to good scientific progress is threading the proverbial needle of important, worthy and possible to answer. When we manage to ask an important, but manageable question, we serve progress well. Easy questions are not valuable, but are attractive due to their lack of risk and susceptibility to management and planning. Sometimes the hardest part of the process is asking the question, and a well-defined and chosen problem can be amenable to trivial resolution. It turns out to be an immensely difficult task with lots of hard work to get to that point.

I have benefited mightily from asking some really great questions in the past. These questions have led to the best research, and most satisfying professional work I’ve done. I would love to recapture this spirit of work again, with good questioning work feeling almost quaint in today’s highly over-managed climate. One simple question occurred in my study of efficient methods for solving the equations of incompressible flow. I was using a pressure projection scheme, which involves solving a Poisson equation at least once, if not more than once a time step. The most efficient way to do this involved using the multigrid method because of its algorithmic scaling being linear. The Poisson equation involves solving a large sparse system of linear equations, and the solution of linear equations scales with powers of the number of equations. Multigrid methods have the best scaling thought to be possible (I’d love to see this assumption challenged and sublinear methods discovered, I think they might well be possible).

As problems with incompressible flows become more challenging such as involving large density jumps, the multigrid method begins to become fragile. Sometimes the optimal scaling breaks down, or the method fails altogether. I encountered these problems, but found that other methods like conjugate gradient could still solve the problems. The issue is that the conjugate gradient method is less efficient in its scaling than multigrid. As a result as problems become larger, the proportion of the solution time spent solving linear equations grows ever larger (the same thing is happening now to multigrid because of the cost of communication on modern computers). I posed the question of whether I could get the best of both methods, the efficiency with the robustness? Others were working on the same class of problems, and all of us found the solution. Combine the two methods together, effectively using a multigrid method to precondition the conjugate gradient method. It worked like a charm; it was both simple and stunningly effective. This approach has become so standard now that people don’t even think about it, its just the status quo.

At this point it is useful to back up and discuss a key aspect of the question-making process essential to refining a question into something productive. My original question was much different, “how can I fix multigrid?” was the starting point. I was working from the premise that multigrid was optimal and fast for easier problems, and conjugate gradient was robust, but slower. A key part of the process was a reframing the question. The question I ended up attacking was “can I get the positive attributes of both algorithms?” This changed the entire approach to solving the problem. At first, I tried switching between the two methods depending on the nature of the linear problem. This was difficult to achieve because the issues with the linear system are not apparent under inspection.

The key was moving from considering the algorithms as different options whole cloth, to combining them. The solution involved putting one algorithm inside the other. As it turns out the most reasonable and powerful way to do this is make multigrid a preconditioner for conjugate gradient. The success of the method is fully dependent on the characteristics of both algorithms. When multigrid is effective by itself, the conjugate gradient method is effectively innocuous. When multigrid breaks down, the conjugate gradient method picks up the pieces, and delivers robustness along with the linear scaling of multigrid. A key aspect of the whole development is embracing an assault on a philosophical constraint in solving linear systems. At the outset of this work these two methods were viewed as competitors. One worked on one or the other, and the two communities do not collaborate, or even talk to each other. They don’t like each other. They have different meetings, or different sessions at the same meeting. Changing the question allows progress, and is predicated on changing assumptions. Ultimately, the results win and the former feud fades into memory. In the process I helped create something wonderful and useful plus learned a huge amount of numerical (and analytical) linear algebra.

The second great question I’ll point to involved the study of modeling turbulent flows with what has become known as implicit large eddy simulation. Starting in the early 1990’s there was a stunning proposition that certain numerical methods seem to automatically (auto-magically) model aspects of turbulent flows. While working at Los Alamos and learning all about a broad class of nonlinearly stable methods, the claim that they could model turbulence caught my eye (I digested it, but fled in terror from turbulence!). Fast forward a few years and combine this observation with a new found interest in modeling turbulence, and a question begins to form. In learning about turbulence I digested a huge amount of theory regarding the physics, and our approaches to modeling it. I found large eddy simulation to be extremely interesting although aspects of the modeling were distressing. The models that worked well were performed poorly on the structural details of turbulence, and the models that matched the structure of turbulence were generally unstable. Numerical methods for solving large eddy simulation were generally based on principles vastly different than those I worked on, which were useful for solving Los Alamos’ problems.

Having methods I worked on for codes that do solve our problems also model turbulence is tremendously attractive. The problem is the seemingly magical nature of this modeling. Being magical does not endow the modeling with confidence. The question that we constructed a research program around was “can we explain the magical capability of numerical methods with nonlinear stability to model turbulence?” We combined the observation that a broad class of methods seemed to provide effective turbulence modeling (or the universal inertial range physics). Basically the aspects of turbulence associated with the large-scale hyperbolic parts of the physics were captured. We found that it is useful to think of this as physics-capturing as an extension of shock-capturing. The explanation is technical, but astoundingly simple.

Upon study of the origins of large eddy simulation we discovered that the origins of the method were the same as shock capturing methods. Once the method was developed it evolved into its own subfield with its own distinct philosophy, and underlying assumptions. These assumptions had become limiting and predicated on a certain point-of-view. Shock capturing had also evolved in a different direction. Each field focused on different foundational principles and philosophy becoming significantly differentiated. For the most part they spoke different scientific languages. It was important to realize that their origins were identical with the first shock capturing method being precisely the first subgrid model for large eddy simulation. A big part of our research was bridging the divides that had developed over almost five decades and learn to translate from one language to the other.

We performed basic numerical analysis of nonlinearly stable schemes using a technique that produced the nonlinear truncation error. A nonlinear analysis is vital here. This uses a technique known as modified equation analysis. The core property of the methods empirically known to be successful in capturing the physics of turbulence is conservation (control volume schemes). It turns out that the nonlinear truncation error for a control volume method for a quadratic nonlinearity produces the fundamental scaling seen in turbulent flows (and shocks for that matter). This truncation error can be destabilizing for certain flow configurations, effectively being anti-dissipative. The nonlinear stability method keeps the anti-dissipative terms under control, producing physically relevant solutions (e.g., entropy-solutions).

A key observation makes this process more reasoned and connected to the traditional large eddy simulation community. The control volume term matches the large eddy simulation models that produce good structural simulations of turbulence (the so-called scale similarity model). The scale similarity model is unstable with classical numerical methods. Nonlinear stability fixes this problem with aplomb. We use as much scale similarity as possible without producing unphysical or unstable results. This helps explain why a disparate set of principles used to produce nonlinear stability provides effective turbulence modeling. Our analysis also shows why some methods are ineffective for turbulence modeling. If the dissipative stabilizing effects are too large and competitive with the scale similarity term, the nonlinear stability is ineffective as a turbulence model.

It is dangerous to be right in matters on which the established authorities are wrong.

― Voltaire

I should spend some time on some bad questions as examples of what shouldn’t be pursued. One prime example is offered as a seemingly wonderful question, the existence of solutions to the incompressible Navier-Stokes equations. The impetus for this question is the bigger question of can we explain, predict or understand fluid turbulence? This problem is touted as a fundamental building block in this noble endeavor. The problem is the almost axiomatic belief that turbulence is contained within this model. The key term is incompressible, which renders the equations unphysical on several key accounts: it gives the system infinite speed of propagation, and divorces the equations from thermodynamics. Both features sever the ties of the equations from the physical universe. The arguing point is whether these two aspects disqualify it from addressing turbulence. I believe the answer is yes.

In my opinion this question should have been rejected long ago based on the available evidence. Given that our turbulence theory is predicated on the existence of singularities in ideal flows, and the clear absence of such singularities in the incompressible Navier-Stokes equations, we should reject the notion that turbulence is contained in them. Despite this evidence, the notion that turbulence is contained whole cloth in these unphysical equations remains unabated. It is treated as axiomatic. This is an example of an assumption that has out lived its usefulness. It will eventually be tossed out, and progress will bloom the path of its departure. One of the key things missing from turbulence is a connection to thermodynamics. Thermodynamics is such a powerful scientific concept and for it to be so absent from turbulence is a huge gap. Turbulence is a fundamental dissipative process and the second law is grounded on dissipation. The two should be joined into a coherent whole allowing unity and understanding to reign where confusion is supreme today.

Another poorly crafted question revolves around the current efforts for exascale class computers for scientific computing. There is little doubt that an exascale computer would be useful for scientific computing. A better question is what is the most beneficial way to push scientific computing forward? How can we make scientific computing more impactful in the real world? Can the revolution of mobile computing be brought to science? How can we make computing (really modeling and simulation) more effective in impacting scientific progress? Our current direction is an example of crafting an obvious question, with an obvious answer, but failing to ask a more cutting and discerning question. The consequence of our unquestioning approach to science will be wasted money and stunted progress.

Trust is equal parts character and competence… You can look at any leadership failure, and it’s always a failure of one or the other.

― Stephen M.R. Covey

This gets at a core issue with how science is managed today. Science has never been managed more tightly and becoming more structurally mismanaged. The tight management of science as exemplified by the exascale computing efforts is driven by an overwhelming lack of trust in those doing science. Rather than ask people open-ended questions subject to refinement through learning, we ask scientists to work on narrowly defined programs with preconceived outcomes. The reality is that any breakthrough, or progress for that matter will take a form not envisioned at the outset of the work. Any work that pushes mankind forward will take a form not foreseeable. By managing so tightly and constraining work, we are predestining the outcomes to be stunted and generally unworthy of the effort put into them.

Whether you’re on a sports team, in an office or a member of a family, if you can’t trust one another there’s going to be trouble.

― Stephen M.R. Covey

This is seeded by an overwhelming lack of trust in people and science. Trust is a powerful concept and its departure from science has been disruptive and expensive. Today’s scientists are every bit as talented and capable as those of past generations, but society has withdrawn its faith in science. Science was once seen as a noble endeavor that embodied the best in humanity, but generally not so today. Progress in the state of human knowledge produced vast benefits for everyone and created the foundation for a better future. There was a sense of an endless frontier constantly pushing out and providing wonder and potential for everyone. This view was a bit naïve and overlooked the maxim that human endeavors in science are neither good or bad, producing outcomes dependent upon the manner of their use. For a variety of reasons, some embedded within the scientific community, the view of society changed and the empowering trust was withdrawn. It has been replaced with suspicion and stultifying oversight.

When I take a look at the emphasis in currently funded work, we see narrow vistas. There is a generally myopic and tactical view of everything. Long-term prospects, career development and broad objectives are obscured by management discipline and formality. Any sense of investment in the long-term is counter to the current climate. Nothing speaks more greatly to the overwhelming myopia is the attitude toward learning and personal development. It is only upon realizing that learning and research are essentially the same thing does it start to become clear how deeply we are hurting the scientific community. We have embraced a culture that is largely unquestioning with a well-scripted orthodoxy. Questions are seen as heresy against the established powers and punished. For most, learning is the acquisition of existing knowledge and skills. Research is learning new knowledge and skills. Generally speaking, those who have achieved mastery of their fields execute research. Since learning and deep career development is so hamstrung by our lack of trust, fewer people actually achieve the sort of mastery needed for research. The consequences for society are profound because we can expect progress to be thwarted.

Curiosity is more important than knowledge.

― Albert Einstein

One clear way to energize learning, and research is encouraging questioning. After encouraging a questioning attitude and approach to conducting work, we need to teach people to ask good questions, going back and refining questions, as better understanding is available. We need to identify and overcome assumptions subjecting them to unyielding scrutiny. The learning, research and development environment is equivalent to a questioning environment. By creating an unquestioning environment we short-circuit everything leading to progress, and ultimately cause much of the creative engine of humanity to stall. We would be well served by embracing the fundamental character of humanity as a creative, progressive and questioning species. These characteristics are parts of the best that people have to offer and allow each of us to contribute to the arc of history productively.

Curiosity is the engine of achievement.

― Ken Robinson

Brandt, Achi. “Multi-level adaptive solutions to boundary-value problems.” Mathematics of computation 31, no. 138 (1977): 333-390.

Briggs, William L., Van Emden Henson, and Steve F. McCormick. A multigrid tutorial. Society for Industrial and Applied Mathematics, 2000.

Kershaw, David S. “The incomplete Cholesky—conjugate gradient method for the iterative solution of systems of linear equations.” Journal of Computational Physics 26, no. 1 (1978): 43-65.

Melson, N. Duane, T. A. Manteuffel, and S. F. Mccormick. “The Sixth Copper Mountain Conference on Multigrid Methods, part 1.” (1993).

Puckett, Elbridge Gerry, Ann S. Almgren, John B. Bell, Daniel L. Marcus, and William J. Rider. “A high-order projection method for tracking fluid interfaces in variable density incompressible flows.” Journal of Computational Physics130, no. 2 (1997): 269-282.

Boris, J. P., F. F. Grinstein, E. S. Oran, and R. L. Kolbe. “New insights into large eddy simulation.” Fluid dynamics research 10, no. 4-6 (1992): 199-228.

Porter, David H., Paul R. Woodward, and Annick Pouquet. “Inertial range structures in decaying compressible turbulent flows.” Physics of Fluids 10, no. 1 (1998): 237-245.

Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821-841.

Grinstein, Fernando F., Len G. Margolin, and William J. Rider, eds. Implicit large eddy simulation: computing turbulent fluid dynamics. Cambridge university press, 2007.

Fefferman, Charles L. “Existence and smoothness of the Navier-Stokes equation.” The millennium prize problems (2006): 57-67.

We don’t receive wisdom we must discover it for ourselves.

― Marcel Proust

Work is best when you start with a good question, analyze and learn until you discover and understand an answer to the question (questions often have many answers). Then you use this understanding to create something wonderful so that you can find a new and better question to answer. This virtuous cycle leads to be best work and provides the foundation for excellence. It is precisely the recipe for the best work experiences I’ve had, built my expertise and definitely how I’d prefer to keep doing work.

I’m on vacation this week (San Francisco is an amazing city!) and it is the perfect opportunity to think deeply about life and work. Work is an extremely important part of life, and I’ve concluded that some key things determine whether or not it is really good. The same things determine your ability to achieve excellence. What I’ve observed is a process that takes place leading up to my happiness and satisfaction. More importantly, it leads to great work, productivity and excellence. The elements of this successful recipe are founded on attacking a question that needs to be answered. This question can either come from something larger than myself, or simple innate personal curiosity. At the end of the process the question has been refined and answered yielding new understanding, knowledge, learning and tools to create something better. For me, the act of creation is the ultimate in job satisfaction for me. This is a virtuous cycle that leads to deep knowledge and the ability to recycle this process with an even better question using what has been learned and created.

Our real discoveries come from chaos, from going to the place that looks wrong and stupid and foolish.

― Chuck Palahniuk

The largest portion and most important part of this process is the analysis that allows us to answer the question. Often the question needs to be broken down into a series of simpler questions some of which are amenable to easier solution. This process is hierarchical and cyclical. Sometimes the process forces us to step back and requires us to ask an even better or more proper question. In sense this is the process working in full with the better and more proper question being an act of creation and understanding. The analysis requires deep work and often study, research and educating oneself. A new question will force one to take the knowledge one has and combine it with new techniques producing enhanced capabilities. This process is on the job education, and fuels personal growth and personal growth fuels excellence. When you are answering a completely new question, you are doing research and helping to push the frontiers of science forward. When you are answering an old question, you are learning and you might answer the question in a new way yielding new understanding. At worst, you are growing as a person and professional.

This is an utterly noble endeavor and embodies the best of mankind. At times you are simply pushing your self forward into areas others know very well already, but to you it is fresh and new. This is OK and even essential to get to the place where your work is unique. An under appreciated aspect of this sort of learning is the path you take is the potential to learn things in new ways. Your path is likely to be different than anyone else’s and grafts your own experience and understanding on to the topic anew. This is immensely valuable and can unveil new paths and depth to existing knowledge. Today this sort of thing is wholly unsupported and under appreciated. We need to make a new commitment to use this path to excellence.

The real voyage of discovery consists not in seeking new landscapes, but in having new eyes.

― Marcel Proust

Sometimes the question being answered has been well studied and one is simply discovering knowledge others have already mastered. This is important growth for a professional getting to the point where the frontier of knowledge exists. This is a necessary element in getting to research, which doesn’t happen automatically. One needs to climb up the mountain of human knowledge before getting to the apex. This is the process of education as a professional and an immensely exciting calling. The mastery of a topic requires many essential elements be mastered drawing together knowledge from diverse forces. Often the best research draws together rather pedestrian bits of knowledge from diverse fields in novel manners heretofore unseen before. When we don’t support this sort of endeavor, we smother important avenues of discovery and deny our society of the most important discoveries. Charting new paths to knowledge is either a wondrous personal journey and/or an alternative way to understand.

Discovery consists of looking at the same thing as everyone else and thinking something different.

― Albert Szent-Györgyi

Ultimately the elements are drawn together and allow the question to be answered productively. This often produces a new kernel of understanding. This knowledge can often be harnessed to produce the wherewithal for something new. The understanding will allow a new and unique act of creation. Sometimes you are creating something that others already know about, but for you it is new. That is enough for excellence; it is the engine of personal excellence. If you complete this cycle often enough eventually the creation will be genuinely original and new. The deep and powerful educational elements of this process leads to outstanding professionals well before one gets to genuinely new and unique research. It is essential to realize that very few creations are completely original with most discoveries being the combination of elements that are well known in other applications. In many cases the analysis and study of the answer to the original question itself creates something new and wonderful of many forms.

What is wanted is not the will to believe, but the will to find out, which is the exact opposite.

― Bertrand Russell

Once this creation is available, new questions can be posed and solved. These creations allow new questions to be asked answered. This is the way of progress where technology and knowledge builds the bridge something better. If we support excellence and a process like this, we will progress. Without support for this process, we simply stagnate and whither away. The choice is simple either embrace excellence by loosening control, or chain people to mediocrity.

Science is the process that takes us from confusion to understanding…

― Brian Greene

A very great deal more truth can become known than can be proven.

― Richard Feynman

In modeling and simulation verification is a set of activities broadly supporting the quality. Verification consists of two modes of practice: code verification where the mathematical correctness of the computer code is assessed, or solution (calculation) verification where the numerical error (uncertainty) is estimated. Both activities are closely linked to each other and they are utterly complementary in nature. To a large extent the methodology used for both types of verification are similar, but the differences between the two are important to maintain.

Modeling and simulation is an activity where continuous mathematics is converted to discrete computable quantities. This process involves approximation of the continuous mathematics and in almost every non-pathological circumstance is inexact. The core of modeling and simulation is the solution of (partial) differential equations using approximation methods. Code verification is a means of assuring that the approximations used to make the discrete solution of differential equations tractable on a computer are correct. A key aspect of code verification is determining that the discrete approximation of the differential equation is consistent with the continuous version of the differential equation.

Consistency demands that the order of approximation of the differential equation be at least one. In other words the discrete equations produce solutions that are the original continuous equations plus terms that are proportional to the size of the discretization. This character may be examined by solving problems with an exact analytical solution (or a problem with very well controlled and characterized errors) using several discretization sizes allowing the computation of errors, and determining the order of approximation. The combination of consistency and stability of the approximation means the approximation converges to the correct solution of the continuous differential equation.

We will examine both the nature of different types of problems to determine code verification and the methods of determining the order of approximation. One of the key aspects of code verification is the congruence of the theoretical order of accuracy for a method, and the observed order of accuracy. It is important to note that the theoretical order of convergence also depends upon the problem being solved. The problem must possess enough regularity to support the convergence rate expected. At this point it is important to point out that code verification produces both an order of approximation and an observed error in solution. Both of these quantities are important. For code verification, the order of approximation is the primary quantity of interest. It depends on both the nature of the approximation method and the problem being solved. If the problem being solved is insufficiently regular and smooth, the order of accuracy will not match the theoretical expectations of the method.

The second form of verification is solution verification. This is quite similar to code verification, but its aim is the estimation of approximation errors in a calculation. When one runs a problem without an analytical solution, the estimation of errors is more intricate. One looks at a series of solutions and compute the solution that is indicated by the sequence. Essentially the question of what solution is the approximation appearing to converge toward is being asked. If the sequence of solutions converges, the error in the solution can be inferred. As with code verification the order of convergence and the error is a product of the analysis. Conversely to the code verification, the error estimate is the primary quantity of interest, and the order of convergence is secondary.

 

 

The approach, procedure and methodology for both forms of verification are utterly complementary. Much of the mathematics and flow of work are shared in all verification, but details, pitfalls and key tips differ. In this post the broader themes of commonality are examined along with distinctions and a general rubric for each type of verification is discussed.

Code verification

Science replaces private prejudice with public, verifiable evidence.

― Richard Dawkins

When one conducts a code verification study there is a basic flow of activities and practices to conduct. One looks at a code to target and a problem to solve. Several key bits of information should be immediately being focused upon before the problem is solved. What is the order of accuracy for the method in the code being examined, and what is the order of accuracy that the problem being solved can expose? In addition the nature of the analytical solution to the problem should be carefully considered. For example what is the nature of the solution? Closed form? Series expansion? Numerical evaluation? Some of these forms of analytical solution have errors that must be controlled and assessed before the code’s method may be assessed. By the same token are there auxiliary aspects of the code’s solution that might pollute results? Solution of linear systems of equations? Stability issues? Computer roundoff or parallel computing issues? In each case these details could pollute results if not carefully excluded from consideration.

Next one needs to produce a solution on a sequence of meshes. For simple verification using a single discretization parameter only two discretizations are needed for verification (two equations to solve for two unknowns). For code verification the model for error is simple, generally a power law, where the error is proportional to the discretization parameter to the power (order) . There is also a constant of proportionality. The order, is the target of the study and one looks at its congruence with the expected theoretical order for the method on the problem being solved. It is almost always advisable to use more than the minimum number of meshes to assure that one simply isn’t examining anamolous behavior from the code.

One of the problems with code verification is the rarity of the observed order of convergence to exactly match the expected order of convergence. The question of how close is close enough haunts investigations. Invariably the observed order will deviate from the expected order by some amount. The question for the practitioner is how close is acceptable? Generally this question is given little attention. There are more advanced verification techniques that can put this issue to rest by producing uncertainties on the observed order, but the standard techniques simply produce a single result. Usually this results in rules of thumb that apply in broad brushes, but undermine the credibility of the whole enterprise. Often the criterion is that the observed order should be within a tenth of the theoretically expected result.

Another key caveat comes up when the problem is discontinuous. In this case the observed order is either set to one for nonlinear solutions, or weakly tied to the theoretical order of convergence. For the wave equation this result was studied by Banks, Aslam and Rider and admits an analytical and firmly determined result. In both cases the issue of inexact congruence with the expected rate of convergence remains. In addition for problems involving systems of equations will have multiple features each having a separate order of convergence, and the rates will combine within a solution. Ultimately in an asymptotic sense the lowest order of convergence will dominate as . This is quite difficult to achieve practically.

The last major issue that comes up in code verification (and solution verification too) is the nature of the discrete mesh and its connection to the asymptotic range of convergence. All of the theoretical results apply when the discretization parameter is small in a broad mathematical sense. This is quite problem specific and generally ill defined. Examining the congruence of the numerical derivatives of the analytical solution with the analytical derivatives can generally assess this. When these quantities are in close agreement, the solution can be considered to be asymptotic. Again these definitions are loose and generally applied with a large degree of professional or expert judgment.

It is useful to examine these issues through a concrete problem in code verification. The example I’ll use is a simple ordinary differential equation integrator for a linear equation coded up in Mathematica. We could solve this problem in a spreadsheet (like MS Excel), python, or a standard programming language. The example will look at two first order methods, forwards and backwards Euler methods. Both of these methods produce leading first order errors in an asymptotic sense, . If is large enough, the high order terms will pollute the error and produce deviations from the pure first-order error. Let’s look at this example and the concrete analysis from verification. This will be instructive in getting to similar problems encountered in general code verification.

Here is the code

ForwardEuler[h_, T_, a_] :=

(

uo = 1;

t = 0.0;

While[t < T,

(* integration *)

t = t + h;

un = uo + a h uo;

Print[“t= “, t, ” u(t) = “, un, ” err = “, Abs[un – Exp[a t]]];

uo = un

];

)

 

BackwardEuler[h_, T_, a_] :=

(

uo = 1;

t = 0.0;

While[t < T,

(* integration *)

t = t + h;

un = uo/(1 + a h);

Print[“t= “, t, ” u(t) = “, un, ” err = “, Abs[un – Exp[a t]]];

uo = un

];

)

Let’s look at the forward Euler integrator for several different choices of , different end times for the solution and number of discrete solutions using the method. We will do the same thing for the backwards Euler method, which is different because it is unconditionally stable with respect to step size. For this simple ODE, the method is stable to a stepsize of and we can solve the problem to two stopping times of , and . The analytical solution is always, . We can solve this problem using a set of step sizes, .

I can give results for various pairs of step sizes with both integrators, and see some common pathologies that we must deal with. Even solving such a simple problem, with simple methods can prove difficult and prone to heavy interpretation (arguably the simplest problem with the simplest methods). Much different results are achieved when the problem is run until different stopping times. We see the impactof accumulated error (since I’m using Mathematica so aspects of round-off error are pushed aside). In these cases round-off error would be another complication. Furthermore the backward Euler method for multiple equations would involve a linear (or nonlinear) solution that itself has an error tolerance that may significantly impact verification results. We see good results for and a systematic deviation for longer ending times. To get acceptable verification results would require much smaller step sizes (for longer calculations!). This shows how easy it is to scratch the surface of really complex behavior in verification that might mask correctly implemented methods. What isn’t so well appreciated is that this behavior is expected and amenable to analysis through standard methods extended to look for it.

h	FE T=1	FE T=10	FE T=100	BE T=1	BE T=10	BE T=100
1	1.64	0.03	~0	0.79	1.87	16.99
0.5	1.20	0.33	4e-07	0.88	1.54	11.78
0.25	1.08	0.65	0.002	0.93	1.30	7.17
0.125	1.04	0.83	0.05	0.96	1.16	4.07
0.0625	1.02	0.92	0.27	0.98	1.08	2.40
0.03125	1.01	0.96	0.55	0.99	1.04	1.63

Computed order of convergence for forward Euler (FE) and backward Euler (BE) methods for various stopping times and step sizes.

Types of Code Verification Problems and Associated Data

 Don’t give people what they want, give them what they need.

― Joss Whedon

The problem types are categorized by the difficulty of providing a solution coupled withthe quality of the solution that can be obtained.   These two concepts go hand-in-hand. As simple closed form solution is easy to obtain and evaluation. Conversely, a numerical solution of partial differential equations is difficult and carries a number of serious issues regarding its quality and trustworthiness. These issues are addressed by an increased level of scrutiny on evidence provided by associated data. Each of benchmark is not necessarily analytical in nature, and the solutions are each constructed in different means with different expected levels of quality and accompanying data. This necessitates the differences in level of required documentation and accompanying supporting material to assure the user of its quality.

Next, we provide a list of types of benchmarks along with an archetypical example of each. This is intended to be instructive to the experienced reader, who may recognize the example. The list is roughly ordered in increasing level of difficulty and need for greater supporting material.

• Closed form analytical solution (usually algebraic in nature). Example: Incompressible, unsteady, 2-D, laminar flow over an oscillating plate (Stokes oscillating plate) given in Panton, R. L. (1984). Incompressible Flow, New York, John Wiley, pp. 266-272.
• Analytical solution with significantly complex numerical evaluation
  • Series solution. Example: Numerous classical problems, in H. Lamb’s book, “Hydrodynamics,” Dover, 1932. Classical separation of variables solution to heat conduction. Example: Incompressible, unsteady, axisymmetric 2-D, laminar flow in a circular tube impulsively started (Szymanski flow), given in White, F. M. (1991). Viscous Fluid Flow, New York, McGraw Hill, pp. 133-134.
  • Nonlinear algebraic solution. Example: The Riemann shock tube problem, J. Gottleib, C. Groth, “Assessment of Riemann solvers for unsteady one-dimensional inviscid flows of perfect gases,” Journal of Computational Physics, 78(2), pp. 437-458, 1988.
  • A similarity solution requiring a numerical solution of nonlinear ordinary differential equations.
  • Manufactured Solution. Example: Incompressible, steady, 2-D, turbulent, wall-bounded flow with two turbulence models (makes no difference to me), given in Eça, L., M. Hoekstra, A. Hay and D. Pelletier (2007). “On the construction of manufactured solutions for one and two-equation eddy-viscosity models.” International Journal for Numerical Methods in Fluids. 54(2), 119-154.
• Highly accurate numerical solution (not analytical). Example: Incompressible, steady, 2-D, laminar stagnation flow on a flat plate (Hiemenz flow), given in White, F. M. (1991). Viscous Fluid Flow, New York, McGraw Hill. pp. 152-157.
• Numerical benchmark with an accurate numerical solution. Example: Incompressible, steady, 2-D, laminar flow in a driven cavity (with the singularities removed), given in Prabhakar, V. and J. N. Reddy (2006). “Spectral/hp Penalty Least-Squares Finite Element Formulation for the Steady Incompressible Navier-Stokes Equations.” Journal of Computational Physics. 215(1), 274-297.
• Code-to-code comparison data. Example: Incompressible, steady, 2-D, laminar flow over a back-step, given in Gartling, D. K. (1990). “A Test Problem for Outflow Boundary Conditions-Flow Over a Backward-Facing Step.” International Journal for Numerical Methods in Fluids. 11, 953-967.

Below is a list of the different types of data associated with verification problems defined above. Depending on the nature of the test problem only a subset of these data are necessary. This will be provided below the list of data types. As noted above, benchmarks with well-defined closed form analytical solutions require relatively less data than a benchmark associated with the approximate numerical solution of PDEs.

• Detailed technical description of the problem (report or paper)
• Analysis of the mathematics of the problem (report or paper)
• Computer analysis of solution (input file)
• Computer solution of the mathematical solution
• Computer implementation of the numerical solution
• Error analysis of the “exact” numerical solution
• Derivation of the source term and software implementation or input
• Computer implementation of the source term (manufactured solution)
• Grids for numerical solution
• Convergence and error estimation of approximate numerical solution
• Uncertainty and sensitivity study of numerical solution
• Description and analysis of computational methods
• Numerical analysis theory associated with convergence
• Code description/manuals
• Input files for problems and auxiliary software
• Patch test description, Derivation, input and analysis
• Unusual boundary conditions (inflow, piston, etc.…)
• Physics restrictions (boundary layer theory, inviscid,)
• Software quality documents
• Scripts and auxiliary software for verification
• Source code
• Metric descriptions
• Verification results including code version, date, etc.
• Numerical sensitivity studies
• Feature coverage in verification

Below, we briefly describe the characteristics of each type of benchmark documentation (could be called artifacts or meta-data) associated with a code verification benchmarks. These artifacts take a number of concrete forms such as a written document, computer code, mathematical solution in document or software form, input files for executable codes, input to automatic computer analysis, output from software quality systems, among others.

• Detailed technical description of the benchmark (report or paper): This can include a technical paper in a journal or conference proceeding describing the benchmark and its solution. Another form would be a report informal or formal from an institution providing the same information.
• Analysis of the mathematics (report or paper): For any solution that is closed form, or requiring a semi-analytical solution, the mathematics must be described in detail. This can be included in the paper (report) discussed previously or in a separate document.
• Computer analysis of solution (input file): If the mathematics or solution is accomplished using a computerized analysis, the program used and the input to the program should be included. Some sort of written documentation such as a manual for the software ideally accompanies this artifact.
• Computer solution of the mathematical solution: The actual computerized solution of the mathematical problem should be included in whatever form the computerized solution takes. This should include any error analysis completed with this solution.
• Computer implementation of the numerical solution: The analytical solution should be implemented in a computational form to allow the comparison with the numerical solution. This should include some sort of error analysis in the form of a report.
• Derivation of the source term and software implementation or input: In the case of the method of manufactured solutions, the source term used to drive the numerical method must be derived through a well-defined numerical procedure. This should be documented through a document, and numerical tools used for the derivation and implementation.
• Computer implementation of the source term (manufactured solution): The source term should be included in a form amenable to direct use in a computer code. The language for the computer code should be clearly defined as well as the compiler and computer system used.
• Grids for numerical solution: If a solution is computed using another simulation code all relevant details on the numerical grid(s) used must be included. This could be direct grid files, or input files to well-defined grid generation software.
• Convergence and error estimation of numerical solution: The numerical solution must include a convergence study and error estimate. These should be detailed in an appropriately peer-reviewed document.
• Uncertainty and sensitivity study of numerical solution: The various modeling options in the code used to provide the numerical solution must be examined vis-a-vis the uncertainty and sensitivity of the solution to these choices. This study should be used to justify the methodology used for the baseline solution.
• Description and analysis of computational methods: The methods used by the code used for the baseline solution must be completely described and analyzed. This can take the form of a complete bibliography of readily available literature
• Numerical analysis theory associated with convergence: The nature of the convergence and the magnitude of error in the numerical solution must be described and demonstrated. This can take the form of a complete bibliography of readily available literature.
• Code description/manuals: The code manual and complete description must be included with the analysis and description.
• Input files for benchmarks and auxiliary software: The input file used to produce the solution must be included. Any auxiliary software used to produce or analyze the solution must be full described or included.
• Unusual boundary conditions (inflow, piston, outflow, Robin, symmetry, …): Should the benchmark require unusual or involved boundary or initial conditions, these must be described in additional detail including the nature of implementation.
• Physics restrictions (boundary layer theory, inviscid, parabolized Navier-Stokes, …): If the solution requires the solution of a reduced or restricted set of equations, this must be fully described. Examples are boundary layer theory, truly inviscid flow, or various asymptotic limits.
• Software quality documents: Of non-commercial software used to produce solutions, the software quality pedigree should be clearly established by documenting the software quality and steps taken to assure the maintenance of the quality.
• Scripts and auxiliary software for verification: Auxiliary software or scripts used to determine the verification or compute error estimates for a software used to produce solution should be included.
• Source code: If possible the actual source code for the software along with instructions for producing an executable (makefile, scripts) should be included with all other documentation.
• A full mathematical or computational description of metrics used in error analysis and evaluation of solution implementation or numerical solution.
• Verification results including code version, date, and other identifying characteristics: The verification basis for the code used to produce the baseline solution must be included. This includes any documentation of verification, peer-review, code version, date completed and error estimates.
• Feature coverage in verification: The code features covered by verification benchmarks must be documented. Any gaps where the feature used for the baseline solution are not verified must be explicitly documented.

Below are the necessary data requirements for each category of benchmark, again arranged in order of increasing level of documentation required. For completeness each data type would expected to be available to describe a benchmark of a given type.

• Common elements for all types of benchmarks (it is notable that the use of proper verification using an analytical solution results in the most compact set of requirements for data, manufactured solutions also).

1. Paper or report
2. Mathematical analysis
3. Computerized solution and input
4. Error and uncertainty analysis
5. Computer implementation of the evaluation of the solution
6. Restrictions
7. Boundary or initial conditions

• Closed form analytical solution

1. Paper or report
2. Mathematical analysis
3. Computerized solution and input
4. Error and uncertainty analysis
5. Computer implementation of the evaluation of the solution
6. Restrictions
7. Boundary or initial conditions

• Manufactured Solution

1. Paper or report
2. Mathematical analysis
3. Computational solution and input
4. Error and uncertainty analysis
5. Computer implementation of the evaluation of the solution
6. Derivation and implementation of the source term
7. Restrictions
8. Boundary or initial conditions

• Numerical solution with analytical solution
• Series solution, Nonlinear algebraic solution, Nonlinear ODE solution

1. Paper or report
2. Mathematical analysis
3. Computerized solution and input
4. Error and uncertainty analysis
5. Computer implementation of the evaluation of the solution
6. Input files
7. Source code
8. Source code SQA
9. Method description and manual
10. Restrictions
11. Boundary or initial conditions

• Highly accurate numerical solution (not analytical), numerical benchmarks or code-to-code comparisons.

1. Paper or report
2. Mathematical analysis
3. Computational solution and input
4. Error and uncertainty analysis for the solution
5. Computer implementation of the evaluation of the solution
6. Input files
7. Grids
8. Source code
9. Source code SQA
10. Method description and manual
11. Method analysis
12. Method verification analysis and coverage
13. Restrictions
14. Boundary or initial conditions

The use of direct numerical simulation requires a similar or even higher level of documentation than analytical solutions. This coincides with the discussion of the last type of verification benchmark where a complex numerical method with significant approximations is utilized to produce the solution. As a numerically computed benchmark, the burden of proof is much larger.   Code verification is best served by exact analytical solutions because of the relative ease in assuring benchmark solution accuracy. Nonetheless, it remains a common practice due to its inherent simplicity. It also appeals to those who have a vested interest in the solutions produced by a certain computer code. The credibility of the comparison is predicated on the credibility of the code producing the “benchmark” used as the surrogate for truth. Therefore the documentation of the benchmark must provide the basis for the credibility.

The use of DNS as a surrogate for experimental data has received significant attention. This practice violates the fundamental definition of validation we have adopted because no observation of the physical world is used to define the data. This practice also raises other difficulties, which we will elaborate upon. First the DNS code itself requires that the verification basis further augmented by a validation basis for its application.   This includes all the activities that would define a validation study including experimental uncertainty analysis numerical and physical equation based error analysis. Most commonly, the DNS serves to provide validation, but the DNS contains approximation errors that must be estimated as part of the “error bars” for the data. Furthermore, the code must have documented credibility beyond the details of the calculation used as data. This level of documentation again takes the form of the last form of verification benchmark introduced above because of the nature of DNS codes. For this reason we include DNS as a member of this family of benchmarks.

There are two ways to do great mathematics. The first is to be smarter than everybody else. The second way is to be stupider than everybody else — but persistent.

― Raoul Bott

Banks, Jeffrey W., T. Aslam, and William J. Rider. “On sub-linear convergence for linearly degenerate waves in capturing schemes.” Journal of Computational Physics 227, no. 14 (2008): 6985-7002.

Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281.

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, W. Rider, T. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LLNL-TR-411291. Lawrence Livermore National Laboratory (LLNL), Livermore, CA, 2008.

Rider, J., James R. Kamm, and V. Gregory Weirs. “Verification, Validation and Uncertainty Quantification Workflow in CASL.” Albuquerque, NM: Sandia National Laboratories (2010).

Rider, William J., James R. Kamm, and V. Gregory Weirs. “Procedures for Calculation Verification.” Simulation Credibility (2016): 31.