You should never be surprised by or feel the need to explain why any physical system is in a high entropy state.

― Brian Greene

For those of you who know me, it’s neither a secret, nor a widely known fact that I’ve gotten some tattoos recently. They aren’t the usual dreck most dudes get (like those tribal ones), but meaningful things to me. Now I have five in total, four of them are science related. One of the things that I wanted was an equation (yeah, I’m a total nerd). The question is what equation do I believe in enough to get permanently inscribed on my skin? A “common” choice for a science tattoo is Maxwell’s equations, and a friend of mine has the Euler equations on this arm from his PhD thesis. This post is about the equation I chose to care enough about to go through with it.

I’ll write the equation in TeX and show all of you a picture, you can make out a little of my other ink too, a lithium-7 atom and a Rayleigh-Taylor instability (I also have my favorite dog’s paw on my right shoulder and the famous Von Karmen vortex street on my forearm). The equation is how I think about the second law of thermodynamics in operation through the application of a vanishing viscosity principle tying the solution of equations to a concept of physical admissibility. In other words I believe in entropy and its power to guide us to solutions that matter in the physical world.

The all-knowing yesterday is obsolete today.

― Jarod Kintz

This is in contrast to much of the mathematical world that often cares about equations that are beautiful, but mean nothing in reality. A lot of the tension is related to following the beauty of Newtonian determinism and its centrality to continuum mathematical physics, and the need to embrace to stochastic nature of the real world. The real world is random and flows along time’s arrow and needs to embrace entropy and uncertainty. Our education and foundational knowledge of the physical world is based on Newton’s simplified view of things (a simplified view that revolutionalized science if not mankind’s understanding and mastery over nature). Newton’s principles can only take us so far, and we are probably reaching to end of its grasp. It is past time to push forward toward incorporating new principles into our model of reality.

Here is the equation in all its glorious mathematical statement, taking the limit as . The equation is the time rate of change of a variable determined by a flux balance and a diffusive term where the limit of diffusion is taken to zero, which implies the satisfaction of the second law of thermodynamics implying that entropy increases. OK, but WTF is it all about? So in words the equation is a hyperbolic conservation law with a diffusive right hand side where the coefficient of viscosity goes to a limit of zero. In this limit we find solutions that are physically admissible, that is ones that could exist in the real World. These solutions lead to satisfaction of the second law of thermodynamics, which implies that entropy or disorder monotonically increases in time. The second law can be viewed as the thing that gives time a direction (time’s arrow!), and without the increase of entropy, time can flow equally well forward or backward, that is being symmetric. We know time flows forward in the real world we all live in, so we want that (or at least I want that, and believe you should too).

lot of effort is spent studying the equations of inviscid flow, flow without dissipative forces most commonly the Euler equations. This form of equation is studied a lot because it is so pure. One can really get some awesomely beautiful mathematics out of it. Commonly the math leads to some great structure by studying these systems through their Hamiltonian and its evolution. Unfortunately, this endeavor while beautiful and hard has no physical merit whatsoever. No physical system really adheres to this Hamiltonian structure (except perhaps isolated systems of very small scale, and I really don’t give a much of a shit about these). They are seductive, pretty and almost without any physical utility. I care about stuff that appears in nature.

The important thing for equations to represent is physical reality (unless you’re doing math for math’s sake). As Wigner pointed out, mathematics has an incredible, almost mystic capacity to model our reality as he says is unreasonably effective. Exploiting this power should be a privilege we exercise whenever possible. In that vein the equations that connect to reality should be favored. In many cases inviscid equations are incredibly useful for modeling, but an important caveat should be exercised, the solutions to the inviscid equations that are favored are those associated with the presence of viscosity. These solutions are found through the application of an asymptotic principle, vanishing viscosity. The application of vanishing viscosity provides a route for these equations to satisfy the second law of thermodynamics, and its demands for increasing disorder.

These principles actually don’t go far enough in distinguishing themselves from inviscid dynamics associated with Hamiltonian systems. Let me explain how they need to go even further. A couple of the most profound observations associated with fluid dynamics are associated with shock waves and turbulence, and share a remarkable similarity (it might be argued that both are inevitable through dimensional similarity arguments!). For shock waves the amount of dissipation occurring via the passage of a shock is proportional to the size of the jump in the variables across the shock cubed (Bethe came up with this in 1942). For turbulence the amount of dissipation is a high Reynolds number flow is proportional to the size of the velocity variation cubed (Kolmogorov came up with this in 1941). Both relations are independent of the specific value of the molecular viscosity.

What people resist is not change per se, but loss.

― Ronald A. Heifetz

These relations are profound in their implications, which are not intuitively obvious upon first seeing it. The dissipation rate being independent of the value of viscosity means that the flow contains something that approaches a singularity. These singularities are called shock waves and have no name at all in turbulence because we don’t know what they are. These singularities are the mystery of turbulence and they are surely ephemeral as they are important. In other words we don’t see the turbulent singularities like we see shocks, but they must be there. Moreover the supposed equations of turbulence, the incompressible Euler equations don’t appear to contain obviously singularity producing features. This whole issue has produced an utterly stagnant scientific endeavor of immense practical importance.

Of course what is usually not covered is the horribly degenerate and unphysical nature of the incompressible Euler or Navier-Stokes equations. The key term is incompressible, which is intrinsically unphysical and removes sound waves from the system making their propagation speed infinite. What if these sound waves, which are always present, contain the essence of what drives dissipation in turbulent system. Incompressibility also removes thermodynamics from the equations and can only be derived from the compressible Navier-Stokes by considering the flow to be adiabatic. Turbulence in its essential character is non-adiabatic and intrinsically dissipative. Anyone see the problem(s)? Perhaps its time to start considering that the lack of progress in turbulence is exposing fundamental flaws in our modeling paradigm.

I would posit that we are trying to solve this monumentally important problem with a set of equations that we have systematically crippled. These equations were posed during an era where the fundamental issues discussed above were not well known. There really isn’t an excuse today. Could our lack of progress with turbulence be completely related to focusing on the wrong set of equations (yes!)?

Let’s dig just a bit deeper on the philosophical implications of the Bethe’s and Kolmogorov’s relations for dissipation of energy. Both of these relations also imply a satisfaction of the second law of thermodynamics by these systems. The limiting value for the satisfaction of the second law is not simply the inequality at zero, but rather an inequality for a finite value of dissipation. This finite value of dissipation is directly related to the large-scale flows structure and quantitatively proportional to the cube of the variations in the inertial range. Thus, the limit of zero dissipation is not physically relevant, the limit is a finite amount of dissipation set by the large scales of the system of interest. This deepens the implications associated with any study of completely dissipation-free dynamics being utterly unphysical. The dissipation-free system is separated from the real world by a finite and non-vanishing distance.

This feature of the physical world should be reflected in how we numerically model things (this is my philosophical point of view). It gets to the core of why I chose the equation to ink on my skin. A lot of numerical work is focused on trying to remove every single bit of dissipation from the method while maintaining stability. This mantra is tied to the belief that dissipation is bad and one is fundamentally interested in the numerical solution to the dissipation-free Euler equations. I believe this is utterly foolhardy and unproductive. The dissipation-free Euler equations are close to useless. The key dissipative relations I’ve introduced above tell us that the dissipation is never zero, but rather non-zero in a very specific way that is irreducible.

Some would argue that this non-zero dissipation should be the target of modeling, and the numerical methods should be pure and not intrude into the modeling. I believe that this perspective is laudable, foolish and unworkable practically. I favor more holistic approaches that combine modeling and numerical methods into a seamless package. This approach works wonderfully well in numerical methods for shocks, and produces a set of methods that revolutionized the field of computational fluid dynamics (CFD). I believe the grasp of these methods is far greater and extends into turbulence through implicit large eddy simulation (ILES). ILES implies strongly that the turbulence modeling is strongly addressed by techniques that practically solve compressible flows in the vanishing viscosity limit.

Generally for turbulence this approach has not been taken and the reason is clear to see. Turbulence modeling remains to this day a dominantly empirical activity. The core reason for this is the comment above about knowing what the dissipative structures are in turbulence. In compressible flows we know that shock waves are the thing to focus on and where the invariant dissipation occurs. Shocks are the hard thing to compute numerically, and we know what to do. For turbulence the same things cannot happen and the result is empirical modeling without targeted numerical methods. What remains is a philosophy that drives numerical methods to be innocuous, and allow the modeling to hold sway. The problem is that the modeling is blind to what the real physics is doing and the precise mechanisms to connect the large-scale flow to the dissipation of energy.

Nothing limits you like not knowing your limitations.

― Tom Hayes

As far as the tattoos are concerned, I haven’t decided yet if I’m getting more ink or not (its probably a yes). Maybe I’ll keep to the theme of science on the left side of my body and personal meaning on the right side of my body. Ideas are hatching, and I need to mind my tendency towards obsessive-compulsive behavior.

The law that entropy always increases holds, I think, the supreme position among the laws of Nature. If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations — then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation — well, these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.

— Sir Arthur Stanley Eddington

Wigner, Eugene P. “The unreasonable effectiveness of mathematics in the natural sciences. Richard courant lecture in mathematical sciences delivered at New York University, May 11, 1959.” Communications on pure and applied mathematics 13.1 (1960): 1-14.

Bethe, H. A. “On the theory of shock waves for an arbitrary equation of state.” Classic papers in shock compression science. Springer New York, 1998. 421-495.

Kolmogorov, Andrey Nikolaevich. “Dissipation of energy in locally isotropic turbulence.” Akademiia Nauk SSSR Doklady. Vol. 32. 1941.

Grinstein, Fernando F., Len G. Margolin, and William J. Rider, eds. Implicit large eddy simulation: computing turbulent fluid dynamics. Cambridge university press, 2007.

Combinatory play seems to be the essential feature in productive thought.

― Albert Einstein

My wife and I take part in a discussion group twice a month at our church. We get an innocuous sounding word to focus upon and set about answering deep questions about it. Everyone gets a chance to speak without interruption and everyone else focuses on listening. It’s hard (I’m really bad at listening), and it’s rewarding. Last week the word was “play”. In talking about what the concept of play means to me first in the context of childhood then adulthood I had several epiphanies about the health and vitality of our current society and workplaces. Basically, the concept of play is under siege by forces that find it too frivolous to be supported. Societally we have destroyed play as a free wheeling unstructured activity for children, and crushed the freedom to play at work under the banner of accountability. We are poorer and more unhappy as a result and it is yet another manifestation of unremitting fear governing our behaviors.

We are never more fully alive, more completely ourselves, or more deeply engrossed in anything, than when we are at play.

― Charles E. Schaefer

The greatest realization in the dialog came when I took note of how I used to play at work and all the good that came from it. The times when I have been the most productive, creative and happy with work have all been associated with being allowed to play at work. By play I mean allowed to experiment, test, and create new ideas in an environment allowing for failure and risk (essentially by placing very few constraints and limitations on what I was doing). The key was the creation and commitment to very high level goals and the freedom to pursue these goals in a relatively free way. The key is the pursuit of the broad objectives using methods that are not strongly prescribed a priori.

Work and play is the same thing just with a different perspective.

― Debasish Mridha

When I was a child, I had immense freedom to play. I would ride bikes around the neighborhood and play at the creek. My parents had a general idea where I was, but not specifically. This level of independence and freedom is almost impossible to imagine today. Children have scheduled and scripted lives where parents know their precise location at almost any time. Instead of learning to manage their lives with a high degree of independence, we teach our children to always be in control. Most of what is being controlled is a set of highly improbable risks that should not warrant such a high degree of control. We are subverting so much of the positive influence that comes from independence to control exotic and tiny probabilities. Societally, the overall impact is counter-productive and hurts us far more than protects us. The treatment of our children is good training for their lives as adults.

The same basic dynamic is working in the adult world of work. We spend an immense amount of time and effort controlling a host of miniscule risks and dangers. The feeling for children and adults alike is that controlling potential bad outcomes is worth the effort. People say things like “if we can prevent just one needless death…” which sounds compelling, but is stupid and inane. Bad things happen all the time, bad things are supposed to happen and the amount of effort spent preventing them is immense. How many lives worth of effort are spent to prevent a single death? No one ever asks if the steps being taken actually have an overall balanced positive effect pro and con.

You cannot build character and courage by taking away people’s initiative and independence

― Abraham Lincoln

A Transportation Security Administration (TSA) officer pats down Elliott Erwitt as he works his way through security at San Francisco International Airport in San Francisco, Wednesday, Nov. 24, 2010. (AP Photo/Jeff Chiu)

The TSA and airline screening is a good example. What is the cost in number of lives wasted going through their idiotic screening procedures to prevent problems. We also appear not to be able to control our reaction to bad things either. A terrorist act unleashes an avalanche of reaction that magnifies any harm the terrorists intends by orders of magnitude. Yet we continue to act the same without any realization that our fear is in fact the greatest weapon the terrorist possess. Terrorism is quite effective because the public is afraid and the societal response to terror will assist the aims of the terrorists. We have given up an incredible amount of resources, freedom and independence to protect ourselves from minuscule threats. There is a lot of evidence that we will continue to empower terrorists through our fearful responses.

Of course these trends are not solely limited to our response to terrorism. Terrorism simply amplifies the generic response of society. These trends in response occur in a variety of settings and drive short-term, low-risk behavior almost across the board. We typically encourage adults to focus on very short-term goals and take very few risks in working. The result is a loss of long-term goals and objectives in almost all settings in work. In addition the goals and objectives that do exist almost always entail little or no risk. The impact of the environment we have created is a systematic undermining of achievement, innovation and creativity in work. One way to capture this outcome is the recognition that play is not encouraged; it is actively discouraged.

We are being overwhelmed in the workplace, in the schoolroom and in every aspect of life with the concept of accountability. Accountability is one of those things that sounds uniformly good and no one can argue that it’s bad. Unfortunately I have come to the conclusion that the form of accountability we are subjecting our selves to is damaging and destructive. Accountability is used to control people and their activities. It is used to make sure people are doing what they are supposed to be doing. These days we are supposed to be doing what we are told to do. We are not supposed to be creative or innovative and do something that is unpredictable. Accountability is the box we are all being put in, which limits what we can do.

We end up working extremely hard across everything in society to make sure that bad things don’t ever happen. We put all sorts of measures in place to prevent bad things. We don’t seem to have the capacity to realize that bad things just happen and it’s a fact of life. We spend so much effort trying to manage all the risks that life is just passing us by. This manifests itself with the destructive belief that the government’s job is to protect all of us from bad things (like terrorism). We are willing to give up freedom, accomplishment and productivity to assure a slight increase in safety. Often the risks we are sacrificing so much to diminish are vanishingly small and trivial (like terrorism), yet we are making this trade over and over again. We are allowing ourselves to drown in a sea of safety measures against risks that are inconsequential. The aggregate cost of all of these risk control measures exceeds the value of almost any of the measures. It represents the true threat to our future.

In today’s world, we are in the box all the time whether as children, or as adults. Children’s playtime used to be unscripted and free more often than not. Today it is highly scripted and controlled. Uncontrolled children are viewed quite unfavorably by society as a whole. As adults the exact same thing is happening. Life and work is to be highly scripted and controlled. Anything off script or uncontrolled is considered to be dangerous and highly suspect. The desired result of this scripting and control is predictivity and reliability without risks and failure. The other impact is less happiness and less creativity, less innovation and generally worse outcomes.

Another thread to this thought process is the avoidance of passion in my work. Increasingly I find that expressing any passion or commitment at work is viewed negatively. Work is being driven to be dispassionate and free of deep of emotional connection. In the past when play was very deep part of my productive work life, I also felt great passion for what I did. That passion was tied to the entire way that I worked, and included commitments of quality and learning. More and more today such passion seems to bring nothing but condemnation and seems to be unwelcome. I don’t think that this is disconnected from the issue of play and its diminished role too.

Men do not quit playing because they grow old; they grow old because they quit playing.

― Oliver Wendell Holmes Sr.

Our most significant opportunities will be found in times of greatest difficulty.

― Thomas S. Monson

In economic policy it is well known that monopolies are bad. They are bad for everyone except the people who own and control those monopolies (who invest a lot in retaining their power!). They are drags on growth, innovation and progress. They are the essence of the too big to fail problem. In a very real sense the same thing is happening in science. We are being swallowed by monopolistic ideas. We are too invested in a variety of traditional solutions to problem (which solve traditional problems). Innovation, invention and progress are falling victim to this seemingly societal-wide trend.

We are seeing this in both computers and the codes running on the computers. The seductive nature of these quite capable behemoths is holding sway over a future that offers so much more than we are getting. The moment of epiphany came to me a while ago during some strategic planning for shock physics at work. Basically we have no strategy at all. We have 25 and 30-year-old legacy codes that we continue to develop because they are the only platforms viable within our fragmented funding picture today. The way we fund and manage the work in science is undermining progress and innovation as surely as the sun rides in the East every day.

Looking at our soon to be, if not already ancient codes based on ancient technology I asked how often did we build a new code in the old days? Sure as could be the answer was radically different than today’s world, we build new codes every five to seven years. FIVE TO SEVEN YEARS!!!! Today we are sheparding codes that are at least a quarter of a century old, and nothing new is in sight. We just continue to accrete capability on to these old codes horribly constrained by sets of decisions increasingly divorced from today’s reality, technology and problems. It is a recipe for failure, but not the good kind of failure, the kind of failure that crushes the future slowly and painlessly like the hardening of the arteries.

The deeper question is why we are functioning in this manner? I’d posit an initial answer as a tendency to be obsessively short-term focused in our goals. The stream of decisions leading to our current legacy codes is surely optimal in a per annum basis, just as it is surely suboptimal in the long run. The problem is that the long run has no constituency today. This stems from a rather fundamental societal lack of leadership and vision. We are too easily swayed by the arguments of optimal short-term thinking and unwilling to take risks or invest in the long run success. We see this spirit manifested in our political, business and scientific communities.

Perhaps no greater emblem of our addiction to shortsightedness exists than the crumbling infrastructure. The roads, bridges, electrical grids, airports, sewers, water systems, power plants,… that our core economy depend upon are in horrible shape and no will exists to support them. We can’t even conjure up the vision to create the infrastructure for the new century and leave it to privatized interests that will never deliver it. We are setting ourselves up to be permanently behind the rest of the World. We have no pride as a nation, no leadership and no vision of anything different. We just have short-term narcissistic self-interest embodied by the low tax, low service mentality. The same dynamic is happening at work.

When you do what you fear most, then you can do anything.

― Stephen Richards

We want short term, sure payoff, work without the sacrifice, risk and effort needed any long term vision or leadership. It is exactly what we are getting. We are creating a shell of our former greatness. In terms of codes and the opportunity they provide for modeling and simulation our reliance on legacy code is deeply damaging. In the days past we created new codes on a regular basis along with new modeling capability and philosophy. As a result our modeling approaches would step forward with each new code along with providing a vehicle for innovation in methods, algorithms and computer science. As a result we could try out new ideas for size without completely divesting from what came before. Without the new codes we are straightjacketed into old ideas and technology passes us by. The inability to replace our old codes resulting in legacy codes produces a massive cost in terms of lost opportunity.

How much I missed, simply because I was afraid of missing it.

― Paulo Coelho

The loss of opportunity is becoming increasingly unacceptable. We are producing a future that is shorn of possibilities that should be lying in front of us. Instead of vast possibilities energized by continual changes in our foundations, we have stale old codes, models, methods and algorithms that ill-serve our potential. The application of too big to fail to our codes is creating a slow-motion failure of epic proportions. The basis for the failure is the loss of innovation and a sense that we are creating the future. Instead we simply curate the past. Our best should be ahead of us and any leadership worth its salt would demand that we work steadfastly to seize greatness. In modeling and simulation the creation of new codes should be an energizing factor creating effective laboratories for innovation, invention and creativity providing new avenues for progress.

I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops.

― Stephen Jay Gould

This week I’m in Las Vegas an appropriately ironic place to see people making stupid gambles on the future. We are having the annual “Principal Investigator’s” meeting for the NNSA’s ASC program, itself a rather ironic name for an echo chamber, which is a more apt description. There is frightfully little investigation or intellectual engagement
in sight, it is more of a programmatic, project meeting with little or no discussion of intellectual depth, at least publicly. Private discussions among the technically oriented attendees are a bit more far reaching, but everyone seems to feel a deep sense of following a fate rather than making choices. The lack of holistic thought and vitality in high performance computing is becoming evident, but the machine focus is a juggernaut too powerful to slow down at this point.

If failure is not an option, then neither is success.

—Seth Godin

Societally, the concept of too big to fail applies to the banking and financial institutions that almost destroyed the World economy eight years ago. We demonstrated that they were both too big to fail and too big and too powerful to change thus remaining a ticking time bomb. It is only a matter of time before the same issues present in 2007 erupt again and wreck havoc on the World economy. All the evidence needed to energize real change is available, but there is simply too much money to be made, and greed is more powerful than common sense. I realized that our application codes and computers probably properly deserve to be thought of in exactly the same light, they are too big to fail too. This character is slowly and steadily poisoning the environment we live in and any discussion of different intellectual paths is simply forbidden.

Only those who dare to fail greatly can ever achieve greatly.

In high performance computing we live in immensely challenging times where deep intellectual engagement is necessary for success. The very nature of the programs seems to be anathema for the very free thought needed for success. The codes and computers are treated as being absolutes for success and immutable. We see decades of investment in codes and capabilities that must be sustained. The systems we have created are immense in terms of expense and size. The idea has taken hold that they must be preserved. This preservation is rather superficial rather than holistic and pervasive; the concept in analogous to playing the codes in amber. As such the intellectual content of the codes is remaining far too static and our intellectual ownership of the contents of the codes is slipping away. It is a dangerous and unsustainable future. Like the banks that should have been split into smaller more manageable chunks, the codes need to be removed from this concept of permanence.

As I said, the depth of intellectual ownership of these very codes is diminishing with each passing day. The essential aspects of these code’s utility and success in our application areas is based on deep knowledge and intense focus of talented individuals. The talent and skills leading to successful codes are difficult to develop and maintain; the skills must be developed by simultaneously pushing several envelopes: the applications, the models, methods to solve models, and computer science-programming. Today we really only focus on the computer science-programming and simply sort all the other details. Rather than continually reinvest in people and science, we are creating an environment where codes are curated. This state is actually a recipe for catastrophic failure rather than glorious success. The path forward should be adaptive, flexible and agile; instead the path is a lumbering goliath and viewed as a fait accompli.

Any fool can know. The point is to understand.

A code is not an investment and shouldn’t ever be viewed as such. A code is simply a computer executable version of independent thought and intellectual content. It is absolutely vital for all of the capability we have in code to be fully understood and known by humans. We need to have humans who understand the basis of models and how these models are solved. When we curate code this key connection is lost. We lose the fundamental nature of the model as our impression of nature, rather than its direct image. We use models as a way of explaining nature rather than a substitute for the natural World. This tie is being systematically undermined by the way we compute today and results in a potentially catastrophic loss of humility. Such loses of humility ultimately produce reactions that are unpleasant and damaging.

We are creating a program that will collide with reality leaving a broken and limping community in its wake. It has a demonstrated track record of not learning from past mistakes, producing a plan for moving ahead that is devoid of innovation and deep thought. Today’s path forward is solely predicated on the idea that we must have the fastest computer rather than the best computing. It is the epitome of bigger and more expensive is better, rather than faster, smarter and more agile. Perhaps more damaging is a perspective that the problems we face are already solved save the availability of more computer power. We will end up eviscerating the very communities of scientists that are the lifeblood of modeling and simulation. The program may be a massive mistake and no one is questioning any of it.

Judge a man by his questions rather than by his answers.

I believe that all of these efforts could vastly benefit from a mindset orthogonal to the prevailing approach. How would we solve the problems facing us today if we had less computing power? If we thought about how to productively solve our problems with less computational horsepower, we could do a far better job with whatever computers we actually have. I find the call for more computer power as a way of dodging deeper and more challenging problems with an effectively kneejerk response. More computing power is undeniably better, but it is almost always a highly suboptimal path to better solutions to problems.

The most effective way to get a better answer is provide a better model of reality to address the questions. If you believe that your model is correct and appropriate for the questions at hand, the method of solution has the most leverage for improving your performance. Many of these solution methods are based on fundamental algorithms, which can provide massive upgrades in performance. In each of these endeavors the use of deep applied mathematics expertise can provide tremendous benefits to the rigor and effectiveness of each aspect. Only when these options have been exhausted should the implementation and hardware be brought to bear as the primary path to improvement. In today’s high performance computing research the highest leverage paths to improved modeling and simulation are virtually ignored by our efforts. Of course part of the issue is the identification of the activity as high performance computing first, and modeling & simulation second. This ordering and priority should be reversed in keeping with their proper role for impacting the real World applications that should be motivating all of our efforts.

There is a cult of ignorance in the United States, and there has always been. The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that ‘my ignorance is just as good as your knowledge.

What are the odds that people will make smart decisions about money if they don’t need to make smart decisions—if they can get rich making dumb decisions?

― Michael Lewis

Left to right: Steve Carell plays Mark Baum and Ryan Gosling plays Jared Vennett in The Big Short from Paramount Pictures and Regency Enterprises

Over the Christmas holiday my family and I went to see “The Big Short” an excellent movie about the financial collapse in 2007-2008. It was almost impossible to come away from the movie without feeling a deep sense of rage over how things unfolded. The level of criminality, greed and distain for humanity that fueled the collapse has largely gone unpunished, and driven precious little change in the system that allowed things to unfold. The reality is that we are primed for the same thing to happen all over again with only the details changing. The issues that allowed this calamity are infesting society at large and corrosively aimed at almost every great institution we depend upon.

“Guys who can’t get a job on Wall Street get a job at Moody’s,” as one Goldman Sachs trader-turned-hedge fund manager put it.”

― Michael Lewis

One scene in “The Big Short” stands out as helping define the depth of the dysfunction in the system, the trip to Moody’s, the rating agency for the securities. The securities created by the banks were incredibly unstable and literally junk, yet the ratings agencies kept putting their top seal of approval on them, AAA. When pressed on the matter, the woman representing Moody’s said, “if we don’t give them the rating they want, the guy down the street will, we want the business.” The people watching the system for fraud were completely in bed with the crooks. The reality is that this practice and problem are everywhere. It is true where I’ve worked, it is obviously true in politics, and sports, and education, and… Our whole nation is living in the Golden Age of Bullshit. This serves no purpose but perpetuating the existing structures of power at the cost of progress, quality and ethical behavior.

Mark Baum: We live in an era of fraud in America. Not just in banking, but in government, education, religion, food. Even baseball…

– The Big Short

The issue exposed in the Big Short is permiating society. The basic problem is that no one can be responsible bcause being responsible will just get you replaced by a more corrupt or corruptible irresponsible person. The sorts of peer reviews that we see at work are the same thing. Everything is graded on a curve, and a bad failing grade is never allowed. Failure isn’t allowed, if it comes up the messenger is “shot” (usually by being dismissed from the peer review). We never confront any problems until they blow up in our faces. This tendency basically allows progress to grind to a proverbial halt. Failures are the fuel for progress and when you disallow failure, you disallow progress.

The signs of this form of corruption are everywhere. We cannot have a decent or deep conversation about anything controversial. Climate change is a great example of the sort of bullshit responses that hamper us today. If everyone is an expert then no one is an expert. If people don’t like the information they get, they find someone else who gives them a different answer. As a result science in our society is in decline. Actual science is being hurt, and science’s role in society is similarly degrading. Look at the whole anti-vaxxer movement, which has absolutely no basis, but lots of proponents. We get ideas where any risk at all is unacceptable and we allow progress to grind to a complete halt. Failure, problems and the identification of things that need to be improved creates the basis of valuable work. We have structurally destroyed mechanisms for doing this by our addiction to praise and inability to identify and confront problems while they are small.

All intellectual tendencies are corrupted when they consort with power.

― Clive James

I run in a lot circles where large scale decisions are made and money is distributed. A culture of empty-headed management has crept into how things are done. People managing scientific programs are much more interested in milestones and Gantt charts than whether the work is any good. They seem to have conflated the practice of project management with quality. As a result we end up managing the wrong things and syphoning off the key elements in useful progressive research.

The ability to make progress and innovate has very little to do with anything project management provides. Progress often comes from applying new thinking to old problems. One of the key things to do is identify and take on unsolved problems, another name for failures. Making progress is often the antithesis of things that can be managed in today’s common fashion, so progress makes way for management satisfaction. The key is the project management is simply a tool, and a useful one at that. It is not a recipe for success or an alternative to thinking deeply and differently.

Lastly, we have a seemingly ever-present din of the question, “will things ever be done?” I hear this repeatedly from those managing and funding stockpile stewardship. The answer is NO and will always be NO. At least if we want a competent and reliable science basis for national defense, the economy or just a generally progressive society, the work is never done. This question is so utterly lacking in thought and intellectual depth that it should be met with complete distain.

Thinking is the hardest work there is, which is probably the reason so few engage in it.

― Henry Ford

People thinking that science is something with a finite starting and stopping point have no business whatsoever being in any form of leadership. Science is a quintessential human activity and we humans make progress continually. To deny progress should be against the nature of our species. Killing progress is certainly opposed to the best interests of society at large. Science in support of national defense has been one of the most fruitful avenues for progress in science with benefits far beyond killing other people more efficiently. Most of today’s economy is founded upon the results from defense related research. These opponents of progress are basically enemies of the state, trying to undermine the foundations of our Nation’s strength. They’re out there, enemies of progress, friends of corruption trying to keep us from solving our problems. They’re everywhere.

Science is a way of thinking much more than it is a body of knowledge.

― Carl Sagan

The essence of the independent mind lies not in what it thinks, but in how it thinks.

― Christopher Hitchens

Even with a day off, work last week really, completely sucked. I got to spend very little time doing my daily habit of writing in a focused manner. Every day at work was a pain and ended the week with hosting a group of visitors who are responsible for part of the new exascale computing initiative. Among the visitors were a few people whom I have history with both good and bad. If you’ve read this blog you know that I’m not a fan of the exascale computing imitative. Despite this, I was expected to be on my best behavior (and I think that I was). It was not the time, nor place to debate the program’s goals or wisdom (to be honest I’m not sure what the right time is). It’s pretty clear to me that there hasn’t been much of any debate or thought put into the whole thing. That’s a discussion for a different day.

Thinking is the hardest work there is, which is probably the reason so few engage in it.

― Henry Ford

Nonetheless some good came from the experience aside from demonstrating my own self-control. I won’t say much about the visit except that the exascale initiative is not terribly compelling as programs go, and I thought it went well from our official perspective. I have a better idea of how they are viewing the program and its objectives and priorities. We had a chance to talk about how we are approaching a similarly structured program. No one is thinking about all the missing elements from the approach in a constructive way, and lots of old mistakes are being made all over again. People show a remarkable lack of historical perspective and ability to engage in revisionist history. The refrains of my “bullshit” post on lack of honesty in the view of success rang in my ears.

Stop thinking, and end your problems.

― Lao Tzu

I also noted the distinct air of control from the visitors and discussion of their colleagues who run our programs. The programs want to give us very little breathing room to exercise our own judgment on priorities. They want to define and narrow our focus to their priorities. Given the lack of technical prowess from those running things it’s dangerous. Awful programs like exascale are the direct result of this sort of control and lack of intellectual thought running research. Everything is politically engineered and nothing is really composed of elements that are designed to maximize science. The result is a long-term malaise in research, progress and science we are suffering from. Ultimately, the system we are laboring under will result is less growth and prosperity for us all. It is the inevitable result of basing our decisions on fear and risk avoidance instead of hope, faith and boldness.

The conventional view serves to protect us from the painful job of thinking.

― John Kenneth Galbraith

Because our program is all about stockpile stewardship the meeting was held in a classified setting. This means no electronics and a requirement that I unplug. It might be a good excuse to get some reading done, but I had to look like I was paying attention all day. So I took copious notes. Not much interesting happened so most of the notes were to self and captured my thoughts, reflections and perspectives. This alone made the entire experience valuable from a personal-professional perspective. I managed to digest a lot of my backlog of thinking that the well-connected World distracts you from constantly. I had some well-structured time with my own thoughts and that’s a really good thing.

The only freedom you truly have is in your mind, so use it.

― M.T. Dismuke

Getting away from the electronic world of web pages, text messages, email for a while is a blessing. I could approach my thoughts with a literal clean sheet. I started by reflecting on all the good ideas I’ve had recently, but haven’t gotten the time to work on. It was a lot, which has a depressing aspect. There is so much to potentially work on that I can’t. Its worse that I don’t exactly see the value in what I am working on. It’s a bit of a personal tragedy. I suspect its one that plays out across the World of research. We have less and less time to work on things we judge to be important.

Aside from the deeper thoughts I also realize that it pays to think in many different ways even from a mechanical point of view. I try walking each day along with a walking meditation, followed by free association. It ends up being a very effective way to self-brainstorm. I keep a notebook for each day in a cloud app. There is this blog, which allows for a freeform prose, but done in an electronic form. Writing things down on paper has subsided a lot, and last week I rediscover the virtue of that medium albeit by the nature of the circumstances. For a long time I kept a pad of lined post-it notes in my car since a lot of good ideas would just come to me driving to and from work. It might be good to force myself to use paper alone more often. By the power of cameras and remarkable text recognition the paper can go directly into my electronic notebook any way.

The important thing to me is to capture the ideas that move from the background of my thinking to the foreground. Some of these thoughts are half-baked, but others are really genius. The human mind is a remarkable thing especially when it’s subjected to lots disparate input. The day away from electronics was good for rebooting how I approached free thinking when its available. I’d like to think its what I’m paid for, but honestly that isn’t really likely to be the truth. Everything about how I’m paid is about not really thinking about the deeper meaning. We are encouraged to simply putter along doing as we’re told. The mantra of today is quit thinking and get back to work.

Power does not corrupt. Fear corrupts… perhaps the fear of a loss of power.

― John Steinbeck

Now we get to the darker aspects of free association, you start to turn your gaze toward the shit show unfolding before you. Life today is full of things that should be regarded with contempt. Our overlords encourage us to ignore the carnage they are subjecting the world to, but it is there hidden in plain sight. Today we live in a coarse and belligerent culture that threatens to undermine everything good. I’m not talking about the sort of moral decay social conservatives would point to. I’m talking about the fundamental rewards, checks and balances that encourage an environment of selfish and greedy behavior. At the same time these same forces work to undermine every effort to pay attention to larger societal, organizational and social imperatives that collectively make everything better. We act selfish in the service of maintaining the power of others, and avoiding the sort of collective service that raises everyone.

So I was offered a front road seat at a primo shit show, and here is what it made me think.

Our research is now running on the basis of money as a scoring system with no real concrete societal objectives in sight. In the 20^th Century many great things were accomplished and the technology that dominates our economy was invented through scientific discovery. A great deal of that discovery was directly associated with fear, first of Germans and then the Nazis then the Soviets. The atomic bomb, hydrogen bomb, jet aircraft, microprocessors, cell phones, GPS, and almost every in our modern world owe their discovery to this response to fear of existential threats. These were real adversaries with well-developed technology, engineering and science requiring a serious response of our Nation-State to the threat they represented. Today, we see a bunch of disorganized barbarians as an existential threat. It is completely pathetic. We really don’t have to have our collective act together to compete. It’s all fear and no benefit of accomplishing great things, and we aren’t. We just have the requisite reduction in freedom in response to this fear without any of the virtues. This dismal state of affairs results in a virtual emptying of meaning from work that used to be important. I work at a place where work ought to have value and importance, yet we’ve managed to ruin it.

Power attracts the corruptible. Suspect any who seek it.

― Frank Herbert

It is utterly stunning that working for an organization committed to National Security does not provide me with any sense that my work is important. I don’t have enough
latitude and capability to exercise my judgment to feel truly empowered at work. All the control and accountability at work is primarily disempowering employees and sucking all the meaning from work. I ought feel an immense amount of importance to what I do. My management, writ large, is managing to destroy something that ought to be completely easy to achieve. This malaise is something we see nationally as the general sense that your work has little larger meaning is used to crush people’s wills. Instead of empowering people and achieving their best efforts, we see control used to minimize contributions and destroy any attempt toward deeper meaning. This sense is deeply reflected in the current political situation in the World and the broad sweeping anger seen in the populace.

The love affair with corporate governance for science is another aspect of the current milieu that is deeply corrupting science. Our corporate culture is corrupting society as a whole and science is no exception. The greed and bottom line infatuation perverts and distorts value systems and has systematically harshened the cultures of everything it touches. Increasingly, the accepted moral thing to do is make yourself as successful as possible. This includes lying, cheating and stealing if necessary (and you can get away with it). More corrosively it means losing any view of broader social, societal, organizational or professional responsibility and obligation. This undermines collaboration and free exchange of ideas, which ultimately destroys innovation and discovery.

Accountability has been instituted that allows people to ethically ignore the broader context in favor of narrow focus. They are told that doing this is the “right” thing to do, and basically they should otherwise mind their own business. This attitude extends to society as a whole and we are all poorer for it. We keep ideas to ourselves, and the narrowly defined parochial interests of those who pay us. Instead we should operate as engaged and collaborative stewards of our society, organizations or professions. We have adopted a system that encourages the worst in people rather than the best. We should absolutely expect problems to be caused by this culture of selfishness. The symptoms are everywhere and threaten our society in a myriad of ways. The only portion of society that benefits from our present culture is the rich and powerful overlords. These systems maintain and expand their ability to keep their corrupt and poisonous stranglehold on everyone else.

A man who has never gone to school may steal a freight car; but if he has a university education, he may steal the whole railroad.

― Theodore Roosevelt

What I cannot create, I do not understand.

― Richard Feynman

The Richard Feynman quote “what I cannot create I do not understand” appeared on his chalkboard at his death. I realized that I have generally run my professional life by this principle. It isn’t enough merely to demonstrate rote knowledge; one needs to understand the principles underlying the knowledge. One way to demonstrate the mastery over knowledge is utilize the current knowledge and then extend the knowledge in that area to something new. This gets to the core of our current problem in science, we are not being asked to extend knowledge, and we are asked to curate knowledge. As a result we are losing the ownership that denotes mastery.

A core example of the issue is the capability to completely understand the material they are responsible for. If we are implementing things in a computer code can we rederive all the expression in the code? Do we understand the assumptions, conditions and caveats with the actual expressions? Can we extend or modify the expressions as the situation calls for it? Today in very many cases the answers to these questions is no. This is true for models we use, methods used for solution and algorithms we depend upon. This broad-based lack of intellectual ownership is a direct threat to our ability to ably provide stewardship of our missions dependent upon these products.

I learned very early the difference between knowing the name of something and knowing something.

― Richard Feynman

I want to be very clear about what I’m commenting on, we have substantial intellectual ownership of the code we write, but not necessarily what that code does. What we often do not have ownership of is the contents of that code, the models, the methods solving those models, and the algorithms the methods are based upon. We own the contents of the implementation, the software libraries, and the mapping of all of these to modern computing architectures. Because of the demise of Moore’s law we are exploring a myriad of extremely exotic computing approaches. These exotic computer architectures are causing implementations to become similarly exotic. In a sense my concern is that the difficulty of simply using these computers has the effect of sucking “all the oxygen” from the system and leaves precious little resource behind for any other creative endeavor, or risk taking. As a result we have no real progress being made in any of the activities in modeling and simulation beyond mere implementations.

A large part of my argument hinges upon the intellectual core of the value proposition associated with modeling and simulation. The question is whether progress in modeling and simulation is most greatly benefitted by greater computing power? Or improved models? Or improved methods? Or improved algorithms? The answers to these questions are not uniform by any means. There are times when the greatest need for modeling and simulation is the capacity of the computing hardware. At other times the models, methods or algorithms are the limiting factors. The question we should answer is what is the limiting factor today? It is not computing hardware. While we can always use more computing power, it is not limiting us today. I believe we are far more limited by our models of reality, and the manner in which we create, analyze and assess these models. Despite this lack of need for improved hardware, computing hardware is the focus of our efforts.

The program I work under is called stockpile stewardship. We act far more like stockpile curators. The general state of affairs is rapidly evolving to a state where no one really has the intellectual ownership of the content of their work. There is frightfully little freedom in defining a path toward greater understanding. The inertia of the “this is the way things are done around here” is so strong that it derails a great degree of progress. Intellectual ownership and progress are intimately related. The true ownership of knowledge and the capacity to produce progress are often, if not almost always one and the same.

In computational modeling we are largely in the business of stewarding legacy codes full of knowledge being curated. The choice of legacy codes is predicated on the ability of the codes to simulate issues of interest from an application point of view. It is a safe way to proceed in the short term while immensely dangerous in the long term. These codes are immensely complicated and full of wide swaths of both explicit and implicit knowledge. In many ways the models, methods and algorithms in the code are most completely documented in code, and other forms of documentation are woefully incomplete. Much of the knowledge of the decisions made in defining the code is contained only in the head’s of the people who originally wrote the code. Over time those people go away and the logic and rationale for the code’s form and function begins to fade away. We often find that certain things in the code can never be changed lest the code become non-functional. We are left with something that looks and feels like magic. It works and we don’t know why or understand how it works, but it does.

As I stated above I run my professional like by recreating existing the things. This makes dead certain that I understand something, and the only way to validate this understand is to move past the strictures and barriers of existing understanding. To truly own knowledge is equivalent to expanding that knowledge. You confirm your ownership of knowledge by the process of creation. This creative process is part of the essence of research. Thus by the virtue of the creation of legacy codes we are confirming the lack of research in the areas of knowledge vital to stewarding our stockpile.

I think nature’s imagination Is so much greater than man’s, she’s never going to let us relax.

― Richard Feynman

With the stewardship program we are in a very precarious position. We are not allowed to design new weapons. We are not allowed to fully test our ideas either. An important part of the space of knowledge is taken off the table and relegated to being purely curated. This full demonstration has the role of providing an important feedback of reality to the work being done. Reality is very good at injecting humility into the system when it is most needed. When the knowledge is curated we rapidly remove important and essential aspects of stewardship. We have immense issues associated with the long-term responsibility of caring for a stockpile. New issues arise that are beyond the set of conditions the systems were originally designed for. All of this needs a fertile intellectual environment to be properly stewarded. We are not doing this today. Instead the intellectual environment is actually being steadily eroded in favor of curating knowledge. In computing, the creation of legacy codes is a key symptom of this environment.

It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.

― Richard Feynman

In taking the raw intellectual material going into a production code many things must be simultaneously integrated (by production code I mean the codes we use to do serious application modeling for applied programs, like stockpile stewardship). This integration of intellectual material includes fundamental models, the closure of those models, the methods of solving those models and the utilization of algorithms for those methods. Furthermore there is a specific implementation in computer code that is suitable or better yet, optimized for the computing hardware.

A number of tricks, or practical accommodations are made in getting the models, methods and algorithms to work effectively for solving “real” problems with all of their dirty and realistic aspects. Most of these tricks are not something people are proud of, or can even explain in a coherent way. In many cases, the trick simply works and often a number of other tricks were tried first, and didn’t produce the desired outcome. The trick itself is rarely documented as such, and the tricks that didn’t work are usually undocumented. Many of the tricks are far more obvious and logical to use, and their failure is usually unexplained. Hence the production code works on the basis of tricks of the trade that are often history dependent, and rarely explained, yet utterly essential.

It is this point where the real horror show of legacy codes unfolds. The production code becomes a key aspect of executing an applied program, and the tricks necessary to make the code work are encoded into all of the results. The author of the tricks gets older and ultimately retires or dies, or gets a better job. When this happens the tricks become part of legend or lore and their capacity to make the code work achieves a magical status. The code’s results depend on all of the tricks, and the custody of the code passes to new people. These new people can’t change the tricks because they don’t understand the tricks. In very many cases, the tricks don’t look any different than the rest of the code and are indistinguishable from the parts of the methodology that are coherent and logical. It is all one big mess of coherent and incoherent ideas that simply gets stewarded much in the manner that monks steward old religious texts.

This is a generically awful situation that happens over and over again. In the programs I work for it is the standard way things unfold. The reason this happens is because creating a new production code is a risky thing. Most of the time the effort fails. The creation of the code requires a good environment that nurtures the effort. If the environment is not biased toward replacing older codes with new codes (i.e., progress and improving technology), the inertia of the status quo will almost invariably win. This inertia is based on the very human tendency to base correctness on what you are already doing. The current answer has a great deal greater propriety than the new answer. In many cases the results of existing codes provide the strongest and clearest mental image of what a phenomena looks like to people utilizing modeling and simulation especially in fields where experimental visuals do not exist.

For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.

― Richard Feynman

Why should things be easy to understand?

― Thomas Pynchon

The feeling is less like an ending than just another starting point.

― Chuck Palahniuk

This is another reply to my critique of our modernization program for codes, which is really creating a whole new generation of legacy codes. This is proposing a solution to this that is the acceptance of the inevitable.

The path toward better performance in modeling and simulation has focused to an unhealthy degree on hardware for the past quarter century. This focus has been driven to a very large degree by a reliance on Moore’s law. It is a rather pathetic risk avoidance strategy. Moore’s law is the not really a law, but rather an empirical observation that computing power (or other equivalent measures) is growing at roughly a rate of doubling every 18 months. This observation has held since 1965 although its demise is now rapidly upon us. The reality is that Moore’s law has held for far longer than it ever could have been expected to hold, and its demise is probably overdue.

For microprocessors, Moore’s law died around 2007, and now only lives via increasing reliance upon parallelism (i.e., lots of processors). Getting the performance out of such massive parallelism is enormously difficult and practically unachievable for an increasingly large span of methods, procedures and algorithms. Our lack of getting the advertised performance out of computers has been a large and growing problem systematically ignored for the same quarter century. It is papered over by measuring performance by a benchmark that has virtually no resemblance of any useful application and is basically immaterial to real progress.

We can almost be certain that Moore’s law will be completely and unequivocally dead by 2020. For most of us its death has already been a fact of life for nearly a decade. Its death during the last decade was actually a good thing, and benefited the computing industry. They stopped trying to sell us new computers every year and unleashed the immense power of mobile computing and unparalleled connectivity. Could it actually be a good thing for scientific computing? Could its demise actually unleash innovation and positive change that we are denying ourselves?

Yes!

What if the death of Moore’s law is actually an opportunity and not a problem? What if accepting the death of Moore’s law is a virtue that the high performance computing community is denying itself? What might be gained through embracing this reality?

Each of these questions can be answered in a deeply affirmative way, but requires a rather complete and well-structured alteration in our current path. The opportunity relies upon the recognition that activities in modeling and simulation that have been under-emphasized for decades provide even greater benefits than advances in hardware. During the quarter century of reliance on hardware for advancing modeling and simulation we have failed to get the benefits of these other activities. These neglected activities are modeling, solution methods and algorithms. Each of these activities entails far higher risk than relying upon hardware, but also produce far greater benefits when breakthroughs are made. I’m a believer in humanity’s capacity for creation and the inevitability of progress if we remove the artificial barriers to creation we have placed upon ourselves.

The reasons for not emphasizing these other opportunities can be chalked up to the tendency to avoid high-risk work in favor of low risk work with seeming guarantees. Such guarantees come from Moore’s law hence the systematic over-reliance on its returns. Advances in models, methods and algorithms tend to be extremely episodic and require many outright failures with the breakthroughs happening in unpredictable ways. The breakthroughs also depend upon creative, innovative work, which is difficult to manage in the manner of the project management techniques so popular today. So under the spell of Moore’s law, why take the chance of having to explain research failures when you can bet on a sure thing?

If we look at the lost opportunities and performance from our failure to invest in these areas, we can easily see how much has been sacrificed. In a nutshell we have (in all probability) lost as much performance (and likely more) as Moore’s law could have given us. If we acknowledge that Moore’s law’s gains are actually not seen in real applications, we have lost an even greater level. Our lack of taste for failure and unpredictable research outcomes is costing us a huge amount of capability. More troublingly, the outcomes from research in each of these areas can actually enable things that are completely different in character than the legacy applications. There are wonderful things we can’t do today because of the lack of courage and vision. Instead the hardware path we are on almost assures that the applications only evolve in incremental, non-revolutionary ways.

If we finally accept that Moore’s law is dead, can we finally stop shooting ourselves in the foot? Can we start to support these activities with a proven track record and undeniable benefits? If we do not, the attempts to utilize the hardware to produce exascale computers will siphon all the energy from the system. The starvation of effort toward models, methods and algorithms will only grow. The gulf between what we might have produced and what we actually have will only grow larger and more extreme. This is an archetypical opportunity cost. Moreover we need to admit to ourselves that for any application we really care about, the term exascale is complete bullshit. If the press release says we have an exascale computer, for an actual application of real interest we might have one-hundredth of that speed. This might actually be optimistic.

To make matters worse, the imbalance in research and effort is poisoning the future. The hardware path has laid waste to an entire generation of modeling-simulation scientists who might have been able to conduct groundbreaking work. Instead they have been marshaled into the foolhardy hardware path. The only reasons for choosing hardware are the belief that it is easier to fund and yields guaranteed (lower) returns. Our management needs to stop embracing this low bar and begin to practices an effective management of the future. The depth of my worry is that we do not have the capacity to manage a creative environment in manner that accepts the failure necessary for success. We have become completely addicted to the “easy” progress of Moore’s law, and forgotten how to do hard work.

Perhaps the end of Moore’s law for real can provide the scientific computing community with a much needed crisis. I believe that the way out of the crisis is simple and easy. The path has been trod before, but we have lost the ability to walk it. We need to allow if not encourage risk and failure in forgotten areas of endeavor. We need to balance the work appropriately realizing the value of each activity, and focus on the work where we have need and opportunity. A quarter century of reduced effort in models, methods and algorithms probably means that efforts will yield an avalanche of breakthroughs if only the effort is marshaled. To unleash this creative tsunami, the steady march toward exascale needs to halt because it swallows all effort. We are creating computers so completely ill-suited to scientific computation that simply using them is catastrophic to every other aspect of computing.

Another more beneficial aspect of changing our perspective and accepting Moore’s law’s death is how we compute. Just as the death of Moore’s law in processors unleashed computing into the mobile market and an era of massive innovation in use of computing, the same can happen for scientific computing. We only need to change the perspective. Today the way we use computing is stuck in the past. The way computing is managed is stuck in the past. Scientists and engineers still solve their problems as they did 25 years ago, and the exascale focus does little to push forward. We still exist in the mainframe era with binding policies from the IT departments choking innovation and progress.

A pessimist sees the difficulty in every opportunity; an optimist sees the opportunity in every difficulty.

― Winston S. Churchill

To avoid criticism say nothing, do nothing, be nothing.

― Aristotle

One warning there will be some bad words in the post today. If you don’t like it, don’t go on. It is the way most of us really talk about stuff!

Earlier this week I gave a talk on modernizing codes to a rather large audience at work. The abstract for the talk was based on the very first draft of my Christmas blog post. It was pointed and fiery enough to almost guarantee me a great audience. I can only hope that the talk didn’t disappoint. A valid critique of the talk was my general lack of solutions to the problems I articulate. I countered that the solutions are dramatically more controversial than the statement of the problems. Nonetheless the critique is valid and I will attempt to provide the start of a response here.

Here are the solutions I would propose at a high level:

1. Constantly question ourselves on whether we are going in the right direction. Today we follow directions mindlessly, and refuse to make reasonable course corrections. (this point was a part of the talk! I think it is the vital starting point)
2. Micromanagement is killing us; we need to macromanage at focus on large objectives.
3. Focus on improving reality and stop focusing on large-scale projects that produce impact in the real world.
4. Destroy the mediocrity, obedience and compliance cultures that have arisen because of fear-based management and decision making
5. Allow failure, encourage risk, and celebrate mistakes
6. If something is really failure, allow it, say it and be all right with it. No more bullshit statements of success where reality is a train wreck.
7. Create a learning environment where failure and mistakes create opportunity for growth
8. Focus on excellence rather than subservience and mediocrity.
9. Stop using money as a measure of success
10. Rid us of short-term thinking and measurement of success. Start looking toward long-term success
11. Rid ourselves of the vast amount wasted effort going into menial tasks that serve no purpose (obedience and compliance related)
12. Get rid of project management of science. This concept is utterly and completely destructive for the conduct of science.
13. We are never prepared for serendipitous outcomes and the change of direction they should provide.
14. We are ruled by fear and use this fear as the reason to avoid risk and fail to reach highly for achievements.
15. Quit allowing our system to turn failure into success. Let failure happen. Celebrate. Learn move forward. By allowing failure to be rebranded as success lets failure serve the wrong purpose.

If failure is not an option, then neither is success.

― Seth Godin

ASC is a prime example of failing to label and learn from failures. As a result we make the same mistakes over and over again. We are currently doing this in ASC in the march toward exascale. The national exascale initiative is doing the same thing. This tendency to relabel failures as success was the topic of my recent “bullshit” post. We need failure to be seen as such so that we do better things in the place of repeating our mistakes. Today the mistakes simply go unacknowledged and become the foundation of a lie. Such lies then become the truth and we lose all contact with reality. Loss of contact with reality is the hallmark of today’s programs.

Remember the two benefits of failure. First, if you do fail, you learn what doesn’t work; and second, the failure gives you the opportunity to try a new approach.

—Roger Von Oech

One of the serious problems for the science programs is their capacity to masquerade as applied programs. For example ASC is sold as an applied program doing stockpile stewardship. It is not. It is a computer hardware program. Ditto for the exascale initiative, which is just a computing hardware program too. Science or the stockpile stewardship missions are mere afterthoughts. The hardware focus becomes invariant to any need for the hardware. Other activities that do not resonate with the hardware focus simply get shortchanged even when they have the greatest leverage in the real world.

[Warning foul language ahead!]

In other words, when things are fucked up, someone needs to say so, and the fuck up needs to be acknowledged. If we don’t we can’t learn and we end up fucking up again. The general bullshit way of reporting success for things that are not successful creates the environment where fuck ups are not viewed as such. Good fuck ups should not end careers they should be highlights as long as you learn from it. If you don’t learn from your mistakes then you have a problem. Today we have the bigger problem of never allowing ourselves to ever learn from a genuine mistake, which only plants the seeds of ever greater fuck ups in the future.

The ubiquity of the humor of the Dilbert comics lends perspective to all of this. Dilbert is dominated by issues from the corporate world and seems to indicate that the issues we are having a pervasive society-wide. Corporate governance is as messed up as government funded research. I’m almost certain that academia is in a similar or worse shape. Some places like NASA are envious of the screwed up state of affairs we have in DOE. I probably don’t court too much controversy by saying that money is at the root of many problems largely because it’s become the universal way of keeping score of success.

Judge a man by his questions rather than by his answers.

― Voltaire

Dare to think for yourself.

― Voltaire

The beginning of the year is a prime time for such a discussion. Too often the question of importance is simply ignored in lieu of simple and thoughtless subservience to other’s judgment. If I listen to my training at work, the guidance is simple. Do what you’re paid to do as directed by your customer. This is an ethos of obedience and your particular judgment and prioritization is not really a guide. This is a rather depressing state of affairs for someone trained to do independent research; let someone else decide for you what is important, what is a priority. This seems to be what the government wants to do to the Lab, destroy them as independent entities, and replace this with an obedient workforce doing whatever they are directed to do.

This is one of the innovator’s dilemmas: Blindly following the maxim that good managers should keep close to their customers can sometimes be a fatal mistake.

― Clayton M. Christensen

When one takes a long critical look at your customer’s credentials and knowledge, the choice to seek their guidance on importance and priority becomes laughable. They are usually radically less competent to make such choices than most of you. Perhaps this is just my good old-fashioned Los Alamos arrogance showing. My customers don’t actually know best! The problem with the current standard of priority setting is that the customer’s guidance is destroying research, and slowly suffocating formerly great sources of innovation and excellence. Where independent and competent scientific and engineering leadership once underpinned the Nation’s security, the forced compliance and obedience has bred cultures of mediocrity and pathological avoidance of risk driven by fear of any failure.

An important, but depressing observation about my work currently is that I do what I am supposed to be doing, but it isn’t what is important to be doing. Instead of some degree of autonomy and judgment being regularly exercised in my choice of daily activities, I do what I’m supposed to do. Part of the current milieu at work is the concept of accountability to customers. If a customer pays you to do something, you’re supposed to do it. Even if the thing you’re being paid for is a complete waste of time. The truth is most of what we are tasked to do at the Labs these days is wasteful and nigh on useless. It’s the rule of the day, so we just chug along doing our useless work, collecting a regular paycheck, and following the rules.

Actually that isn’t completely true, I try to carve out time each day to write something. It’s a priority to myself that I exercise almost without fail. I have endeavored to make it a habit. It is actually several habits in one. For example, I write a personal journal each day to capture my innermost thoughts about my life and it’s an invaluable sounding board for myself. It is writing and that is good in and of it self, but it is completely private. Being private I don’t share it, or expect anyone else to read it. As such it doesn’t have to be clear or make sense to anyone, but me. On the other hand, the blog is intended to be read although that isn’t its purpose. It serves several purposes: the habit of writing, venting off steam, and working on concepts that are occupying my thoughts. All three things are very important to me, and I make them a priority for each and every day.

I wish that the same concept and discipline could be applied more broadly. Sadly it isn’t and most of us spend our time doing useless stuff. Most of what all of us are paid to work on is a complete waste of time. I end up spending most of my time communicating between groups of people, negotiating agreements and troubleshooting. I do far too little of the creative technical work that I love. Most of my writing is focused on raging against the forces of mediocrity eating away at any focus on doing something useful, important and high in quality. I recently came to the realization that the first ten years of my career were easily characterized by having my generally high expectations over-delivered upon at work; while the last fifteen years have seen my ever lowering expectations under-delivered upon in a seemingly endless search for the bottom. I wonder just how much lower my expectations need to drop to match the reality of today.

So, what might be more important than the crap they pay us to do? There are a lot of things in my personal life that qualify as more important than anything at work. This is rather obvious, but sad as the importance of work ought to put up a better fight. The problem with important work is that it is risky and difficult. We might fail at it, and failure is something we can’t bear to do these days. Ultimately we can’t muster the will to do important things because that would make our work real, consequential and meaningful, but also risky.

What’s important?

Judge a man by his questions rather than by his answers.

― Voltaire

The real world is important. Things in the real world are important. This is an important thing to keep in mind at all times with modeling and simulation. We are supposed to be modeling the real world for the purpose of solving real world problems. Too often in the programs I work on this seemingly obvious maxim gets lost. Sometimes it is completely absent from the modeling and simulation narrative. Its lack of presence is palpable in today’s efforts in high performance computing. All the energy is going into producing the “fastest” computers. The United States must have the fastest computer in the World, and if it doesn’t it is a calamity. The fact that this fastest computer will allow us to simulate reality better is a foregone conclusion.

This is a rather faulty assumption. Not just a little bit faulty, but deeply and completely flawed. This is true under a set of conditions that are increasingly under threat. If the model of reality is flawed, no computer, no matter how fast can rescue the model. A whole bunch of other activities can provide an equal or greater impact onto the efficiency of modeling than a faster computer. Moreover the title of fastest computer has less a less to do with having the fastest simulation. The benchmark that measures the fastest computer is becoming ever less relevant to measuring speed with simulations. So in summary, efforts geared toward the fastest computer are not very important. Nonetheless they are the priority for my customer.

So what do I do? Being a good little trooper, I work on the fastest computer stuff cause it pays the bills.

So what should we be doing instead?

I might suggest that we take a page from the success of the wider computing industry, and figure out how to make computing more ubiquitous in the daily life of a scientist. Well that’s already happened by fiat as scientists are functioning (just barely in many cases) members of society. So, let me be a bit more specific; make modeling and simulation part of the daily life of a scientist or engineer. This is something we as a community are failing at. Progress is being made by pure inertia, but that progress is dismal compared to what it could be.

The reason for the lack of progress is simple, high performance computing is still acting as if it were in the mainframe era. We still have the same sort of painful IT departments that typified that era. High performance computing is more Mad Men than Star Trek. The control of computing resources, the policy based use and the culture of specialization all contribute to this community-wide failing. We still relyupon centralized massive computing resources to be the principle delivery mechanism. Instead we should focus energy on getting computing for modeling and simulation to run seamlessly from the mobile computer all the way to the supercomputer without all the barriers we self-impose. We are doing far too little to simply put it at our collective
fingertips and make it a simple and ubiquitous part of what gets done.

The mobile computing is the key and mobile computing is not embraced by high performance computing. The challenge is to get the mobile computing to figure into the workflow of the average scientist. This is to do scientific work, not check Facebook, or email, or right swipe (they’ll continue to do this). The work should seamlessly move between the big iron and the mobile as appropriate. Remember if Linpac is to be believed my iPhone is as the Cray-2 was and certainly capable of doing some real work. If we can create this environment modeling and simulation would take off and truly meet its potential. Without the change it will continue to be a niche activity, and not fulfill its potential.

So what needs to change to accomplish this sort of transformation and success? First, the emphasis on big iron needs to diminish because it takes all the energy available. The trick is to make improvements in modeling, methods and algorithms and enable the more correct and efficient use of computing. Part of the modeling is the practice of modeling and simulation. Today we have generally poor practices, and a deep skewing of modeling by the tendency to adopt hero calculations. These calculations are focused at using the most massive computing available rather than be fitted to the purpose and suitable to assessment. The methods and algorithms can provide fundamental efficiency and scaling that will enable more to be done at the mobile level of computing.

People don’t want to buy a quarter-inch drill. They want a quarter-inch hole.

― Clayton M. Christensen

One of the modern mantras of business is the “disruptive innovation”. These innovations are sought out as the route to marketplace supremacy, creating a product that changes the business landscape and overwhelms the competition. These disruptive innovations are viewed as only being good things. I will challenge that assertion with the belief that massively parallel computing was a disruptive innovation (a common assertion), but not a positive one. It has been a negative innovation and successfully undermined the broader pipeline of innovation and discovery in high performance computing replacing with a slavish devotion to hardware as the route to progress.

To put increasingly outdated codes on these computers has taken all the available energy and robbed us of key innovations in models, methods and algorithms that would have been more beneficial than all the hardware gains. In addition these innovations would have created bona fide intellectual products that the current scientists would own, understand and benefit from mastery over. The computing hardware has become increasingly ill suited for scientific computing as well as being difficult to use. The next generation of computing promises to be worse in almost every regard. The focus on the massively parallel aspects of implementations has also starved focus on creating methods that are well suited to the modern CPU’s. As a result we utilize a tiny fraction of the available computing power from each CPU. In sum the current and future hardware focus has been destructive, wasteful and counter-productive. It may be laying waste to an entire generation of computational scientists.

The high performance computing community would be well advised to choose a different path towards progress before its too late. The path should acknowledge the holistic path to progress and the role of prioritizing work that impact the world beyond computing. It would be inspiring to take a page from the broader computing industry’s embrace of mobile computing as the route to ubiquitous computing that impacts the life of those who use it in immutable ways. I will take note that the death of Moore’s law for microprocessors and the assent of mobile computing happened in the same time frame. Worth thinking about, are these events correlated? We should realize that code is simply a computer recognizable expression of intellectual ideas. It should be understood if it is to be used for anything serious.

All of us should do some serious thought about what is important to work on. I think most of the current research emphasis in high performance computing is focusing on the trivial as the essential, and effectively trivializing the essential. A future where real progress and impact is made depends on changing this dynamic around.

‘Controversial’ as we all know, is often a euphemism for ‘interesting and intelligent.

― Kevin Smith