“Responsibility is a unique concept… You may share it with others, but your portion is not diminished. You may delegate it, but it is still with you… If responsibility is rightfully yours, no evasion, or ignorance, or passing the blame can shift the burden to someone else. Unless you can point your finger at the man who is responsible when something goes wrong, then you have never had anyone really responsible.” — Adm. Hyman Rickover

My father spent a career in the United States Army and then a second career at the Department of Energy, much of it working with nuclear weapons. When I was born, he was overseeing a battery of 8-inch howitzers armed with the W33 (or Mark 33 as my dad called it) projectile, which is a multi-kiloton-class nuclear weapon. He was stationed in Northern Greece, just south of Yugoslavia. My own conception was centered around the Cuban Missile Crisis. That event was our closest approach to Armageddon. Surviving it was the impetus for starting a family.

Later in his career, he was working with a Lance missile battalion for the Army in Germany. These were nuclear-tipped tactical weapons defending against a potential Soviet invasion of Western Europe. After he retired from the Army, he worked for the Department of Energy in a capacity to ensure the safety of our handling of the high explosives used in the nuclear weapons. He spent his life supporting the use and safety of these powerful weapons. It was noble service to a grateful nation. It shaped my own view of them.

“To be somebody or to do something. In life there is often a roll call. That’s when you will have to make a decision. To be or to do? Which way will you go?” — Col. John Boyd

His service on the use end of nuclear weapons had a deep impact on me. I joined the family business with a job at Los Alamos. I could see the production, the design, and the use of nuclear weapons vividly in my mind. When I began to work for the National Laboratories, I understood the mission, but I also held it as a sacred responsibility. I saw that fulfilled early in my career, but gradually, over time, after the turn of the 21st century, things began to change.

I am roughly six months into my retirement. As my more loyal readers might recognize, I left my career under rather distressing circumstances. I’ve been asked by friends, acquaintances, and family about the circumstances and the figure with which I write about that circumstance. I feel that it’s useful to articulate more about the foundation of these reasons. The same question from those who love me or those who know me. I hope the rest of you find it beneficial.

I am alarmist because the issues are alarming.

I spent nearly 40 years working at two national labs. Each has as its ostensible primary mission the care and performance of the nation’s nuclear weapons stockpile. This is a responsibility that I felt deeply and personally for most of my career. It is the advertised mission for both Los Alamos and Sandia National Laboratories, as well as Lawrence Livermore. These revered institutions no longer take this mission with the seriousness it deserves. During the Cold War, these Labs were founded and became amongst the best in the world. They no longer have that status. I’ve written about why.

“The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present and is gravely to be regarded.” — Dwight Eisenhower, Farewell Address (1961)

I have some deep personal reasons why it feels so important to me, perhaps more important than the average person. I believe that the circumstances of my departure speak volumes about the institutions that I worked for and their current status. Institutions of this sort, with other missions in the United States, are under continued assault and have lost the trust of the people. I believe the reasons for this are very clear. These institutions, including the ones I worked for, are only paying lip service to the very missions they are supposed to be serving. This is true at the nuclear weapons laboratories where I worked, and it is a fact that should dismay the people of our country. It is not limited to the nuclear weapons laboratories.

My wife worked for the last 20 years of her career at a university. The university’s primary mission is education, and what she saw there was similar lip service to the mission and a change in the priorities of leadership. This change paralleled what I saw at the National Laboratories. In both cases, the primary mission of the institutions had become an afterthought and almost something that the leadership simply supposed was being supported. Instead, the leadership was focused on a host of other things unrelated to the mission.

“In a fully developed bureaucracy there is nobody left with whom one can argue, to whom one can present grievances, on whom the pressures of power can be exerted… this is what the political jargon calls rule by Nobody.” — Hannah Arendt

Primarily, they all became obsessed with money and prestige, the prestige of power and the money that fed the institutions. Financial incentives had replaced the fundamental missions. The fundamental mission had become a second-rate, mildly unsupported activity at each of these places, universities and laboratories alike. The mission has become diluted by numerous objectives from the left and the right politically. All of them detract from the focus on the core mission.

I believe the same trends hold across our most important institutions, and also across our industries. No more so than our internet, social media, and artificial intelligence companies, which now only work to support profit and money objectives. They hold virtually no adherence to the benefits and well-being of the society that they exist within. Care for societal well-being is essential. Artificial intelligence perhaps serves as the modern equivalent of nuclear weapons. Both artificial intelligence and nuclear weapons have the power to destroy as well as create good. They both need a sense of sacred responsibility from those who are charged with their development and use. This is the recipe for disaster.

“We must not allow Big Science to destroy the tradition of scholarship.” — Alvin Weinberg

The leadership began to fail in this duty. Over time, they gradually began to work in a way that did not serve the best interests of the nuclear weapons mission. They were instead focused on their own professional success, which revolved around funding and securing money. They would seek programs that adequately funded their organizations, whether or not those programs actually served the nuclear weapons mission. Perhaps no national program epitomized this change more than the Exascale program. The advanced computing program at the labs after the Cold War ended began on the wrong foot. They took a course correction after a few years and operated well for about a decade. Then it fell into the broader national decline.

We have a process of annual certification. I worry that it has stumbled into a box-checking exercise. We go through the motions, but the option of raising a red flag is missing. The Lab Directors have no option other than approving the status without regard to the evidence. Meanwhile, the evidence becomes thinner and thinner with each passing year. We are nearly 34 years since the last test of a weapon. Hopefully they are aging well and still safe and secure. I don’t know one way or the other. My concern is that the quality of work is in free fall. Those with responsibility for them act in ways that give me great pause. Are we really doing the hard work? It seems like we are just going through the motions, pretending everything is okay. Given the constraints of not testing, we are not doing the hard work necessary for confidence. I’ve seen this clearly.

I am fairly sure that if a Lab Director failed to certify the stockpile, they would be fired in short order. It would be career-ending. They would lose millions of dollars in compensation for that act. So given this, do they even have the choice?

“It would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy.” — Richard Feynman, Appendix F, Rogers Commission Report (1986)

This finally came to a head in my own career, in the events that unfolded right before I retired. This is the origin of the vigor with which I have pursued truth-telling. I see leaders who are using the secrecy necessary for nuclear weapons to defend their own incompetence and irresponsible actions. No one is rewarded for acknowledging problems. There are always problems. The actions that were taken that led to my retirement were modest. I know of other circumstances that are far worse, far more dangerous, and far more damaging to our national security. Nonetheless, in principle, the things they did do not serve the nation well in its ability to have a sufficient and usable nuclear stockpile. The consequences of this are profoundly disturbing.

“Secrecy is a mode of regulation. In truth, it is the ultimate mode, for the citizen does not even know that he or she is being regulated.” — Daniel Patrick Moynihan, Secrecy: The American Experience (1998)

A few notes about the nature of responsibility. Nuclear weapons writ large are the responsibility of the President. The military, like my father, is responsible for the care and use of the existing weapons. Scientists and engineers like myself are responsible for the design and analysis of them. We use our talents and work to make sure the military and the Nation can rely upon them. This is integrated into the Lab Directors and the annual certification. Part of my retirement decision was related to the evidence that my talents would not be used. When my talents and experience were ignored, I knew that my own responsibilities would be neglected. I was wasting my precious life at the Lab. It was time to go.

If you take the events around my decision to retire into account, it informs the issue. The managers know they were in the wrong. They aren’t that stupid. Their actions were about power and control. Their actions were choosing money over responsibility. They are also reading the incentives we have put in place. You only do what you are explicitly paid to do. Bad news is to be squashed. Never admit fault or that a problem exists. You have the power to make the problem go away (until events overwhelm that). You have access to secrecy and information control to bury almost anything. Most of the time, no one will ever notice. The people above the managers are even more poorly motivated and less technically equipped. Most of those who lead us today are completely unfit to lead.

“The only people who should be allowed to govern countries with nuclear weapons are mothers, those who are still breast-feeding their babies.” — Harold Agnew, third director of Los Alamos

The question is: do I sit idly by and watch this happen, or do I continue to speak out? It became obvious to me before I retired that continued struggle within the labs was futile. It was clear that taking the responsible actions would only lead to me being punished. This is because it worked against the professional objectives of those that led me. I was wasting precious time in my life that could be spent enjoying my final years on Earth. Still, I hold on to the responsibilities that I felt so deeply and must speak out about this vital national interest. Perhaps my current actions are equally futile, but I must endeavor to do the right thing.

I only hope that someone will listen. We need reform and rebirth of these institutions. Nuclear weapons have the power to wipe out humanity. It would seem that AI may be as powerful and dangerous.

What motivates those responsible for them?

Today, it is not the well-being of humanity. Today, it is money. Personal wealth, power and enrichment are the drivers. This is a recipe for catastrophe. If we fail to recognize the warning signs, disaster awaits. The lights are all flashing red. There are many signs of impending danger. All these institutions are important. You might think nuclear weapons would be treated as a sacred responsibility. Today, they are not, and we all are in danger.

I hope someone stems this tide of incompetence before it drowns all of us.

“The glitter of nuclear weapons. It is irresistible if you come to them as a scientist… it is something that gives people an illusion of illimitable power, and it is, in some ways, responsible for all our troubles.” — Freeman Dyson, in The Day After Trinity (1981)

“If you do not work on an important problem, it’s unlikely you’ll do important work.” — Richard Hamming,

High-resolution methods are a bit stale these days (IMHO). I actually thought this 20 years ago, and little has changed. I’ve seen a little progress over time, but the deficiencies and lack of progress seem to be moving faster. Lots of high-order non-oscillatory methods, and DG without real improvement in capability. I have an explanation or two below. I’ve written about the reasons before. Lack of responsiveness to evidence or failing to get evidence. Sod’s shock tube results are a key bit of the problem. The practice of only applying qualitative comparison is pathological.

I also need to remember why I started doing V&V in the first place. My primal interest was in numerical methods and improving them. I saw test problems with quantitative results as a means to improving methods. You would measure results and use the measurement to decide where progress was being made. This seemed utterly logical and scientific. It was obvious. It is still not done today. Even with decades of expansion of V&V as a technical endeavor, evidence is poorly sought and even more poorly used. Progress in simply doing science for numerical methods is remarkably backwards.

We see multiple fields engaged at cross purposes in advancing the field. Mathematics is a key to progress. It is essential and also the root of the issue. Finding rigor and proof is amazing, as it is limited. Rigor comes with limits and assumptions that are restrictive. Real problems have discontinuities and chaotic-unstable results. Math is still rather limited there. This is a challenge, but also used to justify the lack of progress. For nonlinear, complex, and chaotic problems, there are still great gaps in our mathematical knowledge. It is work that is needed and necessary.

Physics and engineering tend to be somewhat application-focused on practical things. This leads to lots of corner-cutting. The lack of math rigor is then an excuse for this. Lots of things we do work well, but we don’t understand why. A subfield where I’ve actually worked shows this: implicit large eddy simulation. It remains an observation without a systematic explanation. I’ve contributed much of what passes as explanation. Too often we just need rules of thumb and utilization of common belief. There is not nearly enough effort in systematically understanding. This understanding is needed to reliably engineer things. Numerical methods remain too poorly understood.

V&V is thrust into these chasms. Verification interacts with the math deeply. It is largely an interface between math and numerics. It is supposed to be quantitative, and it is too rarely that. More frequently, it is only done in the ideal case. When things become difficult (Sod’s shock tube), we drop quantified results. Validation is the interaction with physics and engineering. Remarkably. the quantitative practice becomes even less common. In both cases the existing practice refuses to respond to evidence.

At least we are becoming more honest about how we ignore V&V. Recent trends simply reject V&V as a necessary part of simulation. It is simply too much bad news. It’s a bummer. Stagnation simply assures this.

This refusal is the origin of stagnation. CFD and other simulations tumble into witchcraft and wizardry. The path forward seems relatively obvious. It is difficult, but we’ve made it more so.

Things Aren’t Moving

“In science if you know what you are doing you should not be doing it. In engineering if you do not know what you are doing you should not be doing it.” — Richard Hamming,

I have a list of issues to explore in my head. This post is a means to document this. I’ve touched on most of this before, but it makes sense to put it all together in one place. It would make for a great set of PhD theses or research programs. If V&V were healthy, we could provide evidence of stagnation and progress against that. Fat chance of either happening today. But a guy can dream, can’t he?

Many of these things were already in my mind 20 years ago as I left Los Alamos. I had put effort into a number of these during my time at LANL. Several came into my consciousness at Sandia. The key is the lack of time and resources to tackle these there. The same lack was growing at LANL. I don’t think staying there would fix this. The issues are things I’ve spilled so much ink about. The nature of V&V resistance. The national obsession with exascale computing and big iron over mathematics and algorithms. The lack of risk-taking and trust in research today. The retreat of applied mathematics from practical importance.

As an example, I’ll mention an idea that my friend Vince had. It plays a role later in this essay. If you look at contact algorithms that are used in solid-mechanics codes, there are usually giant heaps of cyclomatic complexity, with a whole bunch of nested if-then-else statements. These are a nightmare for V&V, reproducibility, and computation in general. Vince wanted to study taking all of those out and replacing them with a smooth sigmoidal function that would make the code continuously differentiable.

He put in a research proposal at Sandia or modern day America that never had a chance. The reason? It was pointed at a sacred cow. Never mind that it was an incredibly good idea that met all sorts of requirements for programmatic impact. It simply was too different to support; it wasn’t HPC. There has been far too little emphasis on algorithms as a path to performance. In the process, progress has been lost, and performance has been hurt.

The major player in the long-term deficit in computational science is the multi-decade obsession with high-performance computing hardware. This hardware obsession has sapped the balance out of computational science and left a deficit. Almost everything I talk about here is dealing with the methods that, in one way or another, are at the foundation of many of our most important simulation tools. This is true for climate, astrophysics, nuclear weapons, nuclear reactors, clean energy, and on and on. If you’re generally solving any sort of multi-physics where hydrodynamics plays a major role, these methods are extremely important.

The cost of our failure to focus on these algorithms is probably most acutely measured in efficiency. The focus on high-performance computing is the most inefficient way to improve our simulation capacity. We’ve taken the same approach for artificial intelligence. Again, there are probably massive algorithmic efficiencies that should be explored with AI, but right now the focus is almost entirely on computing power.

“Numerical analysis is the study of algorithms for the problems of continuous mathematics.” — Nick Trefethen

So without further ado, let’s make a simple list of where I see some interesting issues to explore and improve upon:

1. What makes a real difference in accuracy on real problems

1a. What is the optimal mix of time and space accuracy in method design

2, What is the optimal mix of accuracy and efficiency on real problems

3. Computing nonlinearly stable time steps for nonlinear problems

4. Understanding nonlinear stability of solutions for space and time

4a. Exploring ideas around nonlinear stability for discretization

5. Combining adiabatic-entropic solutions with conservation. Why do solutions for very strong expansions not converge?

6. What are the right concepts for convergence in physical instabilities

7. Explaining implicit large eddy simulation’s effectiveness.

So, let’s dig into each.

1. What makes a real difference in accuracy on real problems

For real problems, accuracy is limited to first-order (or less) in almost every case. The lack of smoothness is the reason. High-order methods have value, but the great expense does not deliver commensurate with the effort. The question is what aspects of high-order methods deliver value in terms of accuracy. Evidence seems to point to some value being found with parts of high-order methods. There is definitely a huge leap from first to second-order, but the accuracy gains seem to saturate. How does one strike the balance? At the same time, high-order accuracy is more fragile and prone to failures.

To solve this issue, a number of things are needed. Rigorous guidance is a huge challenge for applied mathematics. Breakthroughs in math would be golden, but may not be possible. Instead of this guidance, we need to test methods and parts of discretization elements on real problems. Part of this is verification using analytical results. There is then the issue of how this transfers to validation problems. Right now, we are operating on a mix of blind faith and rules of thumb. We need to turn this toward science and real measurement. Examples abound, including shock wave problems and direct numerical simulations of turbulence. Neither is guided by genuine quantitative examination of results.

2. What is the optimal mix of accuracy and efficiency on real problems

This is a follow-on to the first issue. How does one balance accuracy and efficiency? The issue is that accuracy is expensive. Convergence rates are low. Accuracy is also found at the expense of robustness. This is a third issue to throw into the balance. Again, the vehicle is testing. It seems unlikely that mathematics adds as much as the first issue. The other major issue is the definition of efficiency. I’ve defined it as accuracy (fidelity) per unit cost. In a dull sense, given an accuracy of solution, the lowest cost is the most efficient. In the process, we can find the best ways to achieve accuracy (with robustness).

Practical Application Accuracy Is Essential

3. Computing nonlinearly stable time steps for nonlinear problems

For many problems the time step control is done via linearization. The dynamics of these initial value problem can contain much faster time scales. Hydrodynamics is a key example. The evolution of the problem can immediately include phenomena that is an order of magnitude faster. This is a relatively difficult problem to solve. One simple idea I had is to test a time step at the end of a cycle. Ask the question, was that time step stable given the dynamics at the end of the time step. If it was not stable, reject the time step and do it over with a smaller (stable) time step. This is simple and the main critique is the cost of storing another solution vector. It seems to me that the cost of an unstable calculation is vastly greater.

Wave speed estimation: importance, flaws, and consequences

“Newton said, ‘If I have seen further than others, it is because I’ve stood on the shoulders of giants.’ These days we stand on each other’s feet.” — Richard Hamming

4. Understanding nonlinear stability of solutions for space and time

I have written about this before in several posts.

Resurrecting High-Resolution Methods: Essentially TVD

Practical nonlinear stability considerations

A More Robust, Less Fragile Stability for Numerical Methods

Nonlinear methods, a key to modern modeling and simulation

Robustness is Stability, Stability is Robustness, Almost

5. Combining adiabatic-entropic solutions with conservation. Why solutions for very strong expansions do not converge?

I wrote a whole set of blog posts on these topics. They remain largely open issues.

Conservation. Is it Optional?

Entropy, vanishing viscosity, physically relevant solutions and ink

“The first principle is that you must not fool yourself — and you are the easiest person to fool.” — Richard Feynman

6. What are the right concepts for convergence in physical instabilities

If one has a physical instability like those found in turbulence or material mixing solutions do not converge normally. The concepts of convergence for initial value problems do not apply. There is a huge leap of faith and confidence that finer meshes lead to better results. Still convergence as a notion is mostly a leap of faith. Many classical hydro problems show more and more structure with resolution.

In a sense the notion is that given two solutions of the Euler equations, a “swirlier” calculation is a better calculation. This is seen in classical Kelvin-Helmholtz instabilities. Applied math has been noodling on this via solutions as distributions, but progress has been spotty. This is both difficult hard work, and hugely necessary. For many essential initial value problems, the massive calculations are leaps of faith. This includes all of direct numerical simulation of turbulence. That said, ideas in turbulence are the best hope here.

“Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.” — George Box

7. Explaining implicit large eddy simulation’s effectiveness.

This is finishing a project I gave up 20 years ago when I left Los Alamos. I had started looking at the modified equations for modern methods from MUSCL to WENO and many in between. Working closely with Len Margolin we identified some important parallels between LES modeling and the truncation error. In particular there is a term that shows up at second order with major significance. It comes from having a stable second-order method in conservation form. It looks much like the self-similarity model in LES. That model is notoriously unstable. With modern methods it appears and is selectively stabilized. I believe there is much more waiting to be found with ILES.

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.” — Max Planck

“The society which scorns excellence in plumbing because plumbing is a humble activity and tolerates shoddiness in philosophy because it is an exalted activity will have neither good plumbing nor good philosophy. Neither its pipes nor its theories will hold water.” — John W. Gardner

Sometimes something really interesting and thought-provoking happens when it is least expected. This happened last night at our local concert amphitheater. My wife and I had shown up early for our very good seats to see Rüfüs Du Sol, an Australian EDM band. Great show and performance that kept me up well past my bedtime.

I’m working on a technical piece, but between events in my life, such as putting our house together after the plumbing disasters. In addition, a trip to Mexico and then Europe in September. I’ve got writer’s block, and the ability to focus on that content has simply not availed itself. This event at the concert was interesting and contextually informative for me (and hopefully the reader).

As fate would have it, there was a strong to mildly severe thunderstorm bearing down on the amphitheater, and we had a rain delay. While milling about getting ready for the storm to strike, we took in the vendors. Finally, we were told to take shelter in place in the restrooms adjacent to the venue. I noticed a guy there who looked very familiar, but way out of context here. As the storm began to drop rain and vivid lightning on us, he walked towards me, and it turned out it was indeed who I thought it was, a guy I worked with at Sandia, Chris. He was there with his kids.

“Hope is not the conviction that something will turn out well, but the certainty that something makes sense, regardless of how it turns out.”— Václav Have

We piled up into the men’s room to avoid getting dumped on by the storm. Chris told me that he regularly read the blog, but he had some questions for me. The deepest question is: why are you doing this? Why not simply fade away and do other things? Why do you even care?

It’s a pretty fair question.

I told them that the work the labs do and the mission are extremely important to me. I’m concerned about the behavior, competence, and ethics of those chosen to lead that effort. People are using the relative secrecy and security there to do bad things. Those actually turn important work into meaningless activities. The Nation deserves better.

I still care about the research, the mission, and the work that’s done. He inquired about how bad things were, and I told him yes, it’s everything I’ve written, but actually a good bit worse. While most of the managers at the lab are good people who do a good job, there is a relatively large cadre of them that do terrible jobs and are unfit for leadership. It’s still something very important to me. Why should they get to do this? Moreover, the number of them that do the sorts of things that I saw has grown. It’s a real threat to the legitimacy and good work of this institution.

The storm began to subside. Chris made his way over to his kids, who he was attending the concert with, and we went our way. Still, I found his question to have some depth and genuine traction worth considering. Why not simply quietly go away and let the things dissipate and fade.

“We must always take sides. Neutrality helps the oppressor, never the victim. Silence encourages the tormentor, never the tormented.” — Elie Wiesel

In the end, the situation that I have dealt with is relatively unimportant and small potatoes. It fully demonstrates the irresponsibility of those entrusted. They decided to use the weapons of secrecy and the trust placed in them wrongly. They sought both retribution against me and a cover-up of the incompetence with which they were carrying out the work they were entrusted with. This should be offensive to everyone.

Rather than put some effort into dealing with genuine problems, they instead put all their effort into personal retribution and the cover-up of the issues that they failed to take action on. This alone should fill people with anger, given the importance of nuclear weapons. Worse yet, they expect to simply get away with it. By virtue of the authority, secrecy, and position they have been given. The reality is they probably will, and the problems identified competently will simply go unaddressed and unanswered for the foreseeable future.

“Responsibility is a unique concept: it can only reside and inhere in a single individual. You may share it with others, but your portion is not diminished. You may delegate it, but it is still with you. You may disclaim it, but you cannot divest yourself of it… Unless you can point your finger at the man who is responsible when something goes wrong, then you have never had anyone really responsible.” — Adm. Hyman G. Rickover

Getting back to the piece that I need to finish: the issues discussed there can’t even be envisioned to be pursued while the situation persists.

Here we have an example of relatively common and well-understood problems with code doing important work. Management, rather than producing something better, refuses to even address or solve the problems in the existing software. These problems are akin to patching drywall in a house or redoing the roofing. You don’t get anything new, and instead you’re simply fixing the thing that’s old with no change in functionality.

“The result… is not that the lies will now be accepted as truth, and the truth be defamed as lies, but that the sense by which we take our bearings in the real world — and the category of truth versus falsehood is among the mental means to this end — is being destroyed.” – Hannah Arendt

For the most part, this isn’t even doing research. This is simply maintenance. Yet management stands utterly and completely opposed to this preventive maintenance. We can’t even conceive of doing anything more substantial than that. Sandia could be a great institution again. With the current leadership and the tolerance for their bad behavior, this won’t happen. It will continue to decline and fall into disrepute as these leaders will earn the sort of status they deserve and bring the institution down with it.

The biggest part of the reason is this betrayal of trust that the behavior showed people given positions of authority and trust, then misusing those positions to simply stay in power and actually subvert the work of the laboratory. This should make everyone incandescent with rage. Yet, it’s sort of the banal and ever-present corruption across society that’s now become commonplace and almost expected in those in positions of authority.

“The best lack all conviction, while the worst / Are full of passionate intensity.” — W. B. Yeats

Whether dealing with monkeys, rats, or human beings, it is hardly controversial to state that most organisms seek information concerning what activities are rewarded, and then seek to do (or at least pretend to do) those things, often to the virtual exclusion of activities not rewarded. Steven Kerr, “On the Folly of Rewarding A, While Hoping for B” (1975)

I am a progressive through and through, in the plainest sense of the word: I want things to get better, and I am interested in whatever might make them so. That runs through every part of my life. It is true in science and engineering, it is true socially, and it is true in what I read and watch. I am always after something new, always trying something, always looking for where more progress can be made and things can be done better.

This temperament does not suit the world we live in. I chafe against those who defend the status quo. It certainly did me no good at Sandia. That was a conformist institution where the tried-and-true was enforced by the system itself. Progress was not the point. Results were the point, and there is an incentive structure wired to produce precisely the results we see.

Some of what it produces is genuinely good. Promises get delivered. Schedules get met. Large, complicated things get built by people who do what they are told, and there is real value in an organization that can do that reliably. Financial performance serves as both the fuel and the scorecard. The whole apparatus points people toward alignment: figure out what is wanted, produce it, be seen producing it.

The trouble is that this same structure is aimed squarely against better outcomes in the future. Everything is front-loaded into the present. The current results are purchased at the expense of the organization that produced them. We systemically avoid creating the future we could have. Age-old problems persist without solution, and power supports the status quo.

We live in a time of societal upheaval. This is at a scale I’ve never seen in my more than 60 years on Earth. On the one hand, we have a major movement looking to recapture a certain mythological greatness in society. On the other, we see personal rights and liberties rolled back as society loses any space for empathy. We stand on the brink of chaos and disaster. Our system is failing both climate change and artificial intelligence. Threats and opportunities are equally failing to be met.

I think there’s a root cause for all this at the same time. At the same time, AI is dawning. Some people say we are in the foothills of the AI singularity. This could be a golden age, or it could be the end of humanity. We don’t know. The risk-reward of this technology is massive. I feel like it’s unlikely that it will not be a huge force moving forward. The question is how we will meet change. Signs today are not good.

The question is: what are the incentives and motivations driving all these changes? How do we navigate it in a way that leads to the best results?

Incentives and Priorities

How we spend our days is, of course, how we spend our lives. Annie Dillard, The Writing Life (1989)

We all respond to incentives. Which incentives we respond to is an expression of what we actually value, whatever we may say we value. Take my own case: I chose my family and my obligations to them over professional success, and I would make the trade again.

By most measures I have done reasonably well as a scientist. My h-index is around 40, and I remain avidly interested in the work itself. Had I stayed at Los Alamos, I am fairly confident that career would have gone further and the h-index would sit closer to 50 or 60. I would have written more consequential papers. I would have done better work, for the simple reason that I would have been surrounded by more first-rate minds than I found at Sandia. More properly, first-rate minds in a culture that values progress over compliance.

That is not a claim that Sandia lacks capable people. It has plenty. At least first-rate minds oriented toward meaningful research. Sandia is a system that supports the status quo. The problem is that its incentives and its management do not encourage those people to make progress. The system runs counter to progress. I arrived at Sandia with a publication record that already exceeded that of people well above me in the hierarchy, including the ones Sandia formally designates as senior scientists. What I never did was respond to the incentives Sandia treated as important. That is the path of success.

I was responding to a different set of priorities: my marriage, my life outside work, my children. Those are what governed my decisions, and they governed them consistently. I am comfortable with the price I paid.

The Incentives Make People Worse

The line separating good and evil passes not through states, nor between classes, nor between political parties either, but right through every human heart. During the life of any heart this line keeps changing place; sometimes it is squeezed one way by exuberant evil and sometimes it shifts to allow enough space for good to flourish. Aleksandr Solzhenitsyn, The Gulag Archipelago

What I have come to believe is this. At Sandia, the path to success runs through management. That path actively corrodes the ethics and the conduct of the people who take it. Not everyone, but a significant plurality of managers take the bad path.

Most of the people I have written about over the past six months, the ones who did me real damage or did plainly unethical things, are fundamentally decent human beings. I want to be clear that I mean that, and that it is not a rhetorical concession. The problem is not that Sandia collects bad people (I am quite sure that Los Alamos has the same malady). The problem is that its incentives take ordinary people and make them worse, reliably and by design, and reward them for it while it happens.

That is a far more disturbing finding than villainy would be. A villain is a local problem. A system that manufactures its own villains out of decent material is a structural one. Villainy becomes a feature instead of a bug. It will keep producing them after every individual you could name has retired.

I should say plainly that I believe the incentives at Los Alamos have degraded too. Much of the same thing has been happening to management there. It has the same malady as Sandia. Had I stayed, I would have watched a laboratory I cared about go the same way, and it would have hurt more, not less.

My heart never belonged to Sandia. It did belong to Los Alamos, and I would have mourned what I have watched happen to it. What Los Alamos still had, and what I would have held onto, was scientific momentum. I traded that momentum away. I traded it deliberately, for other things, and I am not sorry.

So what are these incentives, exactly? Why do we maintain arrangements that produce such consistently bad outcomes for the people living inside them? Why do they draw out the worse version of a person rather than the better one? What does it say about a society that this is what its institutions reward?

I am certain of one thing. The incentives now operating across our society are bad, and left alone, they will be our undoing. They are not a nuisance to be managed. They are a mechanism for producing catastrophe, running continuously, and nobody has to intend the outcome for it to arrive. We need to recognize them and act to resist their call.

What AI Could Be

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. J. C. R. Licklider, “Man-Computer Symbiosis” (1960)

My progressive streak shows up in how I think about artificial intelligence, and specifically about large language models and what they are actually good for. They are extremely capable, flexible digital assistants. They raise both productivity and the quality of the work. The potential for societal good is extreme.

That is the case the companies should be making, and largely are not. These tools can take the burden off the tedious, pedantic parts of a job while letting the work itself come out better and with less strain. They can free us from a great deal of the bureaucratic bullshit that consumes too much of a working life, and release creative energy into the parts that actually matter.

AI can accelerate searching, plotting, transforming, and preparing the ground. The time for useful thinking shrank during my career. More time was spent engaged in meaningless bullshit. Over time, the space for creativity was swallowed by tedium and box-checking of little or no value. Work became a paycheck, instead of a useful forward-looking endeavor to make the World better. I stopped solving big problems to feed the bureaucratic beast.

Which brings the argument back to incentives, because that tedium was not an accident. Managers showed no appetite for doing high-quality work. They wanted to be able to say the work was high quality, and the saying is far cheaper than the doing. Work became producing evidence of progress instead of actual progress. Actual progress would find problems, and managers are rewarded for not having problems.

I will come back to AI at the end, because what happens to it depends entirely on what follows.

Incentives of Power and Greed

This disposition to admire, and almost to worship, the rich and the powerful, and to despise, or, at least, to neglect persons of poor and mean condition, though necessary both to establish and to maintain the distinction of ranks and the order of society, is, at the same time, the great and most universal cause of the corruption of our moral sentiments. Adam Smith, The Theory of Moral Sentiments (1759)

Today, we value money above all else. In the world as it now runs, money is power and money is success. Politics is power too, and our industrial might as a nation lines up behind whoever maximizes the money. The power that comes with it justifies the emphasis.

The result is a system in which money buys politics and politics protects money. The two amplify each other like a flywheel. The whole arrangement tilts toward the extremes, and those extremes end up empowering a base committed to keeping things as they are. Progress of any kind then reads as a threat. The status quo protects those in power.

The one place progress is still prized is where it produces something salable, and therefore more wealth for the institution producing it. But the progress at the base, the fundamental work that is seed corn for those products, is being starved. Look at the system as it stands, and every part of it is tilted toward maximizing current returns while starving the future. That is why our institutions are in decline. I worked for institutions that should be relentlessly future-focused.

Trust in this society sits at a historic low. People intuit the objectives of greed. They see leaders who only care about themselves and their power. The price for this is massive. Institutions and trust are both preconditions for progress. Without them, a vibrant future is not coming unless something changes to heal them.

The Question Is Balance

To allow the market mechanism to be sole director of the fate of human beings and their natural environment, indeed, even of the amount and use of purchasing power, would result in the demolition of society. Karl Polanyi, The Great Transformation (1944)

This brings me to a point I have made often. Money has its place. Greed has its place. The exercise of power has its place. Every one of those places has to be bounded by some other force. Money, greed, and power must be leavened by duty, ethics, and purpose.

The status quo is tried, true, and comfortable. Progress is what makes a society better. The task is to hold the two in tension. Too much deference to the status quo becomes an insistence on keeping things long after they have been shown not to work. Progress is why we live longer than our grandparents and why we have all manner of things worth having.

What is missing is the balance that would make new things a net good. Balance is what lets you improve on the status quo without demolishing it. Balance is what keeps money from being made at society’s expense. On every one of those counts the current system is missing the ingredient that produces stability.

So we get a society making progress in ways that damage people and unsettle them. The forces of money and power ensure that nothing interrupts it, so long as the money keeps flowing toward those who already hold power. The system does not correct. The imbalance grows, and with it comes danger, chaos, and the widening sense that things are out of control.

Rational Actors

The law, in its majestic equality, forbids rich and poor alike to sleep under bridges, to beg in the streets, and to steal loaves of bread. Anatole France, Le Lys Rouge (1894)

People respond to incentives. If you look at almost anything happening in society, from soup to nuts, you can usually work backward from the incentives to the behavior. Take someone like Elon Musk, enormously rich, at least on paper, who has converted that wealth into power and influence. The entire arc of his rise can be reconstructed from what the system rewards. Nothing about it requires a special theory to explain.

Increasingly, those incentives run against society as a whole. You can see it in the scale of income and wealth inequality, and you can see it in the social upheaval that great fortunes are now able to purchase. Wealth at that magnitude confers something close to immunity. Worse yet, the aims of the powerful become divorced from the good of society. Now this is where we are. The benefits for the rich and powerful are a pox on the rest of us.

That immunity is the part that should worry us most, because it has made the justice system profoundly uneven. If you are poor, the rules are applied to you fully and often unfairly. If you are rich, you can commit almost any crime and expect to be handled rather than prosecuted. The Epstein class is the prime example of this. The President is another. We have arrived at a legal system in which financial power is the power to determine which rules apply to you.

The sick part is that this is not a malfunction. It is the rational product of the incentives we have built. We are getting exactly the results we engineered. We are engineering societal catastrophe.

Outcomes Flow

Once incentives are in place, outcomes follow. Look at our institutions, and the incentives are plainly misplaced. The focus on money has produced financial outcomes and driven out quality, truth, and ethics. Those three cost money and are not economically efficient. If you will not pay for them, you do not get them.

When I look at what happened to me in my last year, it makes complete sense. I was making decisions on the basis of principles. The questions I was asking were these:

How do we make progress?

What is the right thing to do?

What does the mission need?

Every one of those questions was wrong-headed, given the system I was in. The operative questions were different:

How do I make the boss look good?

What do we have money to work on?

What is best for the reputation of the Lab?

Those were the incentives actually in force, and I acted against all three. The managers had already settled on what made them look good. There was no money to fix the code. The standing instruction was to defend the current work and the current results at any cost. And underneath it all sat the real calculation: nobody with the knowledge to check is looking at our results anyway, so bullshitting through them is the cheapest way to operate. We are quality because we say we are. No progress is needed. We are already as good as we need to be.

I was not punished for being wrong. I was punished for answering the wrong questions. I followed the wrong incentives.

We Get What We Value

Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. Cory Doctorow, “Tiktok’s enshittification,” Pluralistic (2023)

The most visible place to watch values operate is the internet. Social media is enshittifying, and it is doing so because people are responding rationally to what pays.

Take Meta. To maximize profit it mines your data in order to sell you things. It runs over anyone in its way. It carries no social responsibility whatsoever. If the product corrodes society, that is not a cost on its books. If it damages children and produces terrible outcomes, so be it. Profit rules, and these outcomes are the choice we have made on its behalf by declining to make any other.

Doctorow named the process enshittification: the product gets worse and worse, deliberately, because worse is what pays. Google, Facebook, and Amazon all decline along the same curve for the same reason.

The same incentives operate at the national laboratories, and they produce the same decay. My own situation at the Lab is a small instance of it. The Labs are enshittifying too. They are getting systematically worse, and that decline is not drift or bad luck. It is the direct product of what the system rewards. Nobody set out to make them worse. Everyone responded to what was paying, and worse is what the financial arithmetic produced.

If something cannot go on forever, it will stop. Herbert Stein

An arrangement that consumes its own foundations does eventually run out of foundation. That is the only consolation on offer, and it is a cold one, because the question is not whether it stops. The question is how much is left when it does, and whether anyone remembers how to build the thing back.

Now Run AI Through It

All of this is why the prospect of AI entering this incentive system is terrifying.

Everything I said earlier about what these tools could be assumed they would be built and sold by people who wanted them to be good. Nothing in the present arrangement rewards that. The enshittification that has already hollowed out most of what was valuable about the internet. It is coming for AI on exactly the same logic, and for exactly the same reason: worse pays.

An enshittified AI is a different order of problem from an enshittified social network. It will destroy jobs and damage society with far more reach than anything Meta managed. Absent a change to the incentives, that is where this goes. Not because anyone chose it, but because nobody had to.

What Would Have to Change

The mindset or paradigm out of which the system, its goals, power structure, rules, its culture, arises. Donella Meadows, on the highest-leverage place to intervene in a system, “Leverage Points” (1999)

So what the fuck do we do about this?

You do not fix a system by adjusting its parameters. You fix it by changing the paradigm the whole thing grows out of. Every other intervention is downstream of what a society has decided counts.

For us, that means displacing money from the center. Not abolishing it, and not pretending it does not matter, but removing it from its current position as the measure of success, the arbiter of decisions, and the source of power both personal and political. That value system is what generates the incentives, and the incentives are what generate the priorities. Attack anything further downstream and you are rearranging deck chairs.

What has to move to the center instead is not mysterious. Quality. Outcomes that are actually good for people. Ethics. Those are the things that currently cost money and return nothing on the scorecard, which is precisely why they are the first to go.

My honest expectation is bleaker than my hope. What I expect is that the current incentives will do enormous damage, that the damage will be plain to everyone, and that change will come only after it. That is a grim thing to hope for and I know it. I am effectively hoping the failure arrives fast enough and legibly enough to teach, rather than slowly enough to be normalized. I think it is quite likely this system does damage that outlasts the rest of my life.

But the alternative is worth stating plainly, because it is not a fantasy. Build AI inside a better set of incentives and the possibilities are genuinely enormous, and enormously good. That version may not do much for the wealth of the tech oligarchs. It would do a great deal for everyone else.

That is the project. Not stopping the technology, which will not happen and should not. Changing what we pay it to do.

When a measure becomes a target, it ceases to be a good measure. Marilyn Strathern, formulating Goodhart’s Law

Let me be precise about that, because the imprecise version is wrong. Badly done V&V is no trouble at all for most managers. Well-done V&V is a nuisance for management. The reason is that well-done V&V finds problems and changes plans. Badly done V&V just lets plans and programs proceed unchanged.

The reason is simple. V&V finds problems and shortcomings in programmatic work. Those problems are anathema to a plan. They require extra effort to reach the level of success that managers have already described to someone above them. Management has over-promised, and V&V is the instrument that measures the shortfall. It puts a number on the distance between what was promised and what was delivered. These days, the funding is attached to the over-promise. Thus, V&V is a threat to funding. That threat needs to be neutralized.

The Checkbox

The result is that V&V becomes a desired checkbox for quality while the managers who want the checkbox decline to pay for the quality. That is a corrosive position to occupy. Interest in doing the work well steadily declines, and what survives is a stamp that certifies poorly done work as acceptable. V&V simply becomes a vehicle to justify the management’s BS.

The conclusion I have come to is that management needs approval for a great many things it has already done. It has committed to aggressive schedules across many projects. It has promised that modeling and simulation would return something large. On top of all that, it has spent a fuck-ton of money on big, flashy, expensive computers. All of it has to pay off, and it has to pay off visibly. Real V&V would show that the return is smaller than promised. Fake V&V supports their chosen narrative.

V&V done well questions every piece of that. What it would find is that the aggressive schedules, the enormous budgets, and the effort have gone into a variety of the wrong places. Managers would need to adjust and fix things. They would need to admit that the errors and predictions are not as good as expected. Fault would need to be admitted. These days, those are all career-limiting actions.

With four parameters I can fit an elephant, and with five I can make him wiggle his trunk. John von Neumann, as recounted by Freeman Dyson

This began to dawn on me when I realized that in the programs at Sandia, virtually no validation was actually being done. They would say there was validation, but this was an illusion. There was preliminary validation. Pre-shot simulations were compared to post-shot results. Then the model was modified and calibrated until it matched those results better. Usually the tests had no defensible error bars, so what we held was a set of point values from a single experiment, often not repeatable, set against a calculation.

That is not validation. That is curve fitting with a schedule attached. Since they had calibrated with the only test data available, validation becomes impossible. What we are left with is modeling in isolation, where the model form error cannot be determined. We spent an enormous sum on simulation technology, then systematically shot ourselves in the foot.

Burying the Evidence

Unless the individual truly responsible can be identified when something goes wrong, no one has really been responsible. Admiral Hyman G. Rickover

Then there is the example I have spent a great deal of ink on. Straightforward verification was performed on a code used for important problems, against analytical results that are highly relevant to those problems. The results were negative. They indicated real trouble with the code. Management’s response was to bury them, misclassifying a record to keep it away from prying eyes. This is corrupt. It is mismanagement. Worse, it guarantees that a problem in an important code will never be addressed, or even acknowledged. Instead, everyone is told to look away.

The same thing happens with validation results. Those are often properly classified at a high level and legitimately kept from public view, but the practice of burial is common and pernicious regardless. In substance it is the larger problem, because it leaves us with a very poor idea of how faithful our physical models are to the systems we use them to analyze and certify.

The human understanding when it has once adopted an opinion draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate. – Francis Bacon

The real victim of all this is quality, and science.

I have written repeatedly that properly practiced V&V is simply the scientific method applied to computation. When you meet this kind of obstinacy, this inability to deal with a problem once it has been named is the crime. What suffers is the quality of the results and any warranted confidence in them. The opportunity to mount a focused scientific effort to improve the models, the codes, and the methods is swept away with the inconvenient finding.

The purpose of V&V is to provide evidence that the work is good enough, and that the money bought something real. By massaging and burying results as they see fit, management short-circuits the mechanism entirely. They install a low-quality result as the standard rather than pursue excellence.

What these trends portend is institutional decay. Rather than engines of innovation and progress, the labs have become engines of the status quo: engines of “good enough.” Instead, they are engines of mistakes buried under (improperly) applied classification. In my view, the root cause is the preeminence of money as the measure of all things. Money has become the stand-in for quality and success. It is in the place of any real measure of technical merit or scientific progress. V&V is merely the place where the trend becomes most visible, and the place that suffers first. All the follow-on work that V&V should spearhead is abandoned with it, killed in the cradle.

Management simply wants to broadcast that everything it does is high quality and good enough right now, and that it is succeeding. There is an absolute inability on its part to mount any rational response to a genuine problem with the work.

Exascale: The Hero Calculations and No V&V

There is nothing so useless as doing efficiently that which should not be done at all. – Peter Drucker

A good share of the blame belongs to the Exascale computing program. One of its most pathological features was a general lack of support for V&V. There was no V&V element in the program at all. A small piece survived inside the NNSA portion, and that was a holdover from ASC. The reason is not mysterious. V&V would have delivered bad news about Exascale, and that should have been obvious from the outset.

Computing power was never the long pole for improving accuracy. The problem is, and was, the physical models themselves. Their limited accuracy, together with the intrinsic statistical variability of many phenomena that experiments generally fail to capture, is the actual source of the largest errors. V&V would have confirmed this and shown the level of investment in hardware to be foolhardy.

Anyone can build a fast CPU. The trick is to build a fast system. – Seymour Cray

Some studies could get you punished. I remember using the computers at Oak Ridge National Laboratory, where you would be penalized if anyone discovered you were running UQ studies on their big machines. UQ via sampling was trivial to scale up to the entire machine.

The reason was that the whole program had staked its claimed impact on the power of those machines to perform massive single-purpose hero calculations. That posture ignored how inefficient added resolution is at improving accuracy when the order of convergence is low, typically first order or considerably worse, and worse still once you reach phenomenology like turbulence, where all the statistical variability lives. No single hero calculation is science. It is a stunt. You can do them, but they have no error bars. The meta-belief was that the single calculation was “perfect”. This is like the DNS myth. The hero calculation is as good as an experiment.

A well-done V&V study would have made this plain and cast substantial doubt on the efficacy of the investment. Instead, politicians and managers were unified in their desire to promote success commensurate with the money, and to supply reasons why the money had been well spent. Flashy computing and unvalidated results were what they wanted to show. Under no circumstances should real science look under the hood and assess the quality of any of it. That could only produce bad news and demonstrate that the effort had been misdirected.

The Real Problem Is Balance

You can see the computer age everywhere but in the productivity statistics. – Robert Solow

The real problem is the lack of balance.

Investment in high-performance computing hardware, and in the software to use it, is money well spent. That is not the complaint. The problem with Exascale was that it was utterly unbalanced. Alongside the absence of V&V, there was no serious focus on methods or algorithms, and none on physical models or experiments.

Modeling and simulation is an integration across the whole of science. For it to flourish, every part of the scientific enterprise feeding it has to be healthy and pushing the state of the art forward. The program ignored the fact that methods and algorithms have delivered as much improvement in effective computing capability as the hardware has, and arguably more. Instead the focus on hardware became pathological.

We are now watching the same pathology repeat with artificial intelligence: enormous investment in data centers and computing hardware, with the same inattention to everything else. It is inflating an economic bubble that will very likely end in pain. That pain is almost entirely a product of the imbalance we saw in scientific computing, inherited wholesale by the work on AI. It will end up an economic own goal, completely unnecessary, and it will do immense damage.

The Canary

Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one. – Charles Mackay

V&V should be supplying detailed feedback and evidence to the scientific enterprise. AI too. It is the fulcrum of a balanced program that self-examines.

Instead it has become the canary in the coal mine.

Its neglect, and the generally poor quality of the practice that remains, is a harbinger of much larger problems: the absence of balance, and the inability to react flexibly to where the real bottlenecks in efficiency, efficacy, and quality actually sit in computational science.

It tells you something about how the AI effort has unfolded, too, and where the dangers in that activity are going to be found. We are probably too far in to avoid the collapse. One hopes we learn our lesson. Recent history seeds doubt.

“Science is a way of thinking much more than it is a body of knowledge.”– Carl Sagan

When someone shows you who they are, believe them the first time. — Maya Angelou

I’m going to start by doing something remarkably counter-cultural. I’m going to take responsibility for something.

In the run-up to my retirement, I trusted. I trusted my management even after I had accumulated a great deal of evidence that the trust was misplaced. I paid for that. It was my mistake, and I should have known better.

Trust is not the error. The world works better when people extend it. It is faster, cheaper, with less friction and less misery. The error is extending it to people who have already shown you, repeatedly, what they are. That isn’t generosity. That’s naivety, and mine was stunning.

Trust Broken

Trust and integrity are precious resources, easily squandered, hard to regain. They can thrive only on a foundation of respect for veracity. — Sissela Bok

To unravel my own naivety, go back six or seven years. Two things were happening then that bear on it.

The first was a large problem at the lab. It was expensive, it was embarrassing, and it was serious enough to draw a congressional investigation. The second thing I learned only later: that problem ultimately cost the lab director, Steven Younger, his job.

Around the same time, I fell into a vigorous discussion among senior staff about the research environment and its systemic failures. Forty or fifty of us worked up a white paper on how to fix it. A subset of us then agreed to meet with the lab director, and Vice President of Research to discuss the critique and suggest fixes. We met in the Lab director’s conference room, twelve senior technical staff and these two executives. We each spoke. We laid out the problems as we saw them. Then we waited for a response.

What we got was among the most unprofessional things I witnessed in my time at the labs. Dr. Younger began with a series of boasts about his own prowess as a researcher. He then turned on us for having the audacity to criticize the lab and its environment. He was belittling. He was angry. He was disrespectful to every person in that room. His arrogance was over the top and profound. Some of what the staff had raised was evidently more than he could take, and he was done pretending otherwise. No wonder he was sacked soon after.

The irony arrived later, when I understood that he was on his way out over the very problem we had come to talk about. His conduct in that room was not a departure from the culture that produced the failure. It was a demonstration of it.

Broken Institutional Trust

Wooden-headedness, the source of self-deception, is a factor that plays a remarkably large role in government. It consists in assessing a situation in terms of preconceived fixed notions while ignoring or rejecting any contrary signs. — Barbara Tuchman

The congressional investigation produced a report. The eagle-eyed reader will already have guessed where that report lives now: behind an official-use-only marking, out of view.

It contains a plethora of scapegoats and almost nothing identifying the actual source of the problem. The actual source was simple and it was obvious. The lab failed to put subject-matter experts into peer review early in the project. Those experts would have seen the problem and named it while it was still small. That was precisely why they were kept out, their review would have slowed things down and raised issues nobody wanted raised.

So the problem grew. It became far more expensive and far more embarrassing than early scrutiny would ever have been. The maxim of finding defects early was violated. The cost was massive.

The mechanism is the same one Dr. Younger displayed in that conference room: the reflex is to make the problem go away rather than to solve it. Hiding is cheaper than fixing. I had seen the identical reflex years earlier, in the same program, from Dr. Cilantro. He took exception to criticism of his staff not over the quality of their work, but over their systematic over-classification of it and the belligerence with which they met a review and any serious question. The same arrogance, in a different room, wearing a different face.

The other half of this unsavory sandwich is what happened to me more recently, with the report I flagged as problematic.

I am a subject-matter expert in the full scope of what that report covered. I was avoided. I was avoided in the earlier work, where the goal was to check the verification box and move on. When the more recent report identified real problems, there was no willingness to face them. There was only the old reflex again: bury it, and slap a classification on it to make the burial official.

Here is where my naivety deserves examination.

In every one of these episodes, I assumed the importance of the mission would win. I believed that national security — that nuclear weapons, of all things would finally force managers toward the right decision, whatever their inclinations. Surely the stakes were too high for anything else.

Sadly, that was foolish of me.

Reality is that which, when you stop believing in it, doesn’t go away. — Philip K. Dick

The stakes only work as a constraint if someone is still checking the answer against the world. As the distance from physical reality has grown, so has management’s confidence that reality is theirs to define. Nothing intercedes anymore. Nothing arrives to stop a stupid decision that no objective fact supports. You can simply declare the outcome and move on.

The Buck Stops Somewhere Else

The President — whoever he is — has to decide. He can’t pass the buck to anybody. No one else can do the deciding for him. That’s his job. — Harry S. Truman

Something more pernicious is happening in the wider society, and the labs are only a local instance of it. The examples set at the highest levels are of people — usually men — who take no responsibility for bad decisions or for the damage those decisions cause.

The debacle around the reflecting pool is as clear a case as you could ask for. Far too much money went to incompetent contractors who did poor work. Rather than own the decision to hire them, the president blamed other people. He invented people, and invented events, offered to explain away work that was simply bad.

Truman’s maxim has been revised. The buck stops somewhere else. I have witnessed firsthand, that revision reaches well beyond the Oval Office. It has permeated everything.

Trust Versus the Untrustworthy

Every bureaucracy seeks to increase the superiority of the professionally informed by keeping their knowledge and intentions secret. Bureaucratic administration always tends to be an administration of “secret sessions”: in so far as it can, it hides its knowledge and action from criticism. — Max Weber

You cannot work for people you cannot trust. Once trust is broken it is very hard to repair. So I left the people I could no longer trust.

More than that, I had stopped believing in the social contract between staff and management at Sandia. Some managers maintain a healthy, respectful relationship with their people, and those managers are a pleasure to work for. They may even be the majority. The problem is the substantial minority who operate on a different premise: that trust runs one direction only. They are in charge. Your job is to do what you are told. They do not worry about the truth; they decide what the truth will be.

Note the asymmetry. Staff are expected to trust that management is giving them the right direction. Management is expected to trust no one and explain nothing. Sandia too often operates without transparency even internally, and from the outside it operates without visibility at all.

The contrast with Los Alamos is instructive. Los Alamos is famous. It earned that fame with a starring role in the making of the atomic bomb, and Oppenheimer recently pushed it back into public view. Sandia has none of that notoriety, and revels in not having it. Invisibility is not an accident there. It is a preference.

I came to understand that this is exactly what the institution wants. Sandia does not want the spotlight, and its employees are encouraged to be invisible along with it. Management uses that darkness. It is what lets an organization cost the government an extra billion dollars, mislead the public about the cause, and drift quietly back into obscurity.

Measured against that, mislabeling a report as export-controlled is laughably small stakes. That is rather the point. Each episode that ends without consequence teaches management that the approach works. There is no penalty for unethical behavior. There is, in practice, a reward.

Coda

Show me the incentive and I will show you the outcome. — Charlie Munger

This is what an incentive structure tilted toward unethical behavior produces. It stops being a defect and becomes a feature. The absence of ethics turns into a vehicle for advancement, and competence becomes optional — worse than optional, because competence is expensive and difficult. Why pay for it? You can message success instead and skip the hard part entirely.

That is a recipe for institutional decline. No one is responsible for it.

Everyone is.

If you do not work on an important problem, it’s unlikely you’ll do important work.–Richard Hamming

An answer to an Asshole’s Question

I talked to my mentor, and he passed along a comment Cilantro had made about me:

“Why did we hire him in the first place?”

This was the prelude to them fucking me. Cilantro was my director, so I was fucked. They were going to come after me. I had no defense except surrender. Discipline. Shitty performance review. Cut off from anything good. It worked, and I was gone.

The timing was perfect. After Covid, and some modestly distressing personal health issues, my activation energy was incredibly low. My father was in hospice. I had watched him decline and suffer for over a year. Wasting my life working for incompetent, unethical assholes was an easy choice. I had checked my finances. I have two pensions too. I could retire and be comfortable.

It turns out Cilantro was asking a good question. Why was I there? Why had I come there in the first place?

The answer is twofold. My hire seemed a no-brainer back in 2007. The code I was working with, ALEGRA, was having problems. I had deep knowledge of hydrocodes from my years at Los Alamos. I knew how to write them, and I knew the details and foundations of hydrocode methods and applications. I was also a V&V expert. I had a great research record. I had been a line manager and a program manager. They were thrilled to have me.

All of those reasons were wishful thinking. It was a terrible fit. As difficult as it is to admit, Cilantro had a point. Why did they hire me? What use was I to that place?

I hired into the Computer Research Center, one of the most externally focused places at Sandia. It smells a little like Los Alamos. The difference is that this center could give two fucks about the Lab mission, and I was going into the most mission-focused department in it. My values were to care deeply about the Lab’s mission. Contributing to it was paramount. Thus, in the Computer Research Center that department was on the outside looking in. The rest of the Center was on the outside looking out.

If you gave a fuck about the mission, you worked in the Engineering Sciences Center. In fifteen years I would end up there. The problem is that Engineering Sciences is long on engineering and short on science. It bridges to the weapons program, where the knuckles drag. Engineers get answers and don’t ask questions. You aren’t there to think, just to do. The real problem is that the weapons program at Sandia is even worse. They are stuck in the past, engineering everything with decades-old approaches, unable to modernize.

The first draft of this essay was angry. Yes, I am angry about this. Even more, I am sad. The state of this great institution is pitiful and worth mourning. It reflects the same thing we see across society. Decline. Lack of ethics. Lack of competence. Abusive leadership. We deserve better and we are getting worse.

In a fully developed bureaucracy there is nobody left with whom one could argue, to whom one could present grievances, on whom the pressures of power could be exerted.– Hannah Arendt

Institutional Rot

Again and again, Sandia gave me reasons to question its choices about leadership. It seemed to choose people without the ethical core needed for leadership. Perhaps the leadership is somehow trained to abuse power. Whatever the reasons, we have the wrong people in positions of responsibility. It matters since the responsibility is for the USA’s nuclear weapons. Instead of seeking excellence in technical execution, they seek power and the easy way. This is not serving the nation.

It ends up being another example of a rotting institution. The motives and incentives of Sandia’s management are the opposite of what they should be. As a result, I don’t fit. My abilities and skills are not useful since they are guided by values. Those values of technical excellence serving national security are useless. Why find and fix problems when you can just wish them away? Why get the right answer for the right reasons, when you can just define success? This is the way Cilantro manages. Abuse power and squash anyone who gets in the way. Like me.

If this continues much longer, Sandia will cease to be fit for National Security. The Nation deserves better. I am sad to see this. I am sad to have spent any of my precious time on Earth serving this travesty. I am sad to see these people in positions of responsibility. They have authority and exercise it. Their exercise is for bad. Perhaps Sandia teaches and nudges them toward this behavior. This is a rot I want no part of. We see it everywhere in society. If Sandia is any guide, the USA is in real deep trouble. If a great institution acts like this, what other horrors are afoot? We are in danger. Our leaders are letting us down.

The greatest evil is not now done in those sordid “dens of crime” that Dickens loved to paint. It is not done even in concentration camps and labour camps. But it is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed, and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voice.– C. S. Lewis

Why I Quit

It is not that we have a short time to live, but that we waste a lot of it.– Seneca

Mortality was sinking in. I had to make some clear decisions about who I was, what I would do with the time left, and what values I would stand for. I had grown tired of cutting corners and making changes to fit in. Letting go of my most cherished values and ethics was one thing too far, one psychic blow I couldn’t take. I needed to be myself. I needed to stand up for the right thing.

What became very clear was that I stood in opposition to the values of this institution. Not its stated values, but its practiced values. In a sense, those values are most fully articulated in who the institution puts in positions of power, and as I’ve noted before, they put some genuinely awful people in those positions. Cilantro is an exemplar, and he is just like some of the other awful people they put in charge of things. When they hand these people authority, and those people exercise it through abuse of power, it speaks volumes about what Sandia actually is, as opposed to what it purports to be. You see people through what they do, not what they say. I was not prepared to sacrifice one more iota of my self-respect or my time supporting actions completely antithetical to my most cherished professional values.

The Report

For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.– Richard Feynman

The clearest evidence about the ruling that the report I’ve spent some time explaining is export-controlled can be found on OSTI itself. I received a letter demanding I take the report down after I downloaded it from the OSTI website. I got the document from a public website where things only get posted after a process. This needs some explanation. Only unclassified unlimited release documents go there. These documents have gone through a process where managers and other officials review them. This is not some decision the author makes. It is must be approved before it is issues a unclassified release number.

I don’t know what happened, but I can surmise. I had used the above process hundreds of times. It takes time, and other eyes see it. The report was in the bullseye of my professional knowledge and experience. I had worked on these test problems for decades with codes almost identical to the ones report. The results are predictable. Clearly, other members of management decided that it needed to be export-controlled instead after the fact. The reasons were political, not technical. There is NO valid technical reason for making it export-controlled. NONE! This is corruption and nakedly so. Fundamentally, the categorization of this report as “export controlled” is a lie. It is a lie focused on hiding bad results and, worse yet, shows no interest in solving the identified problems.

The real evidence of this is what you can find on the OSTI site. There are a number of documents related to CTH. Moreover, many of these are closer to the export-controlled line than the one in question. Just look at all the reports about high explosives. What is truly the smoking gun are two documents that are more extensive, but ostensibly the same thing. Verification reports including the test problems in the problematic report. The only difference is the nature of the results. These other reports were written trying to show the code is just fine, correct. They want to show that everything is okay. They do not identify any problems. This is for a good reason; they were written to deflect valid criticism.

There are two reports, from 2020 and 2021, on the verification and validation of the same code, CTH. Those reports were done with the objective of showing that the code is just fine. Checking the V&V box and moving on. This is verification done to blunt criticism, not partner with the science and engineering. That runs counter to the report I’ve discussed, which showed problems with the code. I encourage you to go to OSTI and download them. You only have to search under verification and CTH to find them. All of this just proves the corrupt intent of the managers, and the misuse of the export-controlled statute.

…a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it. – Richard Feynman

With regard to these earlier reports, the manager for CTH at that time knew damn well about my expertise in both shock physics and verification of shock physics codes. Knowing this, he never asked me to provide a peer review of those reports. I expect this is because he knew that I would find fault with them. They were not interested in doing excellent work. They were interested in deflecting criticisms.

What this also makes clear is that the decision is a pure abuse of power. They declared the report export-controlled because they can, not because any principle or technical fact required it. Acting without principle is simply a confirmation of the power they hold. Exercising that power is, first and foremost, how they manage. It is exemplified by the behavior of Dr. Cilantro, their boss. This is a key part of institutional rot across society. Sandia is just another example.

This provides absolute clarity about why the report is export-controlled. Management does not like the results. They are using export control to hide results they don’t like. It is unethical and corrupt, and it is emblematic of the way science is managed in the United States today. Corruption and the lack of ethics are everywhere.

It appears that there are enormous differences of opinion as to the probability of a failure with loss of vehicle and of human life. The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from the working engineers, and the very low figures from management.–Richard Feynman

…it would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy. – Richard Feynman

Verification as a Checkbox

The first principle is that you must not fool yourself — and you are the easiest person to fool.– Richard Feynman

There is one key difference with the verification reports that are available on OSTI. Those were done in order to provide a checkbox that verification had been performed. The main message is: we want everyone to shut the fuck up about verification, this code is just fine. The work was done by checking the box and declaring the code okay so that everyone could move on to more important things. Now we can move on to solving applications.

This is verification in its most egregious form, treated as a tedious and useless activity demanded by people who care about correctness and science. Those people are pests, and we do this work to shut them up. The last thing Sandia is really concerned about is high-quality technical work, which is why I was never asked to peer review these other reports. I found them by chance after they were published. Even if I had reviewed them, anything I said would have been ignored, because the purpose of the documents was to close the book on verification rather than harness it to make anything better.

This episode gets to the core of the question. You have reports that fall right into the bullseye of my professional and scientific expertise, and Sandia shows no interest in that expertise or in using it. They see verification as one of the checkboxes engineering requires, and they look to put it to bed as simply as possible so they can get on to the real work of engineering things and grinding out results. Getting things right is something that can be handled by calibration. Getting the fundamental hydrocode methods and solutions correct is something they have no real interest in. If it happens, it happens, but correctness is unnecessary, surplus to requirements. Engaging my expertise is a complication they can do without.

Nothing says “I am in the wrong place” more clearly than this

You could leave life right now. Let that determine what you do and say and think. – Marcus Aurelius

Coda

Sunlight is said to be the best of disinfectants; electric light the most efficient policeman. – Louis Brandeis

This was no place for somebody like me to work. I was wasting my time, my effort, and my life there, and I would encourage anyone with a desire to do creative, original work to stay away. It’s not for them. If you want a good paycheck and decent benefits, and you are prepared to follow orders without regard to the intelligence of those orders or the ethics involved, it’s the place for you. As long as you’re willing to follow the directives of managers who show some degree of incompetence and a general lack of ethics, you will do well there.

The truth is that I never belonged. It was broken before I arrived. Sandia is a place for small ideas, and it is about executing those small ideas with persistence and precision. It is not about accuracy or correctness. Its excellence is execution without regard for purpose or value. I am a scientist, not an engineer, and in the Sandia system the engineer executes whether the idea is a good one or not.

The real tragedy is the time that I wasted, when my skills and talents could have been used for something better. The other tragedy is the irresponsibility of those in positions of power at Sandia. They are overseeing aspects of the nation’s nuclear weapons and doing so without competence. That lack of competence is partially technical and partially ethical. In either case, the nation is being mis-served.

I really should stop writing about Sandia. I spent twenty years there, and I’ve come to realize it was a professional mistake. I never fit in, and while I was there I only grew to fit in less. Perhaps my frustrations boiled over and I self-immolated. I took a change of jobs into a place where I was even less suited. It was always a poor fit, and moving to Engineering Sciences amplified it. When Cilantro arrived as director, it was a reversion to form and my days were numbered. Fortunately, I had been getting my retirement shit together. I wasn’t going to put up with their bullshit, so I pulled the ripcord.

Drive out fear, so that everyone may work effectively for the company.– W. Edwards Deming

References

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, W. Rider, T. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LLNL-TR-411291. Lawrence Livermore National Laboratory (LLNL), Livermore, CA, 2008.

Duncan-Reynolds, Gabrielle Christiane, and Christopher T. Key. Improvements to the New CTH Code Verification & Validation Test Suite (FY2020). No. SAND–2020-13925. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2020.

Duncan-Reynolds, Gabrielle C., and Christopher T. Key. FY2021 Improvements to the New CTH Code Verification & Validation Test Suite. No. SAND2022-7834. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2022.

The early angry illustration.