If you do not work on an important problem, it’s unlikely you’ll do important work.–Richard Hamming

An answer to an Asshole’s Question

I talked to my mentor, and he passed along a comment Cilantro had made about me:

“Why did we hire him in the first place?”

This was the prelude to them fucking me. Cilantro was my director, so I was fucked. They were going to come after me. I had no defense except surrender. Discipline. Shitty performance review. Cut off from anything good. It worked, and I was gone.

The timing was perfect. After Covid, and some modestly distressing personal health issues, my activation energy was incredibly low. My father was in hospice. I had watched him decline and suffer for over a year. Wasting my life working for incompetent, unethical assholes was an easy choice. I had checked my finances. I have two pensions too. I could retire and be comfortable.

It turns out Cilantro was asking a good question. Why was I there? Why had I come there in the first place?

The answer is twofold. My hire seemed a no-brainer back in 2007. The code I was working with, ALEGRA, was having problems. I had deep knowledge of hydrocodes from my years at Los Alamos. I knew how to write them, and I knew the details and foundations of hydrocode methods and applications. I was also a V&V expert. I had a great research record. I had been a line manager and a program manager. They were thrilled to have me.

All of those reasons were wishful thinking. It was a terrible fit. As difficult as it is to admit, Cilantro had a point. Why did they hire me? What use was I to that place?

I hired into the Computer Research Center, one of the most externally focused places at Sandia. It smells a little like Los Alamos. The difference is that this center could give two fucks about the Lab mission, and I was going into the most mission-focused department in it. My values were to care deeply about the Lab’s mission. Contributing to it was paramount. Thus, in the Computer Research Center that department was on the outside looking in. The rest of the Center was on the outside looking out.

If you gave a fuck about the mission, you worked in the Engineering Sciences Center. In fifteen years I would end up there. The problem is that Engineering Sciences is long on engineering and short on science. It bridges to the weapons program, where the knuckles drag. Engineers get answers and don’t ask questions. You aren’t there to think, just to do. The real problem is that the weapons program at Sandia is even worse. They are stuck in the past, engineering everything with decades-old approaches, unable to modernize.

The first draft of this essay was angry. Yes, I am angry about this. Even more, I am sad. The state of this great institution is pitiful and worth mourning. It reflects the same thing we see across society. Decline. Lack of ethics. Lack of competence. Abusive leadership. We deserve better and we are getting worse.

In a fully developed bureaucracy there is nobody left with whom one could argue, to whom one could present grievances, on whom the pressures of power could be exerted.– Hannah Arendt

Institutional Rot

Again and again, Sandia gave me reasons to question its choices about leadership. It seemed to choose people without the ethical core needed for leadership. Perhaps the leadership is somehow trained to abuse power. Whatever the reasons, we have the wrong people in positions of responsibility. It matters since the responsibility is for the USA’s nuclear weapons. Instead of seeking excellence in technical execution, they seek power and the easy way. This is not serving the nation.

It ends up being another example of a rotting institution. The motives and incentives of Sandia’s management are the opposite of what they should be. As a result, I don’t fit. My abilities and skills are not useful since they are guided by values. Those values of technical excellence serving national security are useless. Why find and fix problems when you can just wish them away? Why get the right answer for the right reasons, when you can just define success? This is the way Cilantro manages. Abuse power and squash anyone who gets in the way. Like me.

If this continues much longer, Sandia will cease to be fit for National Security. The Nation deserves better. I am sad to see this. I am sad to have spent any of my precious time on Earth serving this travesty. I am sad to see these people in positions of responsibility. They have authority and exercise it. Their exercise is for bad. Perhaps Sandia teaches and nudges them toward this behavior. This is a rot I want no part of. We see it everywhere in society. If Sandia is any guide, the USA is in real deep trouble. If a great institution acts like this, what other horrors are afoot? We are in danger. Our leaders are letting us down.

The greatest evil is not now done in those sordid “dens of crime” that Dickens loved to paint. It is not done even in concentration camps and labour camps. But it is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed, and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voice.– C. S. Lewis

Why I Quit

It is not that we have a short time to live, but that we waste a lot of it.– Seneca

Mortality was sinking in. I had to make some clear decisions about who I was, what I would do with the time left, and what values I would stand for. I had grown tired of cutting corners and making changes to fit in. Letting go of my most cherished values and ethics was one thing too far, one psychic blow I couldn’t take. I needed to be myself. I needed to stand up for the right thing.

What became very clear was that I stood in opposition to the values of this institution. Not its stated values, but its practiced values. In a sense, those values are most fully articulated in who the institution puts in positions of power, and as I’ve noted before, they put some genuinely awful people in those positions. Cilantro is an exemplar, and he is just like some of the other awful people they put in charge of things. When they hand these people authority, and those people exercise it through abuse of power, it speaks volumes about what Sandia actually is, as opposed to what it purports to be. You see people through what they do, not what they say. I was not prepared to sacrifice one more iota of my self-respect or my time supporting actions completely antithetical to my most cherished professional values.

The Report

For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled.– Richard Feynman

The clearest evidence about the ruling that the report I’ve spent some time explaining is export-controlled can be found on OSTI itself. I received a letter demanding I take the report down after I downloaded it from the OSTI website. I got the document from a public website where things only get posted after a process. This needs some explanation. Only unclassified unlimited release documents go there. These documents have gone through a process where managers and other officials review them. This is not some decision the author makes. It is must be approved before it is issues a unclassified release number.

I don’t know what happened, but I can surmise. I had used the above process hundreds of times. It takes time, and other eyes see it. The report was in the bullseye of my professional knowledge and experience. I had worked on these test problems for decades with codes almost identical to the ones report. The results are predictable. Clearly, other members of management decided that it needed to be export-controlled instead after the fact. The reasons were political, not technical. There is NO valid technical reason for making it export-controlled. NONE! This is corruption and nakedly so. Fundamentally, the categorization of this report as “export controlled” is a lie. It is a lie focused on hiding bad results and, worse yet, shows no interest in solving the identified problems.

The real evidence of this is what you can find on the OSTI site. There are a number of documents related to CTH. Moreover, many of these are closer to the export-controlled line than the one in question. Just look at all the reports about high explosives. What is truly the smoking gun are two documents that are more extensive, but ostensibly the same thing. Verification reports including the test problems in the problematic report. The only difference is the nature of the results. These other reports were written trying to show the code is just fine, correct. They want to show that everything is okay. They do not identify any problems. This is for a good reason; they were written to deflect valid criticism.

There are two reports, from 2020 and 2021, on the verification and validation of the same code, CTH. Those reports were done with the objective of showing that the code is just fine. Checking the V&V box and moving on. This is verification done to blunt criticism, not partner with the science and engineering. That runs counter to the report I’ve discussed, which showed problems with the code. I encourage you to go to OSTI and download them. You only have to search under verification and CTH to find them. All of this just proves the corrupt intent of the managers, and the misuse of the export-controlled statute.

…a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty — a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid — not only what you think is right about it. – Richard Feynman

With regard to these earlier reports, the manager for CTH at that time knew damn well about my expertise in both shock physics and verification of shock physics codes. Knowing this, he never asked me to provide a peer review of those reports. I expect this is because he knew that I would find fault with them. They were not interested in doing excellent work. They were interested in deflecting criticisms.

What this also makes clear is that the decision is a pure abuse of power. They declared the report export-controlled because they can, not because any principle or technical fact required it. Acting without principle is simply a confirmation of the power they hold. Exercising that power is, first and foremost, how they manage. It is exemplified by the behavior of Dr. Cilantro, their boss. This is a key part of institutional rot across society. Sandia is just another example.

This provides absolute clarity about why the report is export-controlled. Management does not like the results. They are using export control to hide results they don’t like. It is unethical and corrupt, and it is emblematic of the way science is managed in the United States today. Corruption and the lack of ethics are everywhere.

It appears that there are enormous differences of opinion as to the probability of a failure with loss of vehicle and of human life. The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from the working engineers, and the very low figures from management.–Richard Feynman

…it would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy. – Richard Feynman

Verification as a Checkbox

The first principle is that you must not fool yourself — and you are the easiest person to fool.– Richard Feynman

There is one key difference with the verification reports that are available on OSTI. Those were done in order to provide a checkbox that verification had been performed. The main message is: we want everyone to shut the fuck up about verification, this code is just fine. The work was done by checking the box and declaring the code okay so that everyone could move on to more important things. Now we can move on to solving applications.

This is verification in its most egregious form, treated as a tedious and useless activity demanded by people who care about correctness and science. Those people are pests, and we do this work to shut them up. The last thing Sandia is really concerned about is high-quality technical work, which is why I was never asked to peer review these other reports. I found them by chance after they were published. Even if I had reviewed them, anything I said would have been ignored, because the purpose of the documents was to close the book on verification rather than harness it to make anything better.

This episode gets to the core of the question. You have reports that fall right into the bullseye of my professional and scientific expertise, and Sandia shows no interest in that expertise or in using it. They see verification as one of the checkboxes engineering requires, and they look to put it to bed as simply as possible so they can get on to the real work of engineering things and grinding out results. Getting things right is something that can be handled by calibration. Getting the fundamental hydrocode methods and solutions correct is something they have no real interest in. If it happens, it happens, but correctness is unnecessary, surplus to requirements. Engaging my expertise is a complication they can do without.

Nothing says “I am in the wrong place” more clearly than this

You could leave life right now. Let that determine what you do and say and think. – Marcus Aurelius

Coda

Sunlight is said to be the best of disinfectants; electric light the most efficient policeman. – Louis Brandeis

This was no place for somebody like me to work. I was wasting my time, my effort, and my life there, and I would encourage anyone with a desire to do creative, original work to stay away. It’s not for them. If you want a good paycheck and decent benefits, and you are prepared to follow orders without regard to the intelligence of those orders or the ethics involved, it’s the place for you. As long as you’re willing to follow the directives of managers who show some degree of incompetence and a general lack of ethics, you will do well there.

The truth is that I never belonged. It was broken before I arrived. Sandia is a place for small ideas, and it is about executing those small ideas with persistence and precision. It is not about accuracy or correctness. Its excellence is execution without regard for purpose or value. I am a scientist, not an engineer, and in the Sandia system the engineer executes whether the idea is a good one or not.

The real tragedy is the time that I wasted, when my skills and talents could have been used for something better. The other tragedy is the irresponsibility of those in positions of power at Sandia. They are overseeing aspects of the nation’s nuclear weapons and doing so without competence. That lack of competence is partially technical and partially ethical. In either case, the nation is being mis-served.

I really should stop writing about Sandia. I spent twenty years there, and I’ve come to realize it was a professional mistake. I never fit in, and while I was there I only grew to fit in less. Perhaps my frustrations boiled over and I self-immolated. I took a change of jobs into a place where I was even less suited. It was always a poor fit, and moving to Engineering Sciences amplified it. When Cilantro arrived as director, it was a reversion to form and my days were numbered. Fortunately, I had been getting my retirement shit together. I wasn’t going to put up with their bullshit, so I pulled the ripcord.

Drive out fear, so that everyone may work effectively for the company.– W. Edwards Deming

References

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, W. Rider, T. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LLNL-TR-411291. Lawrence Livermore National Laboratory (LLNL), Livermore, CA, 2008.

Duncan-Reynolds, Gabrielle Christiane, and Christopher T. Key. Improvements to the New CTH Code Verification & Validation Test Suite (FY2020). No. SAND–2020-13925. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2020.

Duncan-Reynolds, Gabrielle C., and Christopher T. Key. FY2021 Improvements to the New CTH Code Verification & Validation Test Suite. No. SAND2022-7834. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2022.

The early angry illustration.