Prolog

This is another leftover from when the blog was shut down. Usually, I would think and write about a talk before I gave it, but this is thinking after the talk. This has some advantages, as I got a lot of feedback after my talk about the culture at Lawrence Livermore National Lab. This is where I gave the talk, and the audience came and engaged with me, gave me some of the gaps to fill in about that particular institution.

I also went through my career-ending experience, which certainly changed my impression of the culture and current state of Sandia National Lab. More recently, I’ve re-engaged in a casual way with Los Alamos, and it’s reminded me of some of the aspects of that lab. All of which comes together for an interesting view of cultures: how they are created, how they evolve, and how they change due to the stimuli that they receive.

I gave the talk in October at Lawrence Livermore National Lab at the augustly named Nuclear Explosives Code Development Conference (NECDC). This talk was given in front of an audience from all three labs, but also from the Atomic Weapons Establishment (AWE) in the UK, and it was received well, with a great deal of feedback from the audience that I’ve incorporated into my writing.

“Only someone who is well prepared has the opportunity to improvise.” ― Ingmar Bergman

tl;dr

Working for nearly 40 years at two premier national laboratories is a heady experience. Part of what shapes my career deeply is the underlying culture of each institution. The differences between the two that I worked at are rather stark and interesting. It becomes even more stark when you realize that they have a common origin, but in that common origin, there are different forces that are unleashed that continue to this very day.

Culture, of course, is a subtle and esoteric thing that is hard to completely wrap one’s head around. In addition, the specifics of my career have shaped my experience, and these differences mean that my impressions of both laboratories are skewed towards my work as a computational physicist. Nonetheless, one can make conclusions about each culture and how it shapes the technical work and experience of working at each of these labs.

“Life is rarely about what happened; it’s mostly about what we think happened.” — Chuck Klosterman

A Personal Story

“It is a profound and necessary truth that the deep things in science are not found because they are useful; they are found because it was possible to find them.”— J. Robert Oppenheimer

As I write this story, I need to be very honest about my personal biases. Los Alamos National Lab had a distinct and profound effect on the trajectory of my professional career. I can unequivocally state that the first ten years of my career there were exemplary in every single way, and I gained an immense amount of personal growth. Any sort of sense of career success I’ve had stems from the gifts I was given then. I found an environment that was brimming with generosity, but also a degree of technical excellence and reverence for science. Great values I hold on to today. I benefited from the wisdom and knowledge of many Los Alamos staff members. The management then was dominated by “servant” leaders. It was a perfect incubator for a young scientist.

At the same time, at the close of those ten years, things changed. There was a sequence of scandals and events that deeply damaged Los Alamos and have left a lasting imprint on all three labs. It all started with the saga of Wen Ho Lee. In a very real way, those scandals also exposed the dark side of Los Alamos to me and the world. I fear that those events have also exposed all of the labs to aspects of the modern world that are exceedingly negative. These forces have destroyed much of the good that all three of these institutions. It also destroyed the positives they should be creating.

The reverence and pursuit of science or knowledge in general is a clear vestige of Los Alamos’s impact on me. Los Alamos is also the origin point for all three institutions, as I will describe. They all arose from the Manhattan Project and the Cold War that followed. At the same time, much of that scientific approach is done in pursuit of nuclear weapons. As such, there is a cloud over everything these labs do around one’s belief in the morality associated with nuclear weapons.

I am a generally liberal and progressive person, and see the downside and the problems with nuclear weapons from a moral and ethical perspective. I also have a pragmatic view that nuclear weapons represent a genie that can’t simply be put back in the bottle. As a patriot for my country, I believe that it is essential that the United States have competence and capability in nuclear weapons that is second to none. This is still an issue that fills me with a great deal of conflict internally. What I do remain steadfast in is my belief that science is an important part of societal good and something worth pursuing in and of itself. In sum, the science these labs have (and can) produce is a huge benefit to the USA and mankind.

A major caveat of what I’m going to write is that my personal experience is focused on a combination of computational physics and computer codes developed by the labs. I conduct the examination of those codes and their results through the application of verification and validation. Each of these pursuits means that my viewpoint on the labs is seen through those lenses. I have seen how my efforts are perceived. Thus, I must admit that my own perspective is skewed and biased by the nature of what I do and what I have learned. Notably, the meeting I spoke at is about “code development”, not “computational physics”. This alone speaks to a downgrade for the activity. People doing physics are the users of the codes. This matters and says a lot.

All of these details have a huge impact on the product that the labs produce. The computer codes, the analyses, and the experiments that they conduct all have the imprint of these cultural signatures. This, together with the national culture, directs each place’s culture. Cultures are amazingly persistent. Aspects of the Lab’s cultures have been swept up in the change of the National culture. This might say more about the epic nature of the current time. I will say that over the course of my career spanning nearly 40 years, I learned about these cultures but also watched these cultures evolve. By and large, the evolution of the cultures of the labs has been very negative and parallels and mirrors the negative developments in American culture as the scientific legacy of the Cold War has basically faded from view and been replaced by the post-Cold War view of things.

Each lab has a distinct identity. There is a knee-jerk view of it, which is:

– Los Alamos is the physicist

– Livermore is a computer scientist

– Sandy is an engineer

These are not too far from the truth. The actual reality is a little more subtle. Los Alamos is the experimental physicist; Livermore is the computational physicist, and Sandia is the knuckle-dragging engineer.

I can speak to a set of core events and attitudes that reflect the cultures quite well. This is most evident and most acute in the terms of Los Alamos, where the development of codes for weapons work has never been an enterprise that has been looked on by much favor by the lab’s elites. This is contrasted with Lawrence Livermore, where developing codes and producing numerical methods on supercomputers is the central and highest purpose of the laboratory (along with fusion of all sorts). By and large, the code development at Lawrence Livermore is far, far more successful than either of the other labs.

“Knowledge cannot be pursued without morality.” — J. Robert Oppenheimer

Shaped by Key People.

These labs are identified with certain personalities. Three individuals from the Manhattan Project stand out as much as the culture of each lab. In particular, the obvious one is J. Robert Oppenheimer and Los Alamos. We had an Oscar-winning movie to vividly tell the tale. All three characters are prime players in that story. This is well known, accepted, and obvious. In a similar vein, Edward Teller is often identified as the godfather of Lawrence Livermore. With Sandia, this is less well known and accepted. Sandia had a historian who seemed to completely ignore the impacts of the Manhattan Project on Sandia. I have grave oversight in my opinion. Sandia started in Los Alamos in 1943, not in 1949.

“Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that ‘my ignorance is just as good as your knowledge.'”

— Isaac Asimov

My premise, having worked there for almost 20 years, is that General Leslie Groves is the forefather of Sandia. Groves oversaw and controlled the weapons engineering activity that accompanied the development of the atomic bomb. Groves’ basic mentality of running projects, including his obsession with operational security, is still king there. His zealous application of need-to-know as a principle is the true motto at Sandia. The process of engineering in WW2 is how weapons engineering is done today. If one looks at the archives of the history of the Manhattan Project, one can see the imprint of current weapons engineering at Sandia all over it. So little has changed in 70-plus years. The principles of that day are still alive and well today at Sandia. This is true knuckle-dragging. This engineering is backwards and backward-looking. It is also culturally entrenched and utterly resistant to change. Management claims otherwise are bullshit.

By the same token, the personalities and nature of Oppenheimer live in the spirit of Los Alamos and how people behave there, including his particular nature of rogue scientific excellence and eccentricity. At its core, Los Alamos is dominantly an experimental physics lab. The Trinity event was the model of this spirit. For most of its history the Weapons Working Group (WWG) was the ritual. This was the meeting of all the disciplines to work together on the same experiment. That experiment was the core of the Lab’s heart. Similarly, Teller’s legacy lives on at Livermore in terms of his fierce Cold War and anti-communist attitudes as well as his appetite for theoretical physics. The obsession with fusion is partnered with these. All of this is reflected in the current approach to activities..Weapons and fusion are the Lab. All else is simply a distraction. Both labs favor collaboration and vast swaths of science working together. The emphasis and priority of each is the variance.

There were mentions of secondary figures at each lab. In Los Alamos, the second person to think of as shaping the culture of the lab is Harold Agnew. He was there at the Manhattan Project and ultimately became the lab’s director. Harold was present for history many times, including the Chicago pile (the first critical fission reaction). He was also on a plane over Hiroshima observing that bombing. He is often viewed by the old-timers there with great warmth, and a time when Los Alamos reached its apex during the Cold War.

For Livermore, the person who stands out at Livermore is Johnny Foster. He had great achievements in the 1950s and ultimately became Livermore’s director before moving into even higher echelons in government at the White House. Foster represented some of Livermore’s greatest achievements. He also showed the fierce and deep engagement with National security. This has its echo in the Strategic Defense Initiative (SDI, Star Wars) and in Washington. Los Alamos was the cowboys (for good and ill), Livermore has the suits combined with California suave.

There are no individuals who stand out at Sandia. Having worked there for twenty years, one of the things I noted is that Sandia is very poor at recognizing the achievements of individuals. Most individuals’ achievements are simply subverted to institutional achievements and the identity of people doing the work is usually not celebrated. This is reflected in the lack of big personalities shaping the laboratory itself. The best place to find a few heroes is in the excellent work of Eric Schlosser, author of Command and Control. He documented the principles behind the nuclear safety stockpile. That arose in the 1960s and has served the American stockpile well to this day. These are often embodied in the principles that Sandia stands next to, which are always and never. Nuclear weapons are always ready when the nation calls on them, but never under conditions where they’re not being called for. This is both reliability and safety embodied and seen in how the modern stockpile behaves.

“Dropping a nuclear weapon was never a good idea.”

— Eric Schlosser

The subtle upshot of Sandia’s attitude toward people is throttling greatness. Los Alamos and Livermore have great scientists. Some people make incredible achievements. Nobel Laureates come from their ranks. You will occasionally meet one visiting there (I once met Murray Gell-Mann at daybreak outside the T-Division building). This won’t happen at Sandia. Ever. Achievement is institutional. No one is singled out. This is whether they deserve it or not. It becomes a self-fulfilling prophecy. No one great will arise, and if they do, they leave. It took me far too long to recognize this.

The Manhattan Project Origins

“There is no place for dogma in science. The scientist is free, and must be free to ask any question, to doubt any assertion, to seek for any evidence, to correct any errors.”

— J. Robert Oppenheimer

All three institutions have their origins in the Manhattan Project. Sandia became independent from Los Alamos after World War II. Nonetheless, it carried with it the experience and the structures that the Manhattan Project brought, and this is useful to understand it today. Much of what one experiences at either Lab connects to values and systems instituted there. The origin story for both Labs is still powerful and guiding. These are the legends and mythos for both.

The last lab, Lawrence Livermore, is a place I have not worked. I’ve had a great deal of contact with them over the years, both at Los Alamos and at Sandia. The dark side of an organizational culture is hard to intuit until you’re inside it. This is the part of the culture that is hidden and unknown unless you actually live within it. I have only gotten hints of the darkness there. It is very surely present. Livermore does seem to have enforced silence about their screwups. One keeps quiet about their problems and mistakes until management allows it. This gives them time to clean it up or turn around the narrative. Sandia is similar. Los Alamos leaks like a sieve. Problems are far more transparent. Los Alamos doesn’t have more problems; they are simply more visible. This is probably just the small-town effect.

That said, I can speak to what I learned at the talk I gave and what it all says. The most direct reflection of my experiences is embedded in the notion of what code development looks like at the labs. These are thoroughly imprinted with the technical challenges that the labs have. I worked on codes at Los Alamos and know the details from most of the Lab’s history. My knowledge of Sandia is more limited, mostly because of the lack of common knowledge there. Sandia is simply divided and insular internally. People there are friendly, but it is not a friendly place. Livermore codes are more well-known because they have some of their greatest accomplishments.

Los Alamos and Lawrence Livermore both develop codes for much of the same purpose, and their codes have the same structures, but they work under completely different cultural ethos. Sandia, on the other hand, always has a chip on its shoulder, particularly with computer codes, and perhaps for good reason. The situation they deal with is both simpler from a technical perspective of each code and more complex in terms of how to thread everything together. Sandia also distinctly works in a non-integrated fashion and actually separates all the functions of its work significantly. The divisions at Sandia are driven by the broad application of “need to know”. Common knowledge and information drive connection, and Sandia destroys common knowledge as a matter of course.

“Men build too many walls and not enough bridges.”— Joseph Fort Newton

No issue at Sandia is more separated than the hardcore weapon engineers from the people doing science and codes. They are organizationally separated by quite a distance, and they basically don’t live in the same world or speak the same language. Conversely, Los Alamos and Livermore have their code development and the hard-core weapon teams doing weapons work closely associated with each other organizationally. While there’s tension between them, they tend to speak the same language and, broadly speaking, are all physicists. This leads to a much more unified effort and provides better service to the nation. Sandia takes operational security to an absurd place where it threatens the effective execution of technical work. The same mentality keeps them from innovating and creating anything new. It keeps Sandia and its weapons work living in the past.

Computer Codes and Their Developers

This reality is expressed in the computer codes used by each lab. When one looks at Los Alamos carefully, all the computer codes used heavily for their programs were developed elsewhere, mostly at Livermore, but also in the UK by AWE. Notably, one of the major codes comes from a contractor (a beltway bandit). One of the key things about the big integrated codes at Los Alamos and Livermore is their structure. They are repositories for huge amounts of physics, but hydrodynamics is the core. The key part of those codes is the hydro method and algorithm. The explanation is that all the other physics use the material and mesh map that the hydro creates. As such, the hydrodynamic methods have an outsized impact on the code’s quality and structure. I’ve quipped that the hydro scheme is the skeletal structure of the code that holds the rest of the physics. Sandia builds its own codes, but this code development is always expressed in a severe degree of envy with regard to Los Alamos and Livermore. They always feel like they’re second best and this feeling is not too far from the truth.

I did hear a story at Los Alamos that explains the second-class nature of the co-development there. This comes from the days of Pacific Island testing. In those days, the glory as a Los Alamos weapons person was being in the Pacific with those massive H-bomb tests. As one might expect, travel wasn’t easy, and you were separated from your family and home for long periods of time. This was hard on physicists. Not seeing their kids, not seeing their wives, thus marriages suffered. The sense was that one could take a break from this grind and spend time at home doing code development. Thus, code development became associated with people who were slackers. They were not sufficiently committed to the mission to sacrifice their family and marriage at the altar of nuclear supremacy.

This attitude persists to this day and leads code developers to be viewed as second-class nuclear physicists. The result is an inability of Los Alamos to develop codes for itself. Code development is not really respected there. You were never quite as good as the designers of the weapons. The impact on the lab is profound. They are always importing a code from elsewhere, where computational science is taken more seriously. This was a vicious cycle. It led to self fulfilling prophesy and code development received less support along with the lack of respect. It became second-rate.

This gets to the identity of Livermore, which is heavily grounded in computational physics. Moreover, some of the greatest accomplishments of Livermore are tied to their codes. Thus, computational physics receives support at Livermore. Computer science is treated as an important discipline and is empowered. As a result, the prestige and quality of the code Livermore produces exceeds that from Los Alamos by a large degree. If something is respected, supported, and celebrated, the quality follows. In a sense, Livermore supremacy and code development are a foregone conclusion once you look at the culture of the labs.

In this area, Sandia is always third best. This even includes their development of codes that are key to engineering those that do mechanics calculations, where again the codes developed at Livermore are the precursors and, for most intents and purposes, copied by Sandia. Another area where this is very true is shock physics, where Sandia’s codes are always second best. Sandia is really good at software engineering, but what they engineer isn’t really that good. They are definitely not innovative. Innovation is something not supported by the culture. Innovation is viewed with suspicion. They also do not really look outside their organizations. The insular and isolated nature are all encompassing.

That said, Sandia’s codes work well for engineers who are doing day-in and day-out work, having construction that makes them useful. The technical content of these codes leaves much to be desired. There are some really sketchy ideas embedded in very good quality code. It is really good, high-quality code. Livermore comes close in that regard. The Los Alamos code quality is last here.

Sandia, on the other hand, operates in this separate, individual-focused manner that produces small codes that always look like they are second-rate compared to what the other labs produce. This, of course, is not entirely true, as Sandia has also ended up being the supplier of codes like CTH to the broader defense industry. A large part of this simply stems from the fact that, while the NNSA labs have their problems, they are vastly better than DoD labs. This is remarkable considering the size of the defense budget. NNSA labs are incredibly superior to the Department of Defense labs. The science at the Department of Defense labs is abominable and can never produce anything that holds a candle to what the NNSA labs produced. This, in reality, is more of a condemnation of the scientific environment in the entire country.

Los Alamos Greatness Denied

One of the tragedies of Los Alamos is that it is properly viewed as the origin of computational fluid dynamics, or CFD. It had two of the greats in CFD working there: Peter Lax and Frank Harlow. Neither of them has had much influence on Los Alamos’ weapons codes. Both of them have had massive influences outside the Lab.

For example, the ideas of Frank Harlow found life in a whole batch of areas, including fusion computations and codes outside Los Alamos. Peter Lax defined basic mathematical work that forms the foundation of most compressible CFD codes in the world today. The tragedy is that neither of them had much influence at all on the actual codes developed by Los Alamos. This demonstrated the genuine animosity that weapons physicists at Los Alamos had for the homegrown talent. In computational physics, invented here is disregarded at Los Alamos.

Similarly, ideas of Harlow found much more traction at both Livermore in terms of the arbitrary Lagrangian-Eulerian (ALE) codes and interface tracking. Interface tracking work was picked up and manifested into useful code by David Young at AWE. Ultimately, some of David’s work was imported back into Los Alamos, so ultimately the work of Harlow and Lax had to be taken advantage of elsewhere and then imported back into Los Alamos after being committed to code. The code is only accepted if someone outside the Lab writes it. For the most part, code development at Los Alamos is simply caring for code others write.

The Modern Era After the Cold War

“Nothing is so difficult as not deceiving oneself.” — Ludwig Wittgenstein

My own experience is with two of these national labs: Los Alamos first and Sandia second. I must always be completely aware that at the same time as I was employed, these institutions were evolving due to the nature of the modern world. The whole nature of the nuclear stockpile changed dramatically, and only a couple of years into my career could it be said to have occurred during the Cold War. These institutions were dominantly shaped first by World War II and then by the vestiges of the Cold War itself. The labs grew and flourished during the Cold War. The period after the Cold War has been characterized by decay and destruction. The cultures have been attacked by modern governance.

My experience with Livermore is more superficial. Livermore operates under the same governance as Los Alamos and Sandia. It is also in a different setting in California. Sandia operates a small Lab across the street from Livermore. It is vastly different than Sandia, New Mexico. I suspect many of these differences are present with Livermore itself. They both exist in the East Bay area and show that region’s culture strongly. The issue comes down to the fact that when you interact with a lab, you see mostly the good and very little of the bad. You only see the bad part or the bad face of an institution when you’re inside it. I can certainly speak chapter and verse on the bad sides of Los Alamos and Sandia. I am fairly certain that Livermore has the bad side as well, but that is largely invisible to the outsider.

The shadow of what’s bad with Lawrence Livermore can be seen with the dynamics around NIF, which is one of Livermore’s greatest priorities. It shadows the pursuit of fusion as one of their core initiatives. NIF has had a whole host of scandals and issues along its way. It is undoubtedly an incredible experimental platform. On the other hand, the degree of overselling of its achievements is fairly appalling and should be a black mark. Yet we seem to live in an age where bullshit is favored, and if that bullshit is found out, the news cycle and attention have moved on already, and the bullshitter survives.

The same 30 years have seen the management structure of all the labs change. The effect of the current management approach to labs is to hollow them out. We see science in decline and technical quality and excellence becoming a shadow of their former selves. I saw this start in Los Alamos and really take hold in the 2000s, and I have watched it continue at Sandia. All the labs operate under the same auspices, and I can’t imagine that Livermore has declined as well. The management now only really focuses on money above all else, looking for programs that have large funding, and this prioritizes management who act as empire builders. The lack of trust in our society and the way that government funds things and casts a doubting eye towards everything done has heavily and has damaged the laboratory. This has left us in a situation where I don’t think we are ready for a new strategic competition internationally, even as we seem to be promoting and proposing that it occurs.

In the current state, the United States really has what I would call Schrodinger’s nuclear stockpile. It both works and doesn’t work simultaneously. We won’t know until we look inside the box. God help us if we have to look inside that box. The nation has done a terrible job at caring for these Labs. We have allowed them to decay and decline for the last 40 years. If their work becomes important and visible, we are unlikely to like what we see,

“The best way to predict your future is to create it” ― Peter Drucker