tl;dr

The resolution of the crisis of my Spring grad school failure perfectly set me up to succeed at Los Alamos. Los Alamos was perfectly set up to grow me. It was a place for me to meet my potential. The heart of this was a spirit of trust and generosity. In a deep sense, this was the good part of the spirit (ghost) of Oppenheimer. My time at Los Alamos made the most of this. It ended as his dark side met events. The modern USA of the 2000s could not tolerate any of the echoes of Oppenheimer. It ruined his legacy and dismantled something unique and incredible for the Nation. I was just another casualty.

“No single act of generosity remains in isolation. The ripples are many.” ― Sarah Winman

The start of Los Alamos, the perfect place for a career to gestate.

As I mentioned before, I felt amazing to have the opportunity to work at Los Alamos. Getting the job in the first place felt like a miracle. I realize in retrospect that I was lucky to be looking for a job when I did. This was not a plan or mindful, just pure dumb luck. The key was that I was ready to take advantage of that luck. This all happened in the Spring of 1989, and as I said, interviewing was an adventure. I had six interviews. Three of them took plane trips. One to Idaho Falls and two to Virginia.

In terms of pure humor and great stories, the two interviews in Virginia were both catastrophes. Either of those jobs would have been awful, and the interviews definitely exposed that. I still got job offers from them, but the pay sucked, too. One of them became completely unveiled a few years later at a NASA conference on a manned Mars mission. It still stands as the most insane conference I’ve ever attended. It was 1992, and it included a visit from the NASA administrator (Dan Golden), automatic weapons fire, and trash cans full of cheap Midwest beer. This one contractor impressed me with their sheer incompetence. They were vastly less competent than I had measured in the interview in 1989. I thought they were morons, and I overestimated them. This was my first experience with the dangers of secrecy. They had been working on an SDI “black project”. They could tell me little then, but it felt like stupid bullshit. I learned that they were even dumber. It remains one of the most singularly incompetent things I’ve ever witnessed.

When I started at Los Alamos, my excitement was palpable. I did not realize how incredible an opportunity it was. The environment was ideal for growing me as a scientist. The Lab was the best incubator anyone could ask for. In retrospect, my attitude starting there was perfect for taking advantage of it. The equitable nature of the culture. Having a PhD didn’t make you special. You were just Bill or whatever your name was. People were curious, generous, and interested in assisting you. I started to learn immediately. What I didn’t know was that Los Alamos was in decline. It had fallen from the apex of the long, wide span of excellence. Still, the excellence lingered, and I would benefit immeasurably.

Writing like this is a mechanism of deep thinking. Along the way, you realize things that were lurking right below the surface. This essay is no different. The topic of generosity is top of mind in thinking about Los Alamos. There was a deep culture of sharing knowledge and time that benefited me. That sense of deep generosity seems gone today. When I arrived in 1989, Los Alamos was a more generous place than most. It exhibited a level of generosity that has largely disappeared today. This is true society-wide. We are now a selfish, self-centered society as a whole. The more I confronted this topic, the more this became self-evident. The last 25 years have killed generosity. We are poorer for it, and we will all pay.

Part of it links back to the destruction of trust. It will be a major part of the story for me. These are the “troubles” I’ve called it from 1999 to 2004 with multiple scandals. I will elaborate in depth later.

“There is no exercise better for the heart than reaching down and lifting people up.” ― John Holmes

N-12, Being an engineer

The labs have odd systems for naming organizations. LANL had letters to describe Divisions, with 100’s of people, and numbers for groups with 10’s of people. I started in N-12. “N” was for nuclear, and in this case reactors N-12 was the nuclear reactor analysis group. We did an analysis of nuclear reactors. I got there still in grad student mode, and little about the culture urged me to change. Shorts, sandals, and a T-shirt were still my summer uniform. I started on June 19, 1989.

The group was outside the inner circle of LANL, out at TA-52 on a mesa. I often describe Los Alamos looking like a hand with mesas being the fingers and the mountains as knuckles. In between the fingers were canyons. The canyons had interesting names like “acid canyon,” named for the shit they dumped there in WW2. My boss was a guy named Mike Cappiello, and I’ve got nothing but praise for him. He was an excellent first boss who helped to cultivate my enthusiasm. My first assignment was to learn how to use TRAC LANL’s reactor safety code. I did that, then sought to model a high-temperature gas-cooled reactor proposed to produce tritium for our nuclear weapons.

TRAC fucked it up. It was designed for water-cooled reactors. Gas reactor pumps are big fans with huge area changes. It solved an internal energy equation, and it led to huge errors. The old timers there said, “The kid is full of shit”. Thankfully, Dennis Liles, the author of the code, came to my rescue. He said I was right. Confidence was gained. I had my first victory. Soon, my interest in modern numerical methods got harnessed. I did a unique analysis of the solvers in TRAC. I had also started back at a PhD focused on these methods. I would finish that PhD by February 1992. I harnessed the hard work ethic I developed in my undergraduate years into an unstoppable force.

“The most truly generous persons are those who give silently without hope of praise or reward.” ― Carol Ryrie Brink

Along the way, I established the start of an essential collaboration for my future. I was working on a hair-brained scheme for transmuting nuclear waste into something less dangerous. It involved spraying 800 MeV protons into a high-Z target to create a shitload of neutrons and change material to less awful shit. One concept would use flowing lead as the target. It was a nasty free-surface problem. The experts for solving such problems were in T-3, Frank Harlow’s group. I approach a fellow young scientist, Doug Kothe, for advice. In keeping with LANL’s generous spirit, Doug invited me with open arms and mind. Over the years, this blossomed into a great collaboration. Later, we wrote a paper on interface tracking, which is my most cited article. That article was based on work that no project ever funded. It was merely me learning about an important technique and finding a new way to code it up. In addition, that very code is still being used in a LANL weapons code. Working with Doug also moved me closer to the Lab’s core mission. He also worked on a code that was on the Connection Machine, among other things.

Along the way, I put a modern method into TRAC to follow the solute used to shut the reactors down at Savannah River with unerring accuracy. I also got to see nuclear rockets and the artifacts of that program shut down after Apollo and Vietnam. Once I had my PhD, it was time to move on. I had bigger dreams. An ad was placed in the weekly lab paper, and I applied. Mike, my boss, agreed and actually brought me the ad too. I had already done it. I interviewed and moved. The next chapter was starting. I was moving from the periphery of the lab towards the center.

“It is not often that a man can make opportunities for himself. But he can put himself in such shape that when or if the opportunities come he is ready.” ― Theodore Roosevelt

C-3/CIC-19, National Science

The new job was really exciting. It was part of a national project including LANL, LLNL, and LBL. The leads were a couple of heavy hitters in modern methods, John Bell and Phil Colella. I would have a chance to get to know some luminaries from the literature firsthand. The project was to work on an exotic pulse reactor that involved shock waves and combustion. I would work on the compressible methods they developed, along with incompressible flow solvers. My boss was Jeff Saltzman, also well-known. I took to the job with energy and fervor.

It was a really important time in my life. My first child was born in mid-1994. My wife was finishing her Bachelor’s degree. In a sense, the work I was doing was almost a habilitation. I ended up writing another virtual thesis in verifying and understanding the incompressible flow methods. Along the way, I learned about linear solvers from dense methods to multigrid. I started to use multigrid to precondition Krylov methods. That seeded work later in my career with radiation transfer. I wrote a compendium of work on the various options for the approximate and exact projection methods for incompressible flow. These were coupled with modern methods for advection as well. I wrote several papers on the topics.

I continued and deepened my work with Kothe. In my “free time,” I wrote a code and a paper that now has 2500 citations. I coupled the interface tracking to incompressible flows and multigrid. I continued to learn about high-resolution methods, too. I also gained confidence in directly confronting Bell and Colella. This was something my peers thought was genuinely nuts. I did it and survived their pretty direct disagreement. It was far worse than any of my thesis defenses! I did things no one else thought were rational or wise. See a pattern?

Finally, I started to move to the core mission of LANL, nuclear weapons. In 1992, we stopped testing, and by 1995, the ASCI program was being hatched. I was part of those discussions. In the computing division building, there was a sub-rosa gym called “Bo’s gym”. I worked out there with other nuts. I would read on the stairmaster turned up to 11. A fellow nut at the gym noticed all the interesting shit I was reading. I would read a paper and toss it to the floor to dry off. My material was interesting. This gym goer was Len Margolin, my future boss. He was recruiting for the hydrodynamics group in the (in)famous X-Division. I was ripe for the picking.

It was time for me to get all the way to the core of the lab’s mission. I was a LANL guy now, and this was how to focus on what the Lab was made to do, nuclear weapons. It was also a new dawn; the ASCI program was the future. I would work in this program for the remainder of my career.

“Generosity is the most natural outward expression of an inner attitude of compassion and loving-kindness.” ― Dalai Lama

X-3, Becoming a weapons physicist

In late 1995, I moved to X-3. Len Margolin was my new boss. I shared Len’s passions for methods and codes. Len and I are still friends today. One of the things I value the most is how much we are able to disagree about things. Both of us hold some beliefs very closely that oppose each other. Nevertheless, we continue to engage and spar. This was true in 1995 and is true today, 30 years later. It is one of the things I value the most. Free exchange and combat of ideas wth the goal of understanding. This is science, and amongst my most deeply held beliefs. This is the way to do things.

“Give a bowl of rice to a man and you will feed him for a day. Teach him how to grow his own rice and you will save his life.” ― Confucius

Len gave me a lot of freedom to learn. I took it on myself to learn the business of X-Division. I learned the codes, methods, and weapons. I learned and implemented Lagrangian hydrocodes based on the Von Neumann-Richtmyer methods. I learned all the details of nuclear weapons and the physics associated with them. In many ways, I was pursuing a third PhD-level learning experience. It was another trip deep into the state of knowledge in deep areas. As part of this was the science of turbulence and mixing. A second part of this was verification and validation. The pursuit of both of these would shape much of my career.

In this time, I had that second crisis. I finally felt like I belonged at LANL. I was a fully fledged member of the Lab and deserved it. My time in X-3 was the richest and most integrated time of my career. Early on in my time, Len connected me with Jim Kamm. Jim and I connected as friends and colleagues. We worked together for nearly 20 years. It was an essential friendship and part of the magic that Len worked. Another thread of richness was the hiring of Dana Knoll, a friend from Grad School, plus Dana’s colleague Vince Mousseau. We leveraged our joint knowledge into a rich vein of research on radiation transport and multi-grid. My research life was beyond my wildest dreams. It was rich and varied. It would only get deeper and richer. My life and career were in full bloom.

Things go wrong. CCS-2/X-1, Finding a place to excel

Now that my career was flying, it was time for everything to go to shit. It wasn’t for me personally, but rather for Los Alamos. There was a series of scandals and events to rock the Lab. These came one after the other, shining a national spotlight on Los Alamos in a uniformly negative light. The first of these was a spy scandal starring Wen Ho Lee. Wen Ho was a truly mediocre scientist, but he did some truly huge things. He downloaded a lot of classified material at great personal effort. Why he did this remains unknown, although theories abound. I had an office next to his for around a year and a half. As I’ve said, when someone you know is in the headlines, it usually sucks. Definitely true here.

“You make all kinds of mistakes, but as long as you are generous and true and also fierce, you cannot hurt the world or even seriously distress her.” ― Winston Churchill

The whole scandal was a complete shitshow. Ultimately, it caused a bunch of changes at the Lab. All bad changes. Trust was crushed, and the shit flowed downhill. The government’s reaction fucked the lab up. The response started to break everything good at the Lab. It was a disaster for Los Alamos. The response was akin to pouring gasoline all over a fire. There were more problems ahead. The next was a forest fire, the Cerro Grande fire. It was a “controlled” burn by the forest service that got out of control. It burned 400 homes in Los Alamos and spurred an evacuation of the entire town. It also triggered scandal number two.

This was the disappearance of classified hard drives. Someone went to pick these discs up and take them to Sandia for safekeeping as the fire raged. They were not here. These hard drives were unaccounted for during the fire and about six weeks after. Once it was reported to the government, FBI agents descended on the Lab. It was a second disaster. This took place at the same time as Wen Ho was being tried for treason. The whole thing ended up as a massive cluster fuck where everyone looked like shit. They found the hard discs behind a printer, a couple of doors down from where they were supposed to be. Wen Ho got off as the government case went up in smoke. Worse yet, the two shit shows were connected awfully. FBI agents looked like assholes and got Nazi salutes (that they deserved!).

The Lab got even more fucked up. The disasters were not over. A third catastrophe was coming. Things started to come loose again. There was a new batch of missing classified hard discs (which never actually existed, an accounting error). An accident with a laser partially blinded a student. Now that Los Alamos had a raging asshole as director, the Lab was closed. The asshole was Pete Nanos, a retired Navy admiral. What a dick! He shut the whole place down. It was a lesson in how not to lead. In retrospect, Pete’s reign was also a harbinger of a different kind of leader; the kind our President is. Terror, fear, and intimidation were the tools of power and control. These were another way to separate the leaders from the led. Nanos was clueless about what happened at the working level.

Rumors abounded that Nanos’ predecessor was disconnected from real events. John Browne was a really great guy. He had impressed me when I worked underneath him (way down below). He was a good person, but he had been insulated from the working level. Maybe, maybe not. The events could have just happened anyway. He got canned because he just happened to be there. What I’ve seen over and over is leaders who don’t know what is happening. Each layer of management fails to inform them or just tells them good news. Any bad news is simply killed before it gets to them. Again, we can see this happening at the highest level. Our cabinet members bullshit and ass-kiss the President. No one tells the truth, and reality is just something they control. Power and control are absolute, and that power defines everything. The same with our corporate leaders who exist in a World utterly different from the rest of us. This difference in perspective is the handmaiden of catastrophe.

“Don’t sacrifice yourself too much, because if you sacrifice too much there’s nothing else you can give and nobody will care for you.”

— Karl Lagerfeld.

At the same time, I had a job offer from Livermore. I declined it. Why? The salary and cost of living there were combined to make it a truly dismal prospect for my family. I wonder greatly about how taking that would change my life. I also acted as a group leader at that time to help. I watched some serious shit go down. I watched my deputy group leader resign and retire in a single day as fear swept through the Lab. The Lab that gave me so much was dying right in front of me.

“Selfishness is not living as one wishes to live, it is asking others to live as one wishes to live.” — Oscar Wilde

Meanwhile, my professional life was in bloom. The most clear sign of that was a research project I led on turbulence modeling. One of the greatest things about Los Alamos was the courage and confidence it gave me. Turbulence is an incredibly complex and intimidating topic. I had avoided it like the plague. My commitment to the Lab and its mission gave me the impetus to pursue it. I learned it during the holiday break in 1999, and by 2003, I was leading research on it. That project was incredibly successful, with many well-cited papers and a book. It was merely one of several productive, great projects.

I wrote about 100,000 lines of code on modern shock capturing methods. I used the code to study important methods for Los Alamos and Livermore. I wrote many relatively important papers. I examined exciting ideas to improve methods. They’ve been less successful and influential than I would have liked. I also wrote my first book with Dimitis Drikakis. I still think they are good ideas. I also had the work on Krylov-multigrid, which was quite well-cited and successful. Finally, my work with Jim Kamm on V&V was influential. This is especially true at the Labs. I led a V&V project, and Jim led another. We made huge strides in the practice of verification at Los Alamos. This was the most successful and productive time of my career. All of it stemmed from the environment Los Alamos created. All of it came from the scientist it made me. Everything good about the Lab was being destroyed at the same time due to the scandals.

In the midst of all this, I was looking for solace. In the wake of the scandals, I was part of forming a new computational research division, CCS. I joined, leaving X-Division behind. I was in a good place, but it created new tensions. My collaborator Doug Kothe was my group leader. When he went up to the Division office, I applied for the management job. I didn’t get it because they said I was too “decisive”. It was a real WTF moment with echoes today. It was the impetus for seeking a position at Livermore. In retrospect, it was also the harbinger of management-leadership woes of today. Our leadership across society today suck! It also foresaw the end of my time in CCS. I would return to X-Division. Now that Los Alamos had turned to shit, I was moving toward the door, but hadn’t given up yet.

Can I manage?

I moved back to X-division. At the same time, a management wizard, Paul Hommert, had taken over. Paul had started at Sandia and had been the Lab director at AWE in the UK. He would eventually be the director of Sandia. Paul was vastly over-qualified for the job and replaced a real fuck up as X-division leader. I would say in retrospect that managing X-division is harder than managing either Sandia or AWE. It was enormously difficult. The designers of nuclear weapons work there, and these people are important and powerful. They are also arrogant, often for good reason. Some of them are enormously talented and accomplished. At that time, X-division also had domain scientists in physics, code developers, and computational physicists. Anyone who wasn’t a designer was a second-class citizen.

I had some freedom earned by my successful research in the 2000’s. This was counter-balanced by the damage all the scandals had done. The government’s reaction to these scandals was massively destructive to the Lab. They had deeply harmed the Lab’s ability to work. They had created a vast bureaucracy that was expensive and wasteful. Trust had been annihilated. Worse yet, the scandals kept coming. The latest ones in Los Alamos were amongst the dumbest yet. Someone took classified documents home on a thumb drive, and a meth addict boosted it. Someone bought a car with a P-Card. The result was more bullshit and less trust. Things were spiralling. I hadn’t given up yet. I kept doing my research, but I also wanted to see if I could manage people. I applied to be a deputy group leader in Paul’s new system. Hommert had installed a Sandia-like system in X-division. Now, deputy group leaders would be akin to department managers at Sandia. I got the job.

“Every man must decide whether he will walk in the light of creative altruism or in the darkness of destructive selfishness.” —Martin Luther King Jr.

This was my last act at Los Alamos. Being a manager sucked. In retrospect, I should have seen much more of Sandia in it than I did. My group leader was a big part of the sucking. He dumped work on his deputies so he could do research. Meanwhile, all of us were supposed to work half-time on technical work. The dumping meant we couldn’t, and our programs just had to suck it. So he was a terrible group leader and a selfish piece of shit. It should surprise no one that he’s been enormously successful. He rose through the ranks as a manager (they kept picking him for jobs). A slick British accent helps, too. He also got prestigious awards. Its another abysmal comment on leadership today. We pick too many fucking assholes to lead. He is a really good scientist, and that’s the job he should have stayed in. Instead, he was allowed to be an awful manager.

“There is nothing more frightful than ignorance in action.” — Johann Wolfgang von Goethe

There were a few more nails in the Los Alamos coffin. I can relay the episode that was probably the coup de grace. We had news that funding was short of what was needed to fund everyone. There might be a RIF. I had one person who couldn’t be funded. The why of this was repulsive. He had offended powerful people, primary designers. He used to sit with them at lunch, but they had begun making homophobic comments all the time. He called them on it. He was expelled from the clique. Now he was denied funding. It was completely out of my control or ability to influence. I protected his name, but felt the need to tell him what was happening. This was coming for him, and he was a pariah in X-Division. I went to him and let him know the situation. He was screwed and needed as much time to find something else as possible. It was powerfully emotional and difficult. He found a place to work and could keep his life.

He had also hosted a difficult classified conference. It was a very hard assignment and a minefield. He did a great job, but that meant nothing. It was also the meeting where Sandia recruited me, although I also sought them out.

References: My Top LANL Papers

Rider, William J., and Douglas B. Kothe. “Reconstructing volume tracking.” Journal of computational physics 141, no. 2 (1998): 112-152. (2426 Citations)

Grinstein, Fernando F., Len G. Margolin, and William J. Rider, eds. Implicit large eddy simulation. Vol. 10. Cambridge: Cambridge university press, 2007. (1179 Citations)

Drikakis, Dimitris, and William Rider. High-resolution methods for incompressible and low-speed flows. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005. (320 Citations)

Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821-841. (273 Citations)

Puckett, Elbridge Gerry, Ann S. Almgren, John B. Bell, Daniel L. Marcus, and William J. Rider. “A high-order projection method for tracking fluid interfaces in variable density incompressible flows.” Journal of computational physics 130, no. 2 (1997): 269-282. (747 Citations)

Rider, William, and Douglas Kothe. “Stretching and tearing interface tracking methods.” In 12th computational fluid dynamics conference, p. 1717. 1995. (283 Citations)

Margolin, Len G., William J. Rider, and Fernando F. Grinstein. “Modeling turbulent flow with implicit LES.” Journal of Turbulence 7 (2006): N15. (295 Citations)

Mousseau, V. A., D. A. Knoll, and W. J. Rider. “Physics-based preconditioning and the Newton–Krylov method for non-equilibrium radiation diffusion.” Journal of computational physics 160, no. 2 (2000): 743-765. (188 Citations)

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848. (96 Citations)

Rider, William J. “Revisiting wall heating.” Journal of Computational Physics 162, no. 2 (2000): 395-410. (90 Citations)

Rider, William, Douglas Kothe, S. J. A. Y. Mosso, John Cerutti, and John Hochstein. “Accurate solution algorithms for incompressible multiphase flows.” In 33rd aerospace sciences meeting and exhibit, p. 699. 1995. (119 Citations)

Knoll, D. A., W. J. Rider, and G. L. Olson. “An efficient nonlinear solution method for non-equilibrium radiation diffusion.” Journal of Quantitative Spectroscopy and Radiative Transfer 63, no. 1 (1999): 15-29. (108 Citations)

Rider, William J. “Filtering non‐solenoidal modes in numerical solutions of incompressible flows.” International journal for numerical methods in fluids 28, no. 5 (1998): 789-814. (52 Citations)

Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281. (78 Citations)

Rider, William J., and Len G. Margolin. “Simple modifications of monotonicity-preserving limiter.” Journal of Computational Physics 174, no. 1 (2001): 473-488. (49 Citations)

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, William J. Rider, Timothy G. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LA-14379. Los Alamos National Laboratory (LANL), Los Alamos, NM, 2008. (45 Citations)