tl;dr

I’m getting close to the decision to retire. I’m looking back at my career, trying to put things in perspective. By most accounts, it has been a great career, but it is also short of what it could have been. I have some genuine disappointments. I’ve seen a lot and accomplished some great things. In this Requiem, I’ve talked about my young life. I expanded and looked back at the heart of my professional career in Los Alamos, the apex of my career. Those years made me who I am as a scientist.

Now I am at the closing chapter of my career, the time I have spent at Sandia National Labs. It is too close to have a full perspective, and I cannot offer full disclosure either. I can offer a deep sense of disappointment and sadness about how it has worked out. It has not been a place to fulfill my potential, and, in retrospect, never could have. I made my choices and the results are a wonderful full life, but a loss of professional success in trade.

A Change is Needed

“Things change. And friends leave. Life doesn’t stop for anybody.”― Stephen Chbosky, The Perks of Being a Wallflower

It was October 2006, and I was convinced that things were not going to get better at Los Alamos. The new management team was awful. The new corporate partners were corrupting the essence of the Lab. My oldest child was in middle school, and I was concerned he wouldn’t do well as a teen in Los Alamos. The small-town nature of Los Alamos was starting to get under our skin too. You often would know people three or four or five ways. Work lives and personal lives were not separate, they were completely intertwined. Los Alamos was closing in on us and we needed space.

During that October, I went to a biannual conference put on by the nuclear weapons community focused on computational modeling. This time it was held in Los Alamos. The nuclear weapons world of the USA and the UK joined to talk about computers, modeling, and computational physics. It is totally my jam. Sandia National Labs sent a contingent, including my friend Tim Trucano. I approached him and a Sandia manager there (Randy Summers) with the potential of moving. They were very excited, and before long, I had an interview and shortly thereafter a job offer.

I left Los Alamos Lab on February 16, 2007, and began at Sandia the following Monday. It was MLK day, which Sandia unwisely doesn’t take as a holiday, but Los Alamos does. Maybe this little detail was a sign, and in retrospect, one I should have heeded. I was going to work in Computer Research Institute (“1400” in Sandia’s annoying schema for organizational names… no names, just numbers). My job was to work on a code development team for the MHD code “Alegra”. Alegra is a magneto-hydrodynamic code developed originally under the ASCI program. Another harbinger of issues was its lack of support from ASCI (not ASC, because a permanent program can be an initiative for so long). Alegra was then mostly supported by the US Army for analyzing things like advanced armor concepts. The reality was that I was wildly overqualified for the job. The funding and the code were horrendously constrained, and the code base was around 15 years old and all in C++.

It was a good team with good people. I let myself focus on that and approached the job with optimism and an open mind. Again, with the benefit of time, this was a naive approach. I should have been far more guarded. A big piece of this statement is that the forces that fucked up Los Alamos were present at Sandia. Sandia had always been a little corporate-managed originally by Bell Labs. In 1994 the management changed to Lockheed-Martin. The low-trust corporate management approach was firmly entrenched at Sandia.

The decline in the quality of work, science, and culture at Los Alamos was actually a national problem. Sandia was also in decline. Los Alamos took the blow of being corporately managed; Sandia was already a corporation. The management was terrible for science, but great at giving the government what it asked for. The National Labs were being pulled down by the same forces making the United States awful. The same forces we’ve seen unleashed over the past 10 years with toxic political effects. There is no escaping the decline we found ourselves in. Professionally, I should have been more pragmatic and less hopeful with my new circumstances.

A few things stand out about that time. My move got swallowed by the housing crisis of 2007 and we lost our shirt in real estate selling in Los Alamos. If we’d been three months earlier, we would have escaped, and been about $100,000 better off.

The director of our Center at Sandia also moved from Los Alamos almost at the same time as me. James Peery had started at Sandia then moved to Los Alamos for a time to run the ASC program. He was also on a corporate bid team to manage Los Alamos that did not get the job. He moved to a job managing the Center where he had started at Sandia. Unlike me, Sandia was perfectly suited for him. James is now the Laboratory Director at Sandia, retiring soon.

A few things were evident upon my arrival. A lot of administrative details were immensely better at Sandia. For example, one of the worst things about Los Alamos was travel reimbursement. At Sandia, this process was fast and seamless. The timecard application was great, too. It would have been great, except Sandia is a complete asshole to employees about some auxiliary regulations around the travel credit card. Sandians are obsessive followers of rules and regulations. It is an engineering lab and small “c” conservative organization from top to bottom.

Another difficult thing is Sandia’s devotion to information control and the practice of “need-to-know”. This is baked into their culture. Stepping back from this, one can see that Sandia was corporate in every way, and did mundane bullshit far better than Los Alamos. Thus, in every way that does not matter to National Security Science, it was an improvement. The engineering at Sandia is repressively backward and grounded in the past. Science at Sandia is peripheral and always counterculture. I have learned that culture is almost immutable, and Sandia’s culture does not fit me at all.

My take on Sandia is going to be quite harsh, so I’ll disarm this assessment a bit. On a broader scale, Sandia is a great employer. Most people would be lucky to work there. The reasons for this are twofold: First, most employers in the USA are terrible to employees, and I had been gifted an incredible experience at Los Alamos to start my career. The current Los Alamos and Sandia never stood a chance at living up to that start of my career. Secondly, the Labs have declined greatly largely due to the forces that are sieging America at large. These forces are bipartisan. Distrust and bullshit are coming from both the left and the right. Therefore, a significant drop in workplace satisfaction occurred when I left Los Alamos. Another drop would await me at Sandia. The thing that meant the most about Los Alamos was the culture. I fit into Los Alamos culture, and Sandia’s culture is unnatural for me.

Another red flag with my hire at Sandia was my ranking as a staff member. There were limits in the level I could be hired. These limits existed because they’d been abused in the past with a political hire. I would have to wait a few years to be promoted. Still, there are shades of lack of recognition professionally by Sandia. This theme is still rife today. Back in 2007, I was approaching the job with possibility and generosity. That spirit was not returned by Sandia. Sandia’s ethos was far different from Los Alamos. This was a foreign culture and not one that I was suited for.

A couple years ago, I had an epiphany about this contrast. I was at a conference. There was a contingent of Los Alamos staff there along with my coworkers at Sandia. During the meeting, I noticed that my behavior around the Los Alamos people was completely different than the Sandia ones. The people from Los Alamos were not old friends or people I knew well. In reality, I knew the Sandians better. I realized that I was “code-switching.” It made me realize that the culture and environment at Sandia were oppressive to me. The self I was with Los Alamos staff was far closer to the real me. With Sandia’s staff, I buried and censored myself. This was an absolutely jarring realization. The conclusion was that at Sandia, I could not be myself, either personally or professionally.

So as 2007 sped along, I worked to fit in and do my work. I wanted to make this succeed and I gave Sandia a lot of space. I gave it a lot more space and the benefit of the doubt than it deserved.

On the Alegra team, we did some good work despite all of this. We did run into a crisis early in my Sandia career. The ASC funding was being dropped even further, and the Army sponsorship was at risk. We needed to produce a code that was more robust and handled difficult situations better. The gauntlet was thrown down: either we improve, or Army funding would shrink until it was gone. The project would die. I immediately set about to devise a strategy to solve the problem. It was all hands on deck. No solution was off the table. In the end, we fixed the code with success that exceeded our wildest expectations. This effort remains today as the highlight of my time at Sandia. This was as good as Sandia would ever get for me. It was some really great work.

When I compare this with what I did at Los Alamos, this makes that highlight almost seem tragic. It barely held a candle to what I achieved in Los Alamos.

“Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. About the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do.” ― Steve Jobs

This quote is from an Apple marketing pitch. It describes the best of Los Alamos and captures the ideals of that Lab. If I took the quote and wrote the exact opposite, it would capture the worst of Sandia (albeit slightly unfairly).

This is Not the Right Place for Me

“An eye for an eye, and the whole world would be blind.” ― Kahlil Gibran

Recently, the focus of how much tragedy has become evident. In many respects, what stands out at Sandia is a lack of respect and use of expertise. Throughout my career, I have developed extensive expertise and Sandia simply ignores it. That expertise is one of the most important parts of Stockpile Stewardship. I have a unique knowledge of modeling and simulation—specifically the codes, methods and models used to do it. Part of this can easily be seen in how retirees are treated by the Labs. At Sandia, the prevailing attitude is “Don’t let the door hit you on the way out.” This contrasts with Los Alamos, where retirees and their knowledge are welcome and ever-present. Right now, it is these aspects that exemplify how I see Sandia. The core of my professional identity and accomplishments are not valuable to the institution. I do not feel welcome or even useful when it is obvious I should be.

There is so much more that feels bad. I put a great deal of goodwill into my first few years at Sandia. My inability to bring my real self to work began to get under my skin. (Actually, it influenced what I got on my skin too; more on that later.) The underlying Sandia culture is buttoned up and uptight as hell. It is not a place where free thought and debate happen. Even worse, it is a place that does not feel accepting of differences. While I am a middle-aged cis-gendered white man, I am not typical in many ways. None of those differences feel accepted openly. The workplace feels confined and truly limited in acceptance. I can’t bring either my personal or professional self to work. I am careful to see this feeling as a balance of intrinsic cultural differences from the shitshow of today’s America. I am sure it is a mix of innate culture and current trends.

One of the hardest aspects of working at Sandia was determining a professional development path. Why work hard at developing myself when Sandia already could not use my skills? When I walked in the door in 2007, my skills already exceeded what Sandia could use. In retrospect, I could have worked harder and hid myself at work. It would have been valuable at the office, but harmful to my being. Thus, a plan was hatched to work on my writing with focus.

This was the genesis of my blog. I would write regularly and publish the writing to be seen by others. Writing to be read by others is essential. In retrospect, the blog was one of the best things I did while at Sandia. It also made me a better scientist. Writing is thinking deeply. I would write about presentations I would give, and I approached those talks having thought deeply about the topics. In the end, thinking deeply was viewed as counter-productive unless those thoughts aligned with Sandia’s views.

Unfortunately, the blog ended up being the worst thing that happened to me at Sandia, too. The way I stopped writing exemplified the worst of the Lab but aligned perfectly with its culture. In a way, I should have seen this outcome coming at me clearly. Instead, I was optimistic and trusting. That trust was never returned. I hate saying this, but events clearly point to the worst possible reading of how the blog ended.

This Will Not End Well

In 2013, I turned 50 and started to exhibit many signs of the canonical midlife crisis. Today I see it differently, but one of the signs of the crisis was getting my first tattoo in October. Since then, I’ve gotten 25 more. They fall into some general themes: my science, primal imagery, and philosophical expressions of love and freedom. I’ve come to realize that the tattoos were a personal expression in the environment that limited it. The tattoos were also very essentially modern and fit to Albuquerque culture. The irony is that Sandia culture is orthogonal to Albuquerque culture. I was not going to be stuck into the Sandia straightjacket. I wanted to express myself freely and fully. By putting the expression on my body, I would not be silenced. I don’t think it was a conscious thought, but rather a reaction to the day job with my real self. As the past decade has unfolded, my true self and my work self have diverged rather precipitously. The subtle differences I expressed moved toward a primal scream.

I wrote this blog consistently from 2014-2018. I stopped writing it because I was given no choice in 2018. In a culturally consistent passive-aggressive manner, I was forced to stop. The form of this force was a Sandia ethics investigation. It came from an anonymous source (although I am relatively certain of the identity of the source). The investigation and charges were absurd. The entire blog was done in plain view and as a part of my professional development plan. It was done with permission. I would give the link to my blog at the end of my professional talks. The blog was good for my professional skills and performance. It was only bad for professional success where blind obedience to power is demanded. In a healthy culture, it would have been celebrated. In an unhealthy culture, the blog was a threat.

The problem was that I expressed views that were not the same as my management. I was part of a rather ill-conceived national program that was contributing to the decline in American scientific supremacy. I said as much in the blog. Worse yet, I had an audience of peers, thus I was dangerous. I could be shut up through force—and I was. This program was big money, and in today’s world is all about money. Money is power and truth is determined by power. Opposing the monied interests is dangerous.

I stood down so I could continue to support my family and my life. Intellectual honesty and debate about what are best are not on the menu today. The people in charge of the Labs and our science funding are not open to questioning their priorities. This lack of debate is part of what is fueling the decline of American supremacy in science.

I licked my wounds. The ethics investigation is the worst blight in my entire professional life. It may be the worst thing of my entire adult life, but I’ll admit this means I have a charmed life.

As I attempted to recover, the pandemic arose. The country and the world shut down. We all worked from home for months as most of my work went online. I discovered a massive relief in this arrangement. Even in this lockdown, I felt freedom. I did not have to go to work and put on a disingenuous mask every day. It made me realize how incredibly unhappy Sandia had made me. In the past couple of years, the depth of my dismay professionally has deepened. It has transitioned to a mourning of opportunity lost and disappointment.

“The wound is the place where the Light enters you.”— Jalal ad-Din Muhammad ar-Rumi

Prospectus and Path Ahead

Hindsight is usually a biased way to look at things. The reasons for my move to Sandia and Albuquerque were varied. The professional reasons were naive and quixotic. I approached the new Lab with optimism and hope that was not grounded in actual possibility. I pushed away many obvious signs of disaster for years. I should have been more guarded and far less trusting. That said, I moved for reasons in my personal life. I moved for my family. It was better professionally for my wife and offered better opportunities for my children. We had aging parents to care for.

Each of these concerns played out differently, but all were valid. For my own part, I have changed over these years in many interesting and exciting ways. In sum, I would not trade the benefits and advances in myself and my personal life for professional success. Nonetheless, professional success was sacrificed in the move.

Professionally, Sandia has been just okay. By my own standards, my professional accomplishments have been a severe disappointment. The way my blog was treated figures heavily into my assessment. It was a huge insult and attack on me personally. The lack of professional respect is palpable. The lack of use for my vast professional knowledge and skills is appalling.

This is not to say one cannot be successful at Sandia. It is to say that *I* would never succeed at Sandia. It is not the right place for me to succeed. In the same breath, I can succeed as a person, where I am now. It could also be true that Los Alamos might have been a worse place for that personal success. You really can’t have it all.

Today, I feel like my work has mostly been a waste of my time. It is easy to see how I could have used my skills and knowledge better than they were at Sandia. Perhaps in closing, this is my challenge for the future: to find a valuable path for contribution suitable for my precious time left. I did give Sandia a chance, and that effort was not met nor rewarded in kind.

I have not decided when to retire, or what I will do after retirement. What will happen is taking shape. Felicia is retired and I’m learning from her experience. It is just clear to me that it is around the corner. While my professional life at Sandia was substandard, my life is good. I would not trade the success in my personal life for the sort of professional success I would aspire toward. Perhaps the lesson is a sort of balance to life. Maybe you simply can’t have it all and choices are made. I made mine. I stand by the outcomes.

Finally, I want to give endless love and gratitude to those who have enriched my life and been my friends. I have known many wonderful people who have made my time at work vastly better. Each of them has helped me be a better person and taught me so much. Nothing would have been as good without them. I greatly appreciate Meera Collier for graciously editing my writing, and helping to make it better than I could manage myself.

“Yesterday I was clever, so I wanted to change the world. Today I am wise, so I am changing myself.”— Jalal ad-Din Muhammad ar-Rumi

Important Papers Written While At Sandia

Banks, Jeffrey W., T. Aslam, and William J. Rider. “On sub-linear convergence for linearly degenerate waves in capturing schemes.” Journal of Computational Physics 227, no. 14 (2008): 6985-7002.

Mattsson, Ann E., and William J. Rider. “Artificial viscosity: back to the basics.” International Journal for Numerical Methods in Fluids 77, no. 7 (2015): 400-417.

Rider, W. J., E. Love, M. K. Wong, O. E. Strack, S. V. Petney, and D. A. Labreche. “Adaptive methods for multi‐material ALE hydrodynamics.” International Journal for Numerical Methods in Fluids 65, no. 11‐12 (2011): 1325-1337.

Robinson, Allen, Thomas Brunner, Susan Carroll, Richard Drake, Christopher Garasi, Thomas Gardiner, Thomas Haill et al. “ALEGRA: An arbitrary Lagrangian-Eulerian multimaterial, multiphysics code.” In 46th aiaa aerospace sciences meeting and exhibit, p. 1235. 2008.

Rider, William, Walt Witkowski, James R. Kamm, and Tim Wildey. “Robust verification analysis.” Journal of Computational Physics 307 (2016): 146-163.

Rider, William J. “Reconsidering remap methods.” International Journal for Numerical Methods in Fluids 76, no. 9 (2014): 587-610.

Love, E., William J. Rider, and Guglielmo Scovazzi. “Stability analysis of a predictor/multi-corrector method for staggered-grid Lagrangian shock hydrodynamics.” Journal of Computational Physics 228, no. 20 (2009): 7543-7564.

Hills, Richard Guy, Walter R. Witkowski, Angel Urbina, William J. Rider, and Timothy Guy Trucano. Development of a fourth generation predictive capability maturity model. No. SAND2013-8051. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2013.

Barlow, Andrew J., Pierre-Henri Maire, William J. Rider, Robert N. Rieben, and Mikhail J. Shashkov. “Arbitrary Lagrangian–Eulerian methods for modeling high-speed compressible multimaterial flows.” Journal of Computational Physics 322 (2016): 603-665.

Alexander, Francis, Ann Almgren, John Bell, Amitava Bhattacharjee, Jacqueline Chen, Phil Colella, David Daniel et al. “Exascale applications: skin in the game.” Philosophical Transactions of the Royal Society A 378, no. 2166 (2020): 20190056.

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, William J. Rider, Timothy G. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LA-14379. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States), 2008.

Yanilkin, Yury V., Evgeny A. Goncharov, Vadim Yu Kolobyanin, Vitaly V. Sadchikov, James R. Kamm, Mikhail J. Shashkov, and William J. Rider. “Multi-material pressure relaxation methods for Lagrangian hydrodynamics.” Computers & Fluids83 (2013): 137-143.

Weirs, V. Gregory, James R. Kamm, Laura P. Swiler, Stefano Tarantola, Marco Ratto, Brian M. Adams, William J. Rider, and Michael S. Eldred. “Sensitivity analysis techniques applied to a system of hyperbolic conservation laws.”

tl;dr.

The practice of code verification has focused on finding bugs in codes. This is grounded in the proving that a code is correctly implementing a method. While this is useful and important, it does not inspire. Code verification can be used for far more. It can be the partner to method development and validation or application assessment. It can also provide expectations for code behavior and mesh requirements on applications. Together these steps can make code verification more relevant and inspiring. It can connect it to important scientific and engineering work pulling it away from computer science.

When you can measure what you are speaking about, and express it in numbers, you know something about it.

– Lord Kelvin

My Connection to Code Verification

Writing about code verification might seem like a scheme to reduce my already barren readership. All kidding aside, code verification is not the most compelling topic for most. This includes people making their living writing computational solvers. For me it is a topic of much greater gravity. While I am inspired by the topic, most people are not. The objective here is to widen its scope and importance. Critically, I have noticed the problems getting worse as code verification seems to fade from serious attention. This all points to a need for me to think about this topic deeply. It is time to consider a change to how code verification is talked about.

“If you can not measure it, you can not improve it.”

– Lord Kelvin

My starting point for thinking about code verification is to look at myself. Code verification is something I’ve been doing for more than 30 years. I did it before I knew it was called “code verification.” Originally, I did it to assist my development of improved methods in codes I worked on. I also used it to assure that my code was correct, but this was secondary. This work utilized test problems to measure the correctness and more importantly the quality of methods. As I continued to mature and grow in my scientific career I sought to enhance my craft. The key aspect of growth was utilizing verification to exactly measure method character and quality. It was through verification that I understood if a method passed muster.

“If failure is not an option, then neither is success.”

― Seth Godin

Eventually, I developed new problems to more acutely measure methods. I also developed problems to break methods and codes. When you break a method you help define its limitations. Over time I saw the power of code verification as I practiced it. This contrasted to how it was described by V&V experts. The huge advantage and utility of code verification I found in method development was absent. Code verification was relegated to correctness through code bug detection. In this mode code verification is a spectator to the real work of science. I know it can be so very much more.

“I have been struck again and again by how important measurement is to improving the human condition.”

– Bill Gates

The Problem with Code Verification

In the past year I’ve reviewed many different proposals in computational science. Almost all of them should be utilizing code verification integrally in their work. Almost all of them failed to do so. At best, code verification is given lip service because of proposal expectations. At worst it is completely ignored. The reason is that code verification does not set itself as a serious activity for scientific work. It is viewed as a trivial activity beneath mention in a research proposal. The fault lies with the V&V community’s narrative about it. (I’ve written before on the topic generally https://williamjrider.wordpress.com/2024/08/14/algorithms-are-the-best-way-to-improve-computing-power/)

“Program testing can be used to show the presence of bugs, but never to show their absence!”

― Edsger W. Dijkstra

Let’s take a look at the narrative chosen for code verification more closely. Code verification is discussed primarily as a manner to detect bugs in the code. The bugs are detected when the code does not act as a consistent solution of the governing equations in the manner desired. This comes when the exact solution to those governing equations does not match the order of accuracy designed for the method. This places code verification as part of software development and quality. This is definitely an important topic, but far from a captivating one. At the same time code verification is distanced from math, physics and aapplication space engineering. Thus, code verification does not feel like science.

This is the disconnect. To be focused upon in proposals and work code verification needs to be part of a scientific activity. It simply is not one right now. Of all the parts of V&V, it is the most distant from what the researcher cares about. More importantly, this is completely and utterly unnecessary. Code verification can be a much more holistic and integrated part of the scientific investigation. It can span all the way from software correctness to physics and application science. If the work involves development of better solution methodology, it can be the engine of measurement. Without measurement “better” cannot be determined and is left to bullshit and bluster.

“Change almost never fails because it’s too early. It almost always fails because it’s too late.”

― Seth Godin

What to do about it?

The way forward is to expand code verification to include activities that are more consequential. To constructively discuss the problem, the first thing to recognize that V&V is the scientific method for computational science. It is essential to have correct code. The software correctness and quality aspects of code verification remain important. If one is doing science with simulation, the errors made in simulation are more important. Code verification needs to contribute to error analysis and minimization. Another key part of simulation are choices about the methods used. Code verification can be harnessed to serve better methods. The key in this discussion is that the additional tasks are not discussed in what code verification is. This is an outright oversight.

Appreciate when things go awry. It makes for a better story to share later.

― Simon Sinek

Let’s discuss each of these elements in turn. First we should get to some technical details of code verification practice. The fundamental tool in code verification is using exact solutions to determine the rate of convergence of a method in a code. The objective is to show the code implementation produces the theoretical order of accuracy. This is usually accomplished by computing errors on different meshes.

The order of accuracy comes from numerical analysis of the truncation errors of a method. It is usually takes the form of a power of the mesh size. For example a first order method the error is proportional to the mesh size. For a second order method the error depends on the square of the mesh size. This all follows from the analysis and has the error vanishing as the mesh size goes to zero (see Oberkampf and Roy 2010)

The grounding of code verification is found in the work of Peter Lax. He discovered the fundamental theorem of numerical analysis (Lax and Richtmyer 1956). This theorem says that a method is a convergent approximation of the partial differential equation if it is consistent and is stable. Stability comes from getting an answer that does not fall apart into numerical garbage. Practically speaking, stability is assumed when the code produces a credible answer to problems. The trick of consistency is that the method reproduces the differential equation plus an ordered remainder. Now the trick of verification is that you invert this and use a convergent sequences to infer consistency. This is a bit of a leap of faith.

“Look for what you notice but no one else sees.”

― Rick Rubin

The additional elements for verification

The most important aspect to add to code verification is stronger connection to validation. Numerical error is an important element in validation and application results. Currently code verification is divorced from validation. This makes it ignorable in the scientific enterprise. To connect better, the errors in verification work need to be used to understand mesh requirements for solution features. This means that the exact solutions used need to reflect true aspects of the validation problem.

Current verification practice pushes this activity into the background of validation. In doing “bug hunting” code verification, the method of manufactured solutions (MMS) is invaluable. The problem is that MMS solutions usually bear no resemblance to validation problems. For people concerned with real problems MMS problems have no interest, nor guidance for their solutions. Instead verification problems should be chosen that feature phenomena and structures like those validated. Then the error expectations and mesh requirements can be determined. Code verification can then be used as simple pre-simulation work before validation ready calculations are done. Ultimately this will require the development of new verification problems. This is deep physics and mathematical work. Today this sort of work is rarely done.

The next big change in code verification is connecting code verification more actively to method-algorithm research. Code verification can be used to measure the error of a method directly. Again this requires a focus on error instead of convergence rate. The convergence rate is still relevant and needs to be verified. At the same time methods with the same convergence rate can have greatly different error magnitudes. For more realistic problems the order of accuracy does not determine the error. It has been shown that low order methods can out perform higher order methods in terms of error (see Greenough and Rider 2005).

“There is no such thing as a perfect method. Methods always can be improved upon.”

–  Walter Daiber

In all aspects of developing a method code verification is useful. The base of making sure the implementation is correct remains. The additional aspect that I am suggesting is the ability to assess the method dynamically. This should be done on a wide range of problems biased toward application-validation inspired problems. In terms of making this activity supported by those doing science, the application-validation inspired problems are essential. This is also where code verification fails most miserably. The best example of this failure can be found in shock wave calculations.

“If you can’t measure it, you can’t change it.”

– Peter Drucker

Let’s take a brief digression to how verification currently is practiced in shock wave methods. Invariably the only time you see detailed quantitative error analysis is on a smooth differentable prroblem. This problem has no shocks and can be used to show a method has the “right” order of accuracy. This is expected and common. The only value is the demonstration that a nth order method is indeed nth order. It has no practical value for the use of the codes.

“Measure what is measurable, and make measurable what is not so.”

– Galileo Galilei

Once a problem has a shock in it, the error analysis and convergence rates disappear from the work. Problems are only compared in the “eyeball norm” to an analytic or high resolution solution. The reason for this is that the convergence rate with a discontinuity is one or less. The reality being ignored is that error can be very different (see the paper by Greenough and Rider 2005). When I tried to publish a paper that used errors and convergence rates to assess the method with shock, the material needed to be deleted. As the associate editor told me bluntly, “if you want to publish this paper get that shit out of the paper!” (see Rider, Greenough and Kamm 2007)

Experts are the ones who think they know everything. Geniuses are the ones who know they don’t

― Simon Sinek

Why is this true? Part of the reason is the belief that the accuracy does not matter any longer. The failure is to recognize how different the errors can be. This has become accepted practice. Gary Sod introduced the canonical shock tube problem that bears his name. Sod’s shock tube has been called the “Hello World” problem for shock waves. In Sod’s 1978 paper the run time of different methods was given, but errors were never shown. The comparison with analytical solution to the problem was qualitative, the eyeball norm. Subsequently, this became the accepted practice. Almost no one ever computes the error or convergence rate for Sod’s problem or any other shocked problem.

“One accurate measurement is worth a thousand expert opinions.”

– Grace Hopper

As I have written and shown recently this is a rather profound oversight. The importance of the error level for a given method is actually far greater if the convergence rate is low. The lower the convergence rate, the more important the error is. Thus we are not displaying the errors created by methods in the conditions where it matters the most. This is a huge flaw in the accepted practice and a massive gap in the practice of code verification. It is something that needs to change.

“The Cul-de-Sac ( French for “dead end” ) … is a situation where you work and work and work and nothing much changes”

― Seth Godin

My own practical experience speaks volumes about the need for this. Virtually every practical application problem I have solved or been associated with converges at low order (first order or less). The accuracy of the methods under these circumstances mean the most to the practical use of simulation. Because of how we currently practice code verification applied work is not impacted. There is a tremendous opportunity to improve calculations using code verification. As I noted a couple of blog posts ago, the lower the convergence rate, the more important the error is (https://williamjrider.wordpress.com/2024/08/14/algorithms-are-the-best-way-to-improve-computing-power/). A low error method can end up being orders of magnitude more efficient. This can only be achieved if the way code verification is done and its scope increase. This will also draw it together with the full set of application and validation work.

More related content (https://williamjrider.wordpress.com/2017/12/01/is-the-code-part-of-the-model/, https://williamjrider.wordpress.com/2017/10/27/verification-and-numerical-analysis-are-inseparable/, https://williamjrider.wordpress.com/2015/01/29/verification-youre-doing-it-wrong/, https://williamjrider.wordpress.com/2014/05/14/important-details-about-verification-that-most-people-miss/,

https://williamjrider.wordpress.com/2014/01/31/whats-wrong-with-how-we-talk-about-verification/

“If it scares you, it might be a good thing to try.”

– Seth Godin

Roache, Patrick J. Verification and validation in computational science and engineering. Vol. 895. Albuquerque, NM: Hermosa, 1998.

Oberkampf, William L., and Christopher J. Roy. Verification and validation in scientific computing. Cambridge university press, 2010.

Lax, Peter D., and Robert D. Richtmyer. “Survey of the stability of linear finite difference equations.” In Selected Papers Volume I, pp. 125-151. Springer, New York, NY, 2005.

Roache, Patrick J. “Code verification by the method of manufactured solutions.” J. Fluids Eng. 124, no. 1 (2002): 4-10.

Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281.

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848.

Sod, Gary A. “A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws.” Journal of computational physics 27, no. 1 (1978): 1-31.

tl;dr; Algorithms shape our world today. When a new algorithm is created it can transform a computational landscape. These changes happen in enormous leaps that take us by surprise. The latest changes in the artificial intelligence are the result of such a breakthrough. It is unlikely to be followed by another breakthrough soon reducing the seeming pace of change. For this reason the threats of doom and vast wealth are overblown. If we want more progress it is essential to understand how such breakthroughs happen and their limits.

“The purpose of computing is insight, not numbers.”

– Richard Hamming

We live in the age of the algorithm. In the past ten years this has leapt to the front of mind with social media and the online world. It has actually been true ever since the computer took hold of society. This began in the 1940’s with the first serious computers, and numerical mathematics. A new improved algorithm always drives the use of the computer forward as much as hardware. What people do not realize is that the improvements that get noticed are practically quantum in change. These algorithms get our attention.

“I am worried that algorithms are getting too prominent in the world. It started out that computer scientists were worried nobody was listening to us. Now I’m worried that too many people are listening.”

– Donald Knuth

Now that the internet has become central to our lives we need to understand this. One reason is understanding how algorithms create value for business and stock market valuations. How these sorts of advances fool people on the pace of change? We should also know how this breakthroughs are made. We need to understand how likely we are to see progress? How can we create an environment where advances are possible? How the way we fund and manage work actually destroys the ability to continue progress?

“You can harvest any data that you want, on anybody. You can infer any data that you like, and you can use it to manipulate them in any way that you choose. And you can roll out an algorithm that genuinely makes massive differences to people’s lives, both good and bad, without any checks and balances.”

– Hannah Fry

Two examples come to mind in recent year to illustrate these points. The first is the Google search algorithm, pagerank. The second is the transformer, which elevated large language models to the forefront of the public’s mind in the last two years. What both of these algorithms illustrate clearly is the pattern for algorithmic improvement. A quantum leap in performance and behavior followed by incremental changes. These incremental changes are basically fine tuning and optimization. They are welcome, but do not change the World. The key is realizing the impact of the quantum leap from an algorithm and putting it into proper perspective.

Google is an archetype

Google transformed search and the internet and ushered algorithms into the public eye. Finding things online used to be torture as early services tried to produce a “phone book” for the internet. I used Alta Vista, but Yahoo was another example. Then Google appeared and we never went back. Once you used Google the old indexes of the internet were done. It was like walking through a door. You shut the door and never looked back. This algorithm turned the company Google into a verb, household name and one of the most powerful forces on Earth. Behind it was an algorithm that blended graph theory and linear algebra into an engine of discovery. Today’s online world and its software are built on the foundation of Google.

“The Google algorithm was a significant development. I’ve had thank-you emails from people whose lives have been saved by information on a medical website or who have found the love of their life on a dating website.”

– Tim Berners-Lee

Google changed the internet introduced search and demonstrated the power of information. All of a sudden information was unveiled and shown to be power. Google unleashed the internet into a transformative engine for business, but society as well. The online world we know today owes its existence to Google. We need to acknowledge that Google today is a shadow of the algorithm of the past. Google has become a predatory monopoly and the epitome of “enshitification” of the internet. This is the process of getting worse over time. This is because Google is searching for profits over performance. Instead of giving us the best results they are selling spaces for money. This process is repeated across the internet undermining the power of the algorithms that created it.

“In comparison, Google is brilliant because it uses an algorithm that ranks Web pages by the number of links to them, with those links themselves valued by the number of links to their page of origin.”

– Michael Shermer

The Transformer and LLMs

The next glorious example of algorithmic power comes from Google (Brain) with the Transformer. Invented at Google in 2017 this algorithm has changed the world again. With a few tweaks and practical implementations OpenAI unleashed ChatGPT. This was a large language model (LLM) that ingested large swaths of the internet to teach it. The LLM can then produce results that were absolutely awe-inspiring. This was especially in comparison to what came before where suddenly the LLM could produce almost human like responses. Granted this is true if that human was a corporate dolt. Try to get ChatGPT to talk like a real fucking person! (just proved a person wrote this!)

“An algorithm must be seen to be believed.”

– Donald Knuth

These results were great even after OpenAI lobodomized ChatGPT with reinforcement learning that kept it from being politically incorrect. The LLM’s won’t curse or say racist or sexist stuff either. In the process the LLM becomes as lame as a conversation with your dullest coworker. The unrestrained ChatGPT was almost human in creativity, but also prone to sexist, racist and hate speech (like people). It is amazing to know how much creativity was sacrificed to make it corporately acceptable. It is worth thinking about and how this reflects on people. Does the wonder of creativity depend upon accepting our flaws?

Under the covers in the implementation of the foundation models at the core of ChatGPT is the transformer. The transformer has a couple of key elements. One if the ability to devour data in huge chunks perfectly fitting for modern GPU chips. This has allowed far more data to be used and transformed NVIDIA into a mulit-trillion dollar company overnight. This efficiency is only one of the two bits of magic. The real magic is the attention mechanism. This is what the LLM takes as instructions for its results. The transformer allows longer more complex instructions to be given. It also allows multiple instructions to guide its output. The attention mechanism has led to fundamentally different behavior from the LLMs. Together these elements demonstrate the power of algorithms.

“Science is what we understand well enough to explain to a computer. Art is everything else we do.”

– Donald Knuth

The real key to LLMs is NOT the computing available. A lot of capable computing helps and makes it easier. The real key to the huge leap in performance is the attention mechanism that changed the algorithm. This produced the qualitative change in how LLMs functioned. This produced the sort of results that made the difference. It was not the computers; it was the algorithms!

The world collectively lost their shit when ChatGPT went live. People everywhere freaked the fuck out! As noted above the impact could have easily been more profound without the restraint offered by reinforcement learning. Nonetheless feelings were unleashed that felt like we were on the cusp of exponential change. We are not. The reason why we are not is something key about the change. The real difference with these new LLMs was all predicated on the transformer algorithm’s character. Unless the breakthroughs of the transformer are repeated with new ideas, the exponential growth will not happen. Another change will happen, but it is not likely for a number of years from now.

A look at the history of computational science unveils that such changes happen more slowly. One cannot count on these algorithmic breakthroughs. They happen episodically with sudden leaps followed by periods of fallow growth. The fallow periods are optimization of the breakthrough and incremental change. As 2024 plays out I have become convinced that LLMs are like this. There will be no exponential growth into general AI that people fear. The transformer was the breakthrough and without another breakthrough we are on a pleateau of performance. Nonetheless like Google, ChatGPT was a world changing algorithm. Until a new algorithm is discovered, we will be on a slow path to change.

“So my favorite online dating website is OkCupid, not least because it was started by a group of mathematicians.”

– Hannah Fry

Computational Science and Quantum Leaps from Algorithms

To examine what this sort of algorithmic growth in performance we can look at examples from classical comptuational science. Linear algebra is an archetype of this sort of growth. Over a span of years from 1947 to 1985, the algorithmic performance matched the performance gains from hardware. This meant that Moore’s law for hardware was amplified by better algorithms. Moore’s law is the result of multiple technologies working together to create the exponential growth.

In the 1940’s linear algebra worked using dense matrix algorithms that scaled cubically with problem size. As it turned out most computational science applications were sparse structured matrices. These could be solved more efficiently with quadratic scaling. This was a huge difference. For a system with 1000 equations this is the difference of a million instead of a billion in terms of the work done and storage taken on the computer. Further advances happened with Krylov algorithms and ultimately multigrid where the scaling is linear (1000 in the above example). These are all huge speedups and advances. A key point is that the changes above occurred over the span of 40 years.

The nature of these changes is quantum in nature where the performance of the new algorithm leaps orders of magnitude. The new algorithm allows new problems to be solved and is efficient in ways the former algorithm is incapable of. This is exactly like what happened with the transformer. In between these advances the new algorithm is optimized and gets better. It does not change the fundamental performance. Nothing amazing happens until something is introduced that acts fundamentally differently. This is why there is a giant AI bubble. Unless another algorithmic advance is made, the LLM world will not change dramatically. The power and fears around AI is overblown. People do not understand that this moment is largely algorithmically driven.

These sorts of leaps in performance are not limited to linear algebra. In optimization a 1988 study showed a 43,000,000 times improvement in performance over a 15 year period. Of this improvement 1000 was due to computer improvements, but 43,000 was due to better algorithms. Another example is the profound change in hydrodynamic algorithms based on transport methods. The introduction of “limiters” in the early 1970’s allowed second-order methods to be used for the most difficult problems. Before the limiters the second-order methods produced oscillations that resulted in unphysical results. The difference was transformative. I have recently shown that the leap in performance is about a factor of 50 in three dimensions. Moreover the results also compare to the basic physical laws in ways the first-order methods cannot produce.

How do algorithms leap ahead?

“This is the real secret of life — to be completely engaged with what you are doing in the here and now. And instead of calling it work, realize it is play.” ― Alan Watts

Where do these algorithm breakthroughs come from? Some come out of pure inspiration where someone sees an entirely different way to solve a problem. Others come through the long slog through seeking efficiency. The deep analysis yields observations that are profound and lead to better approaches. Many are pure inspiration coming out of giving people the space to operate in a playful space. This playful space is largely absent in the modern business or government world. To play is to fail and to fail is to learn. Today we have everything planned and everyone should know that breakthroughs are not planned. We cannot play; we cannot fail; we cannot learn; breakthroughs are impossible.

“Our brains are built to benefit from play no matter what our age.”

– Theresa A. Kestly

The problems with algorithm advancements are everywhere in today’s environment. Lack of fundamental trust leads to constrained planning and lack of risk taking. Worse yet, failure is not allowed as the essential engine of learning and discovery. This sort of attitude is pervasive in the government and corporate system. Basic and applied research is both lacking funding and that funding is not free to go after problems.

In the corporate environment the breakthroughs often do not benefit the company where things are discovered. The transformer was discovered by Google (Brain), but the LLM breakthrough was made by OpenAI. Its greatest beneficiary is Google’s rival microsoft. A more natural way to harness the power of innovation is the government funding. There laboratories and universities can produce work that is in the public domain. At the same time the public domain is harmed by various information hiding policies and lack of transparency. We are not organized for success at these things as a society. We have destroyed most of the engines of innovation. Until these engines are restarted we will live in a fallow time.

“There is no innovation and creativity without failure. Period.”

― Brene Brown

I see this clearly at work. There we argue about whether to keep using 30, 40 and 50 year old algorithms rather than invest in the state of the art. They then convince themselves that it is good because their customers like the code. The code is modern because it is written in C++ instead of Fortran. The results feel good simply because they use the most modern computing hardware. Our “leadership” does not realize that this approach is getting substandard return on investment. If the algorithms were advancing the results would be vastly improved. Yet, there is little or no appetite to develop new algorithms or invest in research in finding them. This sort of research is too failure prone to fund.

“Good scientists will fight the system rather than learn to work with the system.”

– Richard Hamming

Page, Lawrence. The PageRank citation ranking: Bringing order to the web. Technical Report, 1999.

Vaswani, A. “Attention is all you need.” Advances in Neural Information Processing Systems (2017).Vaswani, A. “Attention is all you need.” Advances in Neural Information Processing Systems (2017).

Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821-841.

Boris, Jay P., and David L. Book. “Flux-corrected transport. I. SHASTA, a fluid transport algorithm that works.” Journal of computational physics 11, no. 1 (1973): 38-69.

“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.”

― Benjamin Franklin,

The Trigger

The other day I headed into work for a face to face meeting. The meeting was an hour long. It was interesting and thought provoking. It also showed the utter disregard for the cost of actions in two ways. This meeting would cost me far more than an hour due to outright stupidity and lack of proper consideration. This sort of stupidity is rampant accross society today. It is destroying productivity, research and threatening our Nation’s security..

After the meeting I immediately got caught in a traffic jam trying to leave the Air Force base where my Lab is. This happens all too frequently and is maddening. The traffic was jammed up for more than an hour. It was also lunch hour, so more people than usual were on the road. It was a Friday, which mitigated some of the hassle since so many people work from home or don’t work Friday.

Why did this happen?

I heard from a friend that a motorcycle had run the security checks at the gates. This prompted the guards to institute a complete lockdown. This is the safest and most secure thing to do. Can’t take a risk, right? This is done without a thought about the costs. The thousands of people working on the base are frozen in place. As I’ve learned the costs of my time at work are rather extreme, about $350/hour. So if a 1000 people like me are put out for an hour this comes to $350,000. If it is 2000 people this is $700,000.

Is this worth the expense? No.

A rational response would be for the guards to chase the interloper down and arrest them. There is no reason to close the gates down. Surely they do so out of caution, extreme caution, that permiates society today. This caution always operates on the view that no cost be considered if safety or security is at risk. This attitude is absolutely irrational. It exacts costs from society at large that actually harm safety and security in the long run. It gets to the core of why we can’t accomplish great things today.

We choose the short term appearance of safety and security. This choice is destroying our long term safety and security as I will describe next.

“You tell them – you tell them there’s a cost…..Every decision we make in life, there’s always a cost.”

― Brad Meltzer,

It is everywhere one looks

If one looks around we see a lack of progress everywhere. We can’t build anything. Public works projects take forever (the big dig in Boston, or high speed rail). Government projects are all over cost and over time. Examples abound such as the F-35. Almost across the board our essential weapon’s systems are over cost and behind schedule. Under the covers the problem is prioritization of safety and security without thought to benefit compared with cost. For me the example of what my time costs is exhibit A. That cost is driven by out of control safety and security culture.

In my own life I am confronted with lots of deep security concerns that revolve around outlandish scenarios. The scenarios are plausible, but unlikely. The security people are granted carte blanche to take mitigations for the possible impacts. The cost on time and productivity are never considered. The programs funding research and useful work simply eat the costs. Worse yet, it would seem that safety and security professionals are actually rewarded for suggesting concerns. Their recommended mitigations are never put to the test of considering the cost for the benefit gained.

We might take the recent guidance federally regarding medical devices as an archetypical example. Medical devices are becoming increasingly complex and integrating new technology. A good example is a pacemaker that has bluetooth built in. The bluetooth increases the safety and benefit for the patient (my dad has one). The device can be checked often and remotely to monitor the patient and the health of the device. Yet the paranoia about bluetooth makes this a security concern. Another great example are bluetooth enabled hearing aids, which improve life for the hard of hearing.

What person in their right mind would accept a worse pacemaker simply to satisfy outlandish security concerns? Moreover, why would an employer ask someone to do this? This is the height of absurdity. You are either demanding someone risk their health or removing them from the workforce. Frankly I’m disgusted by this choice being imposed on someone. This is an absurdly low risk threat inducing a profoundly high consequence effect. Yes, something could happen possibly, but it is fantastically unlikely. It is not worth the cost of losing the efforts of the professionals removed from the workforce.

Much of this insanity calls back to the issues of trust discussed in the last post.

“As we care about more of humanity, we’re apt to mistake the harms around us for signs of how low the world has sunk rather than how high our standards have risen.”

― Steven Pinker

The TSA is exhibit 1

The lack of cost benefit considerations is perhaps most clear with the TSA’s practices. We have a situation where some asshole tries to light his shoes on fire 20 years ago on a plane and we still waste time over it. Richard Reid, who was an idiot, tried to blow up a plane by creating a shoe bomb. A fantastically stupid plot that got him imprisoned for life. In reaction people have to take off their shoes at security as well as limiting our fluid containers to 100 ml. We keep on doing this more than 20 years later with no end in sight. The cost of these measures on society is huge while the benefit is fleeting and highly questionable.

Let’s look at the cost more closely. For me, if I take 10 trips a year the extra time for these measures is perhaps 10 minutes each way. This tallies up to 3 extra hours a year. Now multiply this by 100 million people, and my $350 per hour rate and you get to $100 billion. This is undoubtably an over estimate, but it is still a huge cost nonetheless.

The time penalty is unambiguous. 300 million hours a year comes easily to 420 human life times. We habe been doing this for 20 years, so we are rapidly coming up on wasting 10,000 human lifetimes on this moronic safety measure. This along with what is likely more than a trillion dollars in lost productivity. All of this is because we can’t manage to examine the cost versus the benefit of this measure. It is also a perfect over-reaction to the act of one complete idiot by more idiots who weaponize safety and security.

We constantly hear this din of the bullshit that safety and security can be perfect with zero incidents. What a load of shit! Merely seeking out this outcome means exacting huge costs for fleeting benefits. I’m not talking about taking wild risks, but rather operating with common sense and reducing risks. Zero risk means zero progress and infinite cost. In many ways the desire for perfect safety and security is a power grab by those employed to do this sort of work. This desire is a disservice. They should be tasked with delivering good results with reasonable costs. Today we just write them a blank check. It is no wonder we can’t get work done and projects all come in late and over budget. This attitude exacts even greater costs on our long term safety and security.

As an example of the impact of this sort of lunacy consider an example. On any given day and for that day the safest thing to do is stay home and stay inside all day. You will shield yourself from driving, traffic, getting hit by a car, getting exposed to a virus and all sorts of other dangers. You will maximize your safety for that given day. If you run your life like this every single day, you will ruin your life. You will destroy your health, be lonely and fail to live. To live requires danger and risk. It requires putting yourself out there. Why do we think this safe at all costs attitude is right for society as a whole? It is not and the costs on all of us are piling up. We are not living like we should be.

Let’s get to the root of it

I’m not naive enough to believe this attitude comes out of the blue. A big reason is the loss of trust pervading society. We have had a huge regulatory response over the last 40-50 years that is the response to the lack of trust. Corporate behavior is a major reason for this. Under the mantra of maximizing shareholder value corporations will do all sorts of horrible things. Look at facebook for an example. They will commit all manners of harm to society to maximize clicks and ad revenue. Nothing except regulation stands in the way of doing harm to make money.

A proviso in the mantra of maximizing shareholder value is doing this within the confines of the law. The problem that has arisen over the past several decades is the capture of politics by money and by money from corporations. Increasingly we see corporations or those enriched by them as defining what the laws are. They are increasingly outside the reach of the law. The judiciary is aligned with this end. Most acutely through numerous Supreme Court decisions this process is accelerating. The most infamous of these decisions is Citizen’s United, which led to vast sums of corporate money distorting our politics. The only end of this process is an acceleration of loss of trust. Without a change this will end in violence or the end of democracy, or both. The same forces are dismantling regulation which was this bulwark and response to these forces to start with.

Puts the whole onus on prevention and none of the focus on improvement and progress. Progress is the main path to both security and safety. We are rapidly devolving into a society without enough trust to allow progress. Progress under the condition of trust is the way forward. Progress in science and culture has led to a better life for all. Medicine has eased suffering and extended lives. Science has given us a myriad of wonders like air travel and the internet. We have gained equality in culture for women and the LBGTQ community. Racial discrimination has faded from centrality culturally. More progress is needed, but crucially the progress made is at risk. All of this is threatened by the forces destroying the essential trust human society depends upon.

“What is progress? You might think that the question is so subjective and culturally relative as to be forever unanswerable. In fact, it’s one of the easier questions to answer. Most people agree that life is better than death. Health is better than sickness. Sustenance is better than hunger. Abundance is better than poverty. Peace is better than war. Safety is better than danger. Freedom is better than tyranny. Equal rights are better than bigotry and discrimination. Literacy is better than illiteracy. Knowledge is better than ignorance. Intelligence is better than dull-wittedness. Happiness is better than misery. Opportunities to enjoy family, friends, culture, and nature are better than drudgery and monotony. All these things can be measured. If they have increased over time, that is progress.”

― Steven Pinker

The costs are bigger than one can imagine

If I look closely at my life I can see the real cost of all this in the decay of the American research institutions. Over the past 40 years the great government laboratories have been destroyed by this dynamic. The lack of trust and inability to understand the benefits of research is crushing our science and technology edge. The Labs of the department of defense and NASA are shadows of their former glory. Remember that the internet came out of defense research. NASA started down the road to ruin after the moon landing then took final blows by the end of Reagan’s disasterous presidency. Now NASA is being brought further down by relying on Boeing for transport. Boeing is in the middle of riding maximizing shareholder value to the ruin of the company.

The DOE-NNSA Labs are a last bastion of American research. They are close to ruin. Over the course of my career, the Labs have been destroyed by the same attitudes. I distinctly remember the first ten years at Los Alamos being magical. The Lab was a wonderful crucible of knowledge, research and learning. Staff were generous with their time and expertise. I grew as a professional and flourished. Then it all ended. Like other institutions, fear and lack of trust entered. Actually it was already declining. Friends tell me that the Lab was even better in the years before I arrived. The generous spirit dried up and was replaced by suspicion and control. In the process research started to lose quality.

Breakthroughs no longer happened regularly with budget-money becoming the focus. No safety or security measure is too extreme. Cost be damned. Management became like business as private companies were the model of governance. The same attitudes revolving around maximizing shareholder value replace curiosity, inquiry and duty to the nation. Maximizing shareholder value has no meaning for the Labs, and yet it is the philosophy of governance. It has become toxic for companies (e.g., Boeing). It is idiotic for the Labs and a vehicle for catastrophe. Now the great

Labs of the USA are mere shadows of their former selves. We are all poorer for this. Recent studies have shown that the USA has completely lost its advantages in most science and technology. It has ceded its edge to China, India and Europe. We are to blame. The model of governance holds the murder weapon. Underneath this is the lack of trust infusing society. The pursuit of safety and security without regard for cost accelerates the process.

Let’s look at a couple of ways our fear and lack of trust play into this. Computer technology is the lifeblood of recent progress. We can see four distinct advances that shaped this period of time: Google search (its a verb now), the iPhone (smartphone), social media and large language models (i.e., ChatGPT). None of these came from the government labs. If you work at a government lab these advances are treated with fear and as risks. Their power is blunted by the fearful trust lacking management. Rather than harness the power of these breakthroughs, they are banned. They are castrated by rules. We see fear of technology and the inability to enable their power. Frankly we are a bunch of fucking morons and cowards. We have no leadership pushing back.

Let’s take this a step further. We see rules that seek to protect our work from prying eyes. Increasingly everything we do is classified in some way (lots of official use only that is now controlled unclassified information). This is just a way to hide things and not interact with the world. If we had a huge advantage this might make sense, but we don’t. We are not in the lead and these rules simply hold us back. They kill progress. Progress is what we need most of all now. The best thing all this hiding of information does is hide how incredibly incompetent we are. Increasingly we are hiding the embarrassingly backwards state of our technology. I am closing my career shaking my head on the collapse of our scientific supremacy.

“As people age, they confuse changes in themselves with changes in the world, and changes in the world with moral decline—the illusion of the good old days.” ― Steven Pinker

If we do not trust one another, we are already defeated.

– Allison Croggon

Most mornings I walk our dog, Duke, at a park near my house. The park is next to an elementary school. Here, I see direct evidence of how little trust Americans have in each other. I see the kids walking to school and if they are walking it is with parents. You even see parents with kids at bustops within eye shot of the house. You never see a kid walking alone to school. In fact this seems to be unthinkable today. If I think about myself all I remember is walking myself to school or later walking with my brother. Usually I would walk with friends the small distance to school.

The significant change is in social and societal trust. We no longer believe that children walking to school are safe. We fear all sorts of terrible things happening to them, many of which are figments of people’s imaginations and highly unlikely dangers. It’s instructive to compare the period from 1968 to 1976, when I walked to school, to the present day,when no one allows their child to walk to school. Regardless, this is a concerning sign for the health of our nation.Ultimately, we can’t avoid the fact that bad things happen and are inevitable (shit happens!). They aren’t blameworthy, but without trust,blame is readily assigned. Without trust people play it safe to avoid the blame.

“We are all mistaken sometimes; sometimes we do wrong things, things that have bad consequences. But it does not mean we are evil, or that we cannot be trusted ever afterward.”

― Alison Croggon

If you look at the United States today you see a nation where no one trusts each other. The impacts from this lack of trust are broad. It is important to look at what trust allows and its lack prohibits. Not trusting is expensive and it limits success. Those costs and impacts are hurting Americans left and right. I see it play out in my life. If we look around this damage is everywhere. It is evident in the politics today. It is evident in our personal lives too.

Fear is driving this change in behavior. Parents are terrified of sexual predators and random violence harming their children, despite the incredibly low probability of such events occurring. This reflects a common aspect of our low-trust society: we manage low-probability events at great cost. This phenomenon is widespread throughout society, as we incur enormous expenses to mitigate minuscule risks. In the case of children, this is ruining childhood. Socially, we see loneliness and isolation. For society as a whole, building or creating anything new becomes difficult and expensive.Everything costs more and takes longer due to the lack of trust.

I thoroughly enjoyed writing about a technical topic. It was both enjoyable and fulfilling, like cooking a delicious meal that you enjoy eating and even better when someone else appreciates it. However, there’s an underlying issue that’s been playing out behind the scenes. While there’s a clear benefit to conducting risky research, something is hindering progress.Ultimately, the reason for not realizing the benefits of risky research is a lack of trust. Risky research requires failure, and lots of it. Without trust, failure becomes unacceptable and suspicious. Without trust, people become cautious, and caution hinders progress. Caution leads to stagnation and decline, which is precisely what we observe across the country

“Trust is the glue of life. It’s the most essential ingredient in effective communication. It’s the foundational principle that holds all relationships.”

― Stephen R. Covey

Trust is best understood within our most intimate and important personal relationships. Whether it’s a romantic partner or friend, trust is foundational. When trust is lost in these relationships, they are at risk. If trust is not repaired, it can lead to the end of the relationship. Studies have shown that trust is built through several essential behaviors.

The first is authenticity, which involves presenting yourself as your true self. Faking your personality has the opposite effect and fosters suspicion. The second aspect is competence in areas relevant to the relationship. This could be athletic ability or intellectual prowess. Finally, trust requires demonstrated empathy, a deep care and concern for the well-being of others. The person you trust will understand your feelings and care about your welfare.

“Trust is the glue of life. It’s the most essential ingredient in effective communication. It’s the foundational principle that holds all relationships.”

― Stephen R. Covey

Stephen Covey’s The Speed of Trust provides valuable insights into the benefits of trust. The book demonstrates how trust can enhance efficiency and productivity. When trust exists, remarkable achievements are possible. Trust is contagious; when we are trusted, we trust others. Trust enables individuals to perform at their best, and organizations to achieve their highest potential. Conversely, a lack of trust is slow and costly. It is destructive. When we don’t trust, we tend to make mistakes and hinder progress. Lack of trust is the root of many fuck-ups.

The leaders who work most effectively, it seems to me, never say ‘I.’ And that’s not because they have trained themselves not to say ‘I.’ They don’t think ‘I.’ They think ‘we’; they think ‘team.’ …. This is what creates trust, what enables you to get the task done.

– Peter Drucker

The decline in American trust can be traced back to the 1970s. Several events shattered the spell of trust that had held the United States since the end of World War II. The upheavals of the 1960s had begun to erode trust with a generational divide, the civil rights movement, and a misguided war. The Nixon administration’s criminal actions exposed corruption at the highest levels of government. Nixon prioritized his own interests and power, seeking revenge against the culture he disliked. While Nixon’s religiosity may have distinguished him from Trump, it nonetheless reflects a decline in trust.

Other factors contributed to the unraveling of trust in the United States. The mid-1970s marked a peak in economic equality. Americans could comfortably achieve middle-class status with a single blue-collar income. People across the nation enjoyed a more level playing field, fostering empathy and trust. This shared experience and common culture allowed for authenticity to flourish. The nation was thriving and a global economic powerhouse, demonstrating competence. However, the energy crisis of the mid-1970s challenged these elements. The economy suffered, and blue-collar industries took a hit, further eroding trust.

“Never trust anyone, Daniel, especially the people you admire. Those are the ones who will make you suffer the worst blows.”

― Carlos Ruiz Zafón

The 1980s introduced new factors that undermined these trust drivers. The Reagan Revolution, characterized by a focus on business success through tax cuts and legal changes, significantly increased corporate wealth and power. The simultaneous assault on labor further weakened the ability of blue-collar jobs to provide a comfortable living. This marked the beginning of a widening economic inequality in the United States, which continues to grow today. This inequality erodes all aspects of societal and social trust, as people now live vastly different lives and hold radically different views of success. Consequently, people struggle to understand one another. This lack of understanding undermines empathy and destroys trust.

Other societal developments have accelerated the loss of trust. The terrorist attacks of September 11, 2001 led to a decline in trust and a rise in fear. The fear-based responses and societal changes that followed have persisted. Instead of progress toward a more inclusive society, division and bigotry are on the rise. The internet and the attention-driven economy have further exacerbated these trends. The cumulative effect of these factors is a massive political and cultural divide. The lack of trust now extends to the political system, threatening democracy itself and potentially spiraling further into an abyss.

“I’m not upset that you lied to me, I’m upset that from now on I can’t believe you.”

― Friedrich Nietzsche

Trust is cultivated through countless small acts. It was top of mind this past week and repeatedly demonstrated to me. I clearly distinguished between what was said privately and publicly. This inconsistency was painful to experience and significantly damaged trust in an important relationship. At work, I observe technical accuracy and competence being overshadowed by expediency. People hesitate to engage openly on topics due to fear of retaliation. All of this stems from and exacerbates a lack of trust

Building and fostering trust is paramount in all these situations. Trust in our relationships, with our coworkers, and among our fellow citizens is essential. With trust, things will improve, but it requires courage and effort. Trust is a product of strong character. It unleashes competence and grows alongside it. Trust is efficient and the foundation of success. We need leadership that guides us toward trust and away from fear and suspicion. This involves identifying the factors that have eroded trust and changing course. Many people benefit from these trust-destroying elements. To achieve trust, society needs to become more equitable with a deeper shared culture. We need a spirit that recognizes a future where trust prevails. Living, relating, and working in a place of trust is a more positive experience

Trust is the highest form of human motivation. It brings out the very best in people.

– Stephen Covey