tl;dr

I’m getting close to the decision to retire. I’m looking back at my career, trying to put things in perspective. By most accounts, it has been a great career, but it is also short of what it could have been. I have some genuine disappointments. I’ve seen a lot and accomplished some great things. In this Requiem, I’ve talked about my young life. I expanded and looked back at the heart of my professional career in Los Alamos, the apex of my career. Those years made me who I am as a scientist.

Now I am at the closing chapter of my career, the time I have spent at Sandia National Labs. It is too close to have a full perspective, and I cannot offer full disclosure either. I can offer a deep sense of disappointment and sadness about how it has worked out. It has not been a place to fulfill my potential, and, in retrospect, never could have. I made my choices and the results are a wonderful full life, but a loss of professional success in trade.

A Change is Needed

“Things change. And friends leave. Life doesn’t stop for anybody.”― Stephen Chbosky, The Perks of Being a Wallflower

It was October 2006, and I was convinced that things were not going to get better at Los Alamos. The new management team was awful. The new corporate partners were corrupting the essence of the Lab. My oldest child was in middle school, and I was concerned he wouldn’t do well as a teen in Los Alamos. The small-town nature of Los Alamos was starting to get under our skin too. You often would know people three or four or five ways. Work lives and personal lives were not separate, they were completely intertwined. Los Alamos was closing in on us and we needed space.

During that October, I went to a biannual conference put on by the nuclear weapons community focused on computational modeling. This time it was held in Los Alamos. The nuclear weapons world of the USA and the UK joined to talk about computers, modeling, and computational physics. It is totally my jam. Sandia National Labs sent a contingent, including my friend Tim Trucano. I approached him and a Sandia manager there (Randy Summers) with the potential of moving. They were very excited, and before long, I had an interview and shortly thereafter a job offer.

I left Los Alamos Lab on February 16, 2007, and began at Sandia the following Monday. It was MLK day, which Sandia unwisely doesn’t take as a holiday, but Los Alamos does. Maybe this little detail was a sign, and in retrospect, one I should have heeded. I was going to work in Computer Research Institute (“1400” in Sandia’s annoying schema for organizational names… no names, just numbers). My job was to work on a code development team for the MHD code “Alegra”. Alegra is a magneto-hydrodynamic code developed originally under the ASCI program. Another harbinger of issues was its lack of support from ASCI (not ASC, because a permanent program can be an initiative for so long). Alegra was then mostly supported by the US Army for analyzing things like advanced armor concepts. The reality was that I was wildly overqualified for the job. The funding and the code were horrendously constrained, and the code base was around 15 years old and all in C++.

It was a good team with good people. I let myself focus on that and approached the job with optimism and an open mind. Again, with the benefit of time, this was a naive approach. I should have been far more guarded. A big piece of this statement is that the forces that fucked up Los Alamos were present at Sandia. Sandia had always been a little corporate-managed originally by Bell Labs. In 1994 the management changed to Lockheed-Martin. The low-trust corporate management approach was firmly entrenched at Sandia.

The decline in the quality of work, science, and culture at Los Alamos was actually a national problem. Sandia was also in decline. Los Alamos took the blow of being corporately managed; Sandia was already a corporation. The management was terrible for science, but great at giving the government what it asked for. The National Labs were being pulled down by the same forces making the United States awful. The same forces we’ve seen unleashed over the past 10 years with toxic political effects. There is no escaping the decline we found ourselves in. Professionally, I should have been more pragmatic and less hopeful with my new circumstances.

A few things stand out about that time. My move got swallowed by the housing crisis of 2007 and we lost our shirt in real estate selling in Los Alamos. If we’d been three months earlier, we would have escaped, and been about $100,000 better off.

The director of our Center at Sandia also moved from Los Alamos almost at the same time as me. James Peery had started at Sandia then moved to Los Alamos for a time to run the ASC program. He was also on a corporate bid team to manage Los Alamos that did not get the job. He moved to a job managing the Center where he had started at Sandia. Unlike me, Sandia was perfectly suited for him. James is now the Laboratory Director at Sandia, retiring soon.

A few things were evident upon my arrival. A lot of administrative details were immensely better at Sandia. For example, one of the worst things about Los Alamos was travel reimbursement. At Sandia, this process was fast and seamless. The timecard application was great, too. It would have been great, except Sandia is a complete asshole to employees about some auxiliary regulations around the travel credit card. Sandians are obsessive followers of rules and regulations. It is an engineering lab and small “c” conservative organization from top to bottom.

Another difficult thing is Sandia’s devotion to information control and the practice of “need-to-know”. This is baked into their culture. Stepping back from this, one can see that Sandia was corporate in every way, and did mundane bullshit far better than Los Alamos. Thus, in every way that does not matter to National Security Science, it was an improvement. The engineering at Sandia is repressively backward and grounded in the past. Science at Sandia is peripheral and always counterculture. I have learned that culture is almost immutable, and Sandia’s culture does not fit me at all.

My take on Sandia is going to be quite harsh, so I’ll disarm this assessment a bit. On a broader scale, Sandia is a great employer. Most people would be lucky to work there. The reasons for this are twofold: First, most employers in the USA are terrible to employees, and I had been gifted an incredible experience at Los Alamos to start my career. The current Los Alamos and Sandia never stood a chance at living up to that start of my career. Secondly, the Labs have declined greatly largely due to the forces that are sieging America at large. These forces are bipartisan. Distrust and bullshit are coming from both the left and the right. Therefore, a significant drop in workplace satisfaction occurred when I left Los Alamos. Another drop would await me at Sandia. The thing that meant the most about Los Alamos was the culture. I fit into Los Alamos culture, and Sandia’s culture is unnatural for me.

Another red flag with my hire at Sandia was my ranking as a staff member. There were limits in the level I could be hired. These limits existed because they’d been abused in the past with a political hire. I would have to wait a few years to be promoted. Still, there are shades of lack of recognition professionally by Sandia. This theme is still rife today. Back in 2007, I was approaching the job with possibility and generosity. That spirit was not returned by Sandia. Sandia’s ethos was far different from Los Alamos. This was a foreign culture and not one that I was suited for.

A couple years ago, I had an epiphany about this contrast. I was at a conference. There was a contingent of Los Alamos staff there along with my coworkers at Sandia. During the meeting, I noticed that my behavior around the Los Alamos people was completely different than the Sandia ones. The people from Los Alamos were not old friends or people I knew well. In reality, I knew the Sandians better. I realized that I was “code-switching.” It made me realize that the culture and environment at Sandia were oppressive to me. The self I was with Los Alamos staff was far closer to the real me. With Sandia’s staff, I buried and censored myself. This was an absolutely jarring realization. The conclusion was that at Sandia, I could not be myself, either personally or professionally.

So as 2007 sped along, I worked to fit in and do my work. I wanted to make this succeed and I gave Sandia a lot of space. I gave it a lot more space and the benefit of the doubt than it deserved.

On the Alegra team, we did some good work despite all of this. We did run into a crisis early in my Sandia career. The ASC funding was being dropped even further, and the Army sponsorship was at risk. We needed to produce a code that was more robust and handled difficult situations better. The gauntlet was thrown down: either we improve, or Army funding would shrink until it was gone. The project would die. I immediately set about to devise a strategy to solve the problem. It was all hands on deck. No solution was off the table. In the end, we fixed the code with success that exceeded our wildest expectations. This effort remains today as the highlight of my time at Sandia. This was as good as Sandia would ever get for me. It was some really great work.

When I compare this with what I did at Los Alamos, this makes that highlight almost seem tragic. It barely held a candle to what I achieved in Los Alamos.

“Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes. The ones who see things differently. They’re not fond of rules. And they have no respect for the status quo. You can quote them, disagree with them, glorify or vilify them. About the only thing you can’t do is ignore them. Because they change things. They push the human race forward. And while some may see them as the crazy ones, we see genius. Because the people who are crazy enough to think they can change the world, are the ones who do.” ― Steve Jobs

This quote is from an Apple marketing pitch. It describes the best of Los Alamos and captures the ideals of that Lab. If I took the quote and wrote the exact opposite, it would capture the worst of Sandia (albeit slightly unfairly).

This is Not the Right Place for Me

“An eye for an eye, and the whole world would be blind.” ― Kahlil Gibran

Recently, the focus of how much tragedy has become evident. In many respects, what stands out at Sandia is a lack of respect and use of expertise. Throughout my career, I have developed extensive expertise and Sandia simply ignores it. That expertise is one of the most important parts of Stockpile Stewardship. I have a unique knowledge of modeling and simulation—specifically the codes, methods and models used to do it. Part of this can easily be seen in how retirees are treated by the Labs. At Sandia, the prevailing attitude is “Don’t let the door hit you on the way out.” This contrasts with Los Alamos, where retirees and their knowledge are welcome and ever-present. Right now, it is these aspects that exemplify how I see Sandia. The core of my professional identity and accomplishments are not valuable to the institution. I do not feel welcome or even useful when it is obvious I should be.

There is so much more that feels bad. I put a great deal of goodwill into my first few years at Sandia. My inability to bring my real self to work began to get under my skin. (Actually, it influenced what I got on my skin too; more on that later.) The underlying Sandia culture is buttoned up and uptight as hell. It is not a place where free thought and debate happen. Even worse, it is a place that does not feel accepting of differences. While I am a middle-aged cis-gendered white man, I am not typical in many ways. None of those differences feel accepted openly. The workplace feels confined and truly limited in acceptance. I can’t bring either my personal or professional self to work. I am careful to see this feeling as a balance of intrinsic cultural differences from the shitshow of today’s America. I am sure it is a mix of innate culture and current trends.

One of the hardest aspects of working at Sandia was determining a professional development path. Why work hard at developing myself when Sandia already could not use my skills? When I walked in the door in 2007, my skills already exceeded what Sandia could use. In retrospect, I could have worked harder and hid myself at work. It would have been valuable at the office, but harmful to my being. Thus, a plan was hatched to work on my writing with focus.

This was the genesis of my blog. I would write regularly and publish the writing to be seen by others. Writing to be read by others is essential. In retrospect, the blog was one of the best things I did while at Sandia. It also made me a better scientist. Writing is thinking deeply. I would write about presentations I would give, and I approached those talks having thought deeply about the topics. In the end, thinking deeply was viewed as counter-productive unless those thoughts aligned with Sandia’s views.

Unfortunately, the blog ended up being the worst thing that happened to me at Sandia, too. The way I stopped writing exemplified the worst of the Lab but aligned perfectly with its culture. In a way, I should have seen this outcome coming at me clearly. Instead, I was optimistic and trusting. That trust was never returned. I hate saying this, but events clearly point to the worst possible reading of how the blog ended.

This Will Not End Well

In 2013, I turned 50 and started to exhibit many signs of the canonical midlife crisis. Today I see it differently, but one of the signs of the crisis was getting my first tattoo in October. Since then, I’ve gotten 25 more. They fall into some general themes: my science, primal imagery, and philosophical expressions of love and freedom. I’ve come to realize that the tattoos were a personal expression in the environment that limited it. The tattoos were also very essentially modern and fit to Albuquerque culture. The irony is that Sandia culture is orthogonal to Albuquerque culture. I was not going to be stuck into the Sandia straightjacket. I wanted to express myself freely and fully. By putting the expression on my body, I would not be silenced. I don’t think it was a conscious thought, but rather a reaction to the day job with my real self. As the past decade has unfolded, my true self and my work self have diverged rather precipitously. The subtle differences I expressed moved toward a primal scream.

I wrote this blog consistently from 2014-2018. I stopped writing it because I was given no choice in 2018. In a culturally consistent passive-aggressive manner, I was forced to stop. The form of this force was a Sandia ethics investigation. It came from an anonymous source (although I am relatively certain of the identity of the source). The investigation and charges were absurd. The entire blog was done in plain view and as a part of my professional development plan. It was done with permission. I would give the link to my blog at the end of my professional talks. The blog was good for my professional skills and performance. It was only bad for professional success where blind obedience to power is demanded. In a healthy culture, it would have been celebrated. In an unhealthy culture, the blog was a threat.

The problem was that I expressed views that were not the same as my management. I was part of a rather ill-conceived national program that was contributing to the decline in American scientific supremacy. I said as much in the blog. Worse yet, I had an audience of peers, thus I was dangerous. I could be shut up through force—and I was. This program was big money, and in today’s world is all about money. Money is power and truth is determined by power. Opposing the monied interests is dangerous.

I stood down so I could continue to support my family and my life. Intellectual honesty and debate about what are best are not on the menu today. The people in charge of the Labs and our science funding are not open to questioning their priorities. This lack of debate is part of what is fueling the decline of American supremacy in science.

I licked my wounds. The ethics investigation is the worst blight in my entire professional life. It may be the worst thing of my entire adult life, but I’ll admit this means I have a charmed life.

As I attempted to recover, the pandemic arose. The country and the world shut down. We all worked from home for months as most of my work went online. I discovered a massive relief in this arrangement. Even in this lockdown, I felt freedom. I did not have to go to work and put on a disingenuous mask every day. It made me realize how incredibly unhappy Sandia had made me. In the past couple of years, the depth of my dismay professionally has deepened. It has transitioned to a mourning of opportunity lost and disappointment.

“The wound is the place where the Light enters you.”— Jalal ad-Din Muhammad ar-Rumi

Prospectus and Path Ahead

Hindsight is usually a biased way to look at things. The reasons for my move to Sandia and Albuquerque were varied. The professional reasons were naive and quixotic. I approached the new Lab with optimism and hope that was not grounded in actual possibility. I pushed away many obvious signs of disaster for years. I should have been more guarded and far less trusting. That said, I moved for reasons in my personal life. I moved for my family. It was better professionally for my wife and offered better opportunities for my children. We had aging parents to care for.

Each of these concerns played out differently, but all were valid. For my own part, I have changed over these years in many interesting and exciting ways. In sum, I would not trade the benefits and advances in myself and my personal life for professional success. Nonetheless, professional success was sacrificed in the move.

Professionally, Sandia has been just okay. By my own standards, my professional accomplishments have been a severe disappointment. The way my blog was treated figures heavily into my assessment. It was a huge insult and attack on me personally. The lack of professional respect is palpable. The lack of use for my vast professional knowledge and skills is appalling.

This is not to say one cannot be successful at Sandia. It is to say that *I* would never succeed at Sandia. It is not the right place for me to succeed. In the same breath, I can succeed as a person, where I am now. It could also be true that Los Alamos might have been a worse place for that personal success. You really can’t have it all.

Today, I feel like my work has mostly been a waste of my time. It is easy to see how I could have used my skills and knowledge better than they were at Sandia. Perhaps in closing, this is my challenge for the future: to find a valuable path for contribution suitable for my precious time left. I did give Sandia a chance, and that effort was not met nor rewarded in kind.

I have not decided when to retire, or what I will do after retirement. What will happen is taking shape. Felicia is retired and I’m learning from her experience. It is just clear to me that it is around the corner. While my professional life at Sandia was substandard, my life is good. I would not trade the success in my personal life for the sort of professional success I would aspire toward. Perhaps the lesson is a sort of balance to life. Maybe you simply can’t have it all and choices are made. I made mine. I stand by the outcomes.

Finally, I want to give endless love and gratitude to those who have enriched my life and been my friends. I have known many wonderful people who have made my time at work vastly better. Each of them has helped me be a better person and taught me so much. Nothing would have been as good without them. I greatly appreciate Meera Collier for graciously editing my writing, and helping to make it better than I could manage myself.

“Yesterday I was clever, so I wanted to change the world. Today I am wise, so I am changing myself.”— Jalal ad-Din Muhammad ar-Rumi

Important Papers Written While At Sandia

Banks, Jeffrey W., T. Aslam, and William J. Rider. “On sub-linear convergence for linearly degenerate waves in capturing schemes.” Journal of Computational Physics 227, no. 14 (2008): 6985-7002.

Mattsson, Ann E., and William J. Rider. “Artificial viscosity: back to the basics.” International Journal for Numerical Methods in Fluids 77, no. 7 (2015): 400-417.

Rider, W. J., E. Love, M. K. Wong, O. E. Strack, S. V. Petney, and D. A. Labreche. “Adaptive methods for multi‐material ALE hydrodynamics.” International Journal for Numerical Methods in Fluids 65, no. 11‐12 (2011): 1325-1337.

Robinson, Allen, Thomas Brunner, Susan Carroll, Richard Drake, Christopher Garasi, Thomas Gardiner, Thomas Haill et al. “ALEGRA: An arbitrary Lagrangian-Eulerian multimaterial, multiphysics code.” In 46th aiaa aerospace sciences meeting and exhibit, p. 1235. 2008.

Rider, William, Walt Witkowski, James R. Kamm, and Tim Wildey. “Robust verification analysis.” Journal of Computational Physics 307 (2016): 146-163.

Rider, William J. “Reconsidering remap methods.” International Journal for Numerical Methods in Fluids 76, no. 9 (2014): 587-610.

Love, E., William J. Rider, and Guglielmo Scovazzi. “Stability analysis of a predictor/multi-corrector method for staggered-grid Lagrangian shock hydrodynamics.” Journal of Computational Physics 228, no. 20 (2009): 7543-7564.

Hills, Richard Guy, Walter R. Witkowski, Angel Urbina, William J. Rider, and Timothy Guy Trucano. Development of a fourth generation predictive capability maturity model. No. SAND2013-8051. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States), 2013.

Barlow, Andrew J., Pierre-Henri Maire, William J. Rider, Robert N. Rieben, and Mikhail J. Shashkov. “Arbitrary Lagrangian–Eulerian methods for modeling high-speed compressible multimaterial flows.” Journal of Computational Physics 322 (2016): 603-665.

Alexander, Francis, Ann Almgren, John Bell, Amitava Bhattacharjee, Jacqueline Chen, Phil Colella, David Daniel et al. “Exascale applications: skin in the game.” Philosophical Transactions of the Royal Society A 378, no. 2166 (2020): 20190056.

Kamm, James R., Jerry S. Brock, Scott T. Brandon, David L. Cotrell, Bryan Johnson, Patrick Knupp, William J. Rider, Timothy G. Trucano, and V. Gregory Weirs. Enhanced verification test suite for physics simulation codes. No. LA-14379. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States), 2008.

Yanilkin, Yury V., Evgeny A. Goncharov, Vadim Yu Kolobyanin, Vitaly V. Sadchikov, James R. Kamm, Mikhail J. Shashkov, and William J. Rider. “Multi-material pressure relaxation methods for Lagrangian hydrodynamics.” Computers & Fluids83 (2013): 137-143.

Weirs, V. Gregory, James R. Kamm, Laura P. Swiler, Stefano Tarantola, Marco Ratto, Brian M. Adams, William J. Rider, and Michael S. Eldred. “Sensitivity analysis techniques applied to a system of hyperbolic conservation laws.”

tl;dr.

The practice of code verification has focused on finding bugs in codes. This is grounded in the proving that a code is correctly implementing a method. While this is useful and important, it does not inspire. Code verification can be used for far more. It can be the partner to method development and validation or application assessment. It can also provide expectations for code behavior and mesh requirements on applications. Together these steps can make code verification more relevant and inspiring. It can connect it to important scientific and engineering work pulling it away from computer science.

When you can measure what you are speaking about, and express it in numbers, you know something about it.

– Lord Kelvin

My Connection to Code Verification

Writing about code verification might seem like a scheme to reduce my already barren readership. All kidding aside, code verification is not the most compelling topic for most. This includes people making their living writing computational solvers. For me it is a topic of much greater gravity. While I am inspired by the topic, most people are not. The objective here is to widen its scope and importance. Critically, I have noticed the problems getting worse as code verification seems to fade from serious attention. This all points to a need for me to think about this topic deeply. It is time to consider a change to how code verification is talked about.

“If you can not measure it, you can not improve it.”

– Lord Kelvin

My starting point for thinking about code verification is to look at myself. Code verification is something I’ve been doing for more than 30 years. I did it before I knew it was called “code verification.” Originally, I did it to assist my development of improved methods in codes I worked on. I also used it to assure that my code was correct, but this was secondary. This work utilized test problems to measure the correctness and more importantly the quality of methods. As I continued to mature and grow in my scientific career I sought to enhance my craft. The key aspect of growth was utilizing verification to exactly measure method character and quality. It was through verification that I understood if a method passed muster.

“If failure is not an option, then neither is success.”

― Seth Godin

Eventually, I developed new problems to more acutely measure methods. I also developed problems to break methods and codes. When you break a method you help define its limitations. Over time I saw the power of code verification as I practiced it. This contrasted to how it was described by V&V experts. The huge advantage and utility of code verification I found in method development was absent. Code verification was relegated to correctness through code bug detection. In this mode code verification is a spectator to the real work of science. I know it can be so very much more.

“I have been struck again and again by how important measurement is to improving the human condition.”

– Bill Gates

The Problem with Code Verification

In the past year I’ve reviewed many different proposals in computational science. Almost all of them should be utilizing code verification integrally in their work. Almost all of them failed to do so. At best, code verification is given lip service because of proposal expectations. At worst it is completely ignored. The reason is that code verification does not set itself as a serious activity for scientific work. It is viewed as a trivial activity beneath mention in a research proposal. The fault lies with the V&V community’s narrative about it. (I’ve written before on the topic generally https://williamjrider.wordpress.com/2024/08/14/algorithms-are-the-best-way-to-improve-computing-power/)

“Program testing can be used to show the presence of bugs, but never to show their absence!”

― Edsger W. Dijkstra

Let’s take a look at the narrative chosen for code verification more closely. Code verification is discussed primarily as a manner to detect bugs in the code. The bugs are detected when the code does not act as a consistent solution of the governing equations in the manner desired. This comes when the exact solution to those governing equations does not match the order of accuracy designed for the method. This places code verification as part of software development and quality. This is definitely an important topic, but far from a captivating one. At the same time code verification is distanced from math, physics and aapplication space engineering. Thus, code verification does not feel like science.

This is the disconnect. To be focused upon in proposals and work code verification needs to be part of a scientific activity. It simply is not one right now. Of all the parts of V&V, it is the most distant from what the researcher cares about. More importantly, this is completely and utterly unnecessary. Code verification can be a much more holistic and integrated part of the scientific investigation. It can span all the way from software correctness to physics and application science. If the work involves development of better solution methodology, it can be the engine of measurement. Without measurement “better” cannot be determined and is left to bullshit and bluster.

“Change almost never fails because it’s too early. It almost always fails because it’s too late.”

― Seth Godin

What to do about it?

The way forward is to expand code verification to include activities that are more consequential. To constructively discuss the problem, the first thing to recognize that V&V is the scientific method for computational science. It is essential to have correct code. The software correctness and quality aspects of code verification remain important. If one is doing science with simulation, the errors made in simulation are more important. Code verification needs to contribute to error analysis and minimization. Another key part of simulation are choices about the methods used. Code verification can be harnessed to serve better methods. The key in this discussion is that the additional tasks are not discussed in what code verification is. This is an outright oversight.

Appreciate when things go awry. It makes for a better story to share later.

― Simon Sinek

Let’s discuss each of these elements in turn. First we should get to some technical details of code verification practice. The fundamental tool in code verification is using exact solutions to determine the rate of convergence of a method in a code. The objective is to show the code implementation produces the theoretical order of accuracy. This is usually accomplished by computing errors on different meshes.

The order of accuracy comes from numerical analysis of the truncation errors of a method. It is usually takes the form of a power of the mesh size. For example a first order method the error is proportional to the mesh size. For a second order method the error depends on the square of the mesh size. This all follows from the analysis and has the error vanishing as the mesh size goes to zero (see Oberkampf and Roy 2010)

The grounding of code verification is found in the work of Peter Lax. He discovered the fundamental theorem of numerical analysis (Lax and Richtmyer 1956). This theorem says that a method is a convergent approximation of the partial differential equation if it is consistent and is stable. Stability comes from getting an answer that does not fall apart into numerical garbage. Practically speaking, stability is assumed when the code produces a credible answer to problems. The trick of consistency is that the method reproduces the differential equation plus an ordered remainder. Now the trick of verification is that you invert this and use a convergent sequences to infer consistency. This is a bit of a leap of faith.

“Look for what you notice but no one else sees.”

― Rick Rubin

The additional elements for verification

The most important aspect to add to code verification is stronger connection to validation. Numerical error is an important element in validation and application results. Currently code verification is divorced from validation. This makes it ignorable in the scientific enterprise. To connect better, the errors in verification work need to be used to understand mesh requirements for solution features. This means that the exact solutions used need to reflect true aspects of the validation problem.

Current verification practice pushes this activity into the background of validation. In doing “bug hunting” code verification, the method of manufactured solutions (MMS) is invaluable. The problem is that MMS solutions usually bear no resemblance to validation problems. For people concerned with real problems MMS problems have no interest, nor guidance for their solutions. Instead verification problems should be chosen that feature phenomena and structures like those validated. Then the error expectations and mesh requirements can be determined. Code verification can then be used as simple pre-simulation work before validation ready calculations are done. Ultimately this will require the development of new verification problems. This is deep physics and mathematical work. Today this sort of work is rarely done.

The next big change in code verification is connecting code verification more actively to method-algorithm research. Code verification can be used to measure the error of a method directly. Again this requires a focus on error instead of convergence rate. The convergence rate is still relevant and needs to be verified. At the same time methods with the same convergence rate can have greatly different error magnitudes. For more realistic problems the order of accuracy does not determine the error. It has been shown that low order methods can out perform higher order methods in terms of error (see Greenough and Rider 2005).

“There is no such thing as a perfect method. Methods always can be improved upon.”

–  Walter Daiber

In all aspects of developing a method code verification is useful. The base of making sure the implementation is correct remains. The additional aspect that I am suggesting is the ability to assess the method dynamically. This should be done on a wide range of problems biased toward application-validation inspired problems. In terms of making this activity supported by those doing science, the application-validation inspired problems are essential. This is also where code verification fails most miserably. The best example of this failure can be found in shock wave calculations.

“If you can’t measure it, you can’t change it.”

– Peter Drucker

Let’s take a brief digression to how verification currently is practiced in shock wave methods. Invariably the only time you see detailed quantitative error analysis is on a smooth differentable prroblem. This problem has no shocks and can be used to show a method has the “right” order of accuracy. This is expected and common. The only value is the demonstration that a nth order method is indeed nth order. It has no practical value for the use of the codes.

“Measure what is measurable, and make measurable what is not so.”

– Galileo Galilei

Once a problem has a shock in it, the error analysis and convergence rates disappear from the work. Problems are only compared in the “eyeball norm” to an analytic or high resolution solution. The reason for this is that the convergence rate with a discontinuity is one or less. The reality being ignored is that error can be very different (see the paper by Greenough and Rider 2005). When I tried to publish a paper that used errors and convergence rates to assess the method with shock, the material needed to be deleted. As the associate editor told me bluntly, “if you want to publish this paper get that shit out of the paper!” (see Rider, Greenough and Kamm 2007)

Experts are the ones who think they know everything. Geniuses are the ones who know they don’t

― Simon Sinek

Why is this true? Part of the reason is the belief that the accuracy does not matter any longer. The failure is to recognize how different the errors can be. This has become accepted practice. Gary Sod introduced the canonical shock tube problem that bears his name. Sod’s shock tube has been called the “Hello World” problem for shock waves. In Sod’s 1978 paper the run time of different methods was given, but errors were never shown. The comparison with analytical solution to the problem was qualitative, the eyeball norm. Subsequently, this became the accepted practice. Almost no one ever computes the error or convergence rate for Sod’s problem or any other shocked problem.

“One accurate measurement is worth a thousand expert opinions.”

– Grace Hopper

As I have written and shown recently this is a rather profound oversight. The importance of the error level for a given method is actually far greater if the convergence rate is low. The lower the convergence rate, the more important the error is. Thus we are not displaying the errors created by methods in the conditions where it matters the most. This is a huge flaw in the accepted practice and a massive gap in the practice of code verification. It is something that needs to change.

“The Cul-de-Sac ( French for “dead end” ) … is a situation where you work and work and work and nothing much changes”

― Seth Godin

My own practical experience speaks volumes about the need for this. Virtually every practical application problem I have solved or been associated with converges at low order (first order or less). The accuracy of the methods under these circumstances mean the most to the practical use of simulation. Because of how we currently practice code verification applied work is not impacted. There is a tremendous opportunity to improve calculations using code verification. As I noted a couple of blog posts ago, the lower the convergence rate, the more important the error is (https://williamjrider.wordpress.com/2024/08/14/algorithms-are-the-best-way-to-improve-computing-power/). A low error method can end up being orders of magnitude more efficient. This can only be achieved if the way code verification is done and its scope increase. This will also draw it together with the full set of application and validation work.

More related content (https://williamjrider.wordpress.com/2017/12/01/is-the-code-part-of-the-model/, https://williamjrider.wordpress.com/2017/10/27/verification-and-numerical-analysis-are-inseparable/, https://williamjrider.wordpress.com/2015/01/29/verification-youre-doing-it-wrong/, https://williamjrider.wordpress.com/2014/05/14/important-details-about-verification-that-most-people-miss/,

“If it scares you, it might be a good thing to try.”

– Seth Godin

Roache, Patrick J. Verification and validation in computational science and engineering. Vol. 895. Albuquerque, NM: Hermosa, 1998.

Oberkampf, William L., and Christopher J. Roy. Verification and validation in scientific computing. Cambridge university press, 2010.

Lax, Peter D., and Robert D. Richtmyer. “Survey of the stability of linear finite difference equations.” In Selected Papers Volume I, pp. 125-151. Springer, New York, NY, 2005.

Roache, Patrick J. “Code verification by the method of manufactured solutions.” J. Fluids Eng. 124, no. 1 (2002): 4-10.

Greenough, J. A., and W. J. Rider. “A quantitative comparison of numerical methods for the compressible Euler equations: fifth-order WENO and piecewise-linear Godunov.” Journal of Computational Physics 196, no. 1 (2004): 259-281.

Rider, William J., Jeffrey A. Greenough, and James R. Kamm. “Accurate monotonicity-and extrema-preserving methods through adaptive nonlinear hybridizations.” Journal of Computational Physics 225, no. 2 (2007): 1827-1848.

Sod, Gary A. “A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws.” Journal of computational physics 27, no. 1 (1978): 1-31.

tl;dr; Algorithms shape our world today. When a new algorithm is created it can transform a computational landscape. These changes happen in enormous leaps that take us by surprise. The latest changes in the artificial intelligence are the result of such a breakthrough. It is unlikely to be followed by another breakthrough soon reducing the seeming pace of change. For this reason the threats of doom and vast wealth are overblown. If we want more progress it is essential to understand how such breakthroughs happen and their limits.

“The purpose of computing is insight, not numbers.”

– Richard Hamming

We live in the age of the algorithm. In the past ten years this has leapt to the front of mind with social media and the online world. It has actually been true ever since the computer took hold of society. This began in the 1940’s with the first serious computers, and numerical mathematics. A new improved algorithm always drives the use of the computer forward as much as hardware. What people do not realize is that the improvements that get noticed are practically quantum in change. These algorithms get our attention.

“I am worried that algorithms are getting too prominent in the world. It started out that computer scientists were worried nobody was listening to us. Now I’m worried that too many people are listening.”

– Donald Knuth

Now that the internet has become central to our lives we need to understand this. One reason is understanding how algorithms create value for business and stock market valuations. How these sorts of advances fool people on the pace of change? We should also know how this breakthroughs are made. We need to understand how likely we are to see progress? How can we create an environment where advances are possible? How the way we fund and manage work actually destroys the ability to continue progress?

“You can harvest any data that you want, on anybody. You can infer any data that you like, and you can use it to manipulate them in any way that you choose. And you can roll out an algorithm that genuinely makes massive differences to people’s lives, both good and bad, without any checks and balances.”

– Hannah Fry

Two examples come to mind in recent year to illustrate these points. The first is the Google search algorithm, pagerank. The second is the transformer, which elevated large language models to the forefront of the public’s mind in the last two years. What both of these algorithms illustrate clearly is the pattern for algorithmic improvement. A quantum leap in performance and behavior followed by incremental changes. These incremental changes are basically fine tuning and optimization. They are welcome, but do not change the World. The key is realizing the impact of the quantum leap from an algorithm and putting it into proper perspective.

Google is an archetype

Google transformed search and the internet and ushered algorithms into the public eye. Finding things online used to be torture as early services tried to produce a “phone book” for the internet. I used Alta Vista, but Yahoo was another example. Then Google appeared and we never went back. Once you used Google the old indexes of the internet were done. It was like walking through a door. You shut the door and never looked back. This algorithm turned the company Google into a verb, household name and one of the most powerful forces on Earth. Behind it was an algorithm that blended graph theory and linear algebra into an engine of discovery. Today’s online world and its software are built on the foundation of Google.

“The Google algorithm was a significant development. I’ve had thank-you emails from people whose lives have been saved by information on a medical website or who have found the love of their life on a dating website.”

– Tim Berners-Lee

Google changed the internet introduced search and demonstrated the power of information. All of a sudden information was unveiled and shown to be power. Google unleashed the internet into a transformative engine for business, but society as well. The online world we know today owes its existence to Google. We need to acknowledge that Google today is a shadow of the algorithm of the past. Google has become a predatory monopoly and the epitome of “enshitification” of the internet. This is the process of getting worse over time. This is because Google is searching for profits over performance. Instead of giving us the best results they are selling spaces for money. This process is repeated across the internet undermining the power of the algorithms that created it.

“In comparison, Google is brilliant because it uses an algorithm that ranks Web pages by the number of links to them, with those links themselves valued by the number of links to their page of origin.”

– Michael Shermer

The Transformer and LLMs

The next glorious example of algorithmic power comes from Google (Brain) with the Transformer. Invented at Google in 2017 this algorithm has changed the world again. With a few tweaks and practical implementations OpenAI unleashed ChatGPT. This was a large language model (LLM) that ingested large swaths of the internet to teach it. The LLM can then produce results that were absolutely awe-inspiring. This was especially in comparison to what came before where suddenly the LLM could produce almost human like responses. Granted this is true if that human was a corporate dolt. Try to get ChatGPT to talk like a real fucking person! (just proved a person wrote this!)

“An algorithm must be seen to be believed.”

– Donald Knuth

These results were great even after OpenAI lobodomized ChatGPT with reinforcement learning that kept it from being politically incorrect. The LLM’s won’t curse or say racist or sexist stuff either. In the process the LLM becomes as lame as a conversation with your dullest coworker. The unrestrained ChatGPT was almost human in creativity, but also prone to sexist, racist and hate speech (like people). It is amazing to know how much creativity was sacrificed to make it corporately acceptable. It is worth thinking about and how this reflects on people. Does the wonder of creativity depend upon accepting our flaws?

Under the covers in the implementation of the foundation models at the core of ChatGPT is the transformer. The transformer has a couple of key elements. One if the ability to devour data in huge chunks perfectly fitting for modern GPU chips. This has allowed far more data to be used and transformed NVIDIA into a mulit-trillion dollar company overnight. This efficiency is only one of the two bits of magic. The real magic is the attention mechanism. This is what the LLM takes as instructions for its results. The transformer allows longer more complex instructions to be given. It also allows multiple instructions to guide its output. The attention mechanism has led to fundamentally different behavior from the LLMs. Together these elements demonstrate the power of algorithms.

“Science is what we understand well enough to explain to a computer. Art is everything else we do.”

– Donald Knuth

The real key to LLMs is NOT the computing available. A lot of capable computing helps and makes it easier. The real key to the huge leap in performance is the attention mechanism that changed the algorithm. This produced the qualitative change in how LLMs functioned. This produced the sort of results that made the difference. It was not the computers; it was the algorithms!

The world collectively lost their shit when ChatGPT went live. People everywhere freaked the fuck out! As noted above the impact could have easily been more profound without the restraint offered by reinforcement learning. Nonetheless feelings were unleashed that felt like we were on the cusp of exponential change. We are not. The reason why we are not is something key about the change. The real difference with these new LLMs was all predicated on the transformer algorithm’s character. Unless the breakthroughs of the transformer are repeated with new ideas, the exponential growth will not happen. Another change will happen, but it is not likely for a number of years from now.

A look at the history of computational science unveils that such changes happen more slowly. One cannot count on these algorithmic breakthroughs. They happen episodically with sudden leaps followed by periods of fallow growth. The fallow periods are optimization of the breakthrough and incremental change. As 2024 plays out I have become convinced that LLMs are like this. There will be no exponential growth into general AI that people fear. The transformer was the breakthrough and without another breakthrough we are on a pleateau of performance. Nonetheless like Google, ChatGPT was a world changing algorithm. Until a new algorithm is discovered, we will be on a slow path to change.

“So my favorite online dating website is OkCupid, not least because it was started by a group of mathematicians.”

– Hannah Fry

Computational Science and Quantum Leaps from Algorithms

To examine what this sort of algorithmic growth in performance we can look at examples from classical comptuational science. Linear algebra is an archetype of this sort of growth. Over a span of years from 1947 to 1985, the algorithmic performance matched the performance gains from hardware. This meant that Moore’s law for hardware was amplified by better algorithms. Moore’s law is the result of multiple technologies working together to create the exponential growth.

In the 1940’s linear algebra worked using dense matrix algorithms that scaled cubically with problem size. As it turned out most computational science applications were sparse structured matrices. These could be solved more efficiently with quadratic scaling. This was a huge difference. For a system with 1000 equations this is the difference of a million instead of a billion in terms of the work done and storage taken on the computer. Further advances happened with Krylov algorithms and ultimately multigrid where the scaling is linear (1000 in the above example). These are all huge speedups and advances. A key point is that the changes above occurred over the span of 40 years.

The nature of these changes is quantum in nature where the performance of the new algorithm leaps orders of magnitude. The new algorithm allows new problems to be solved and is efficient in ways the former algorithm is incapable of. This is exactly like what happened with the transformer. In between these advances the new algorithm is optimized and gets better. It does not change the fundamental performance. Nothing amazing happens until something is introduced that acts fundamentally differently. This is why there is a giant AI bubble. Unless another algorithmic advance is made, the LLM world will not change dramatically. The power and fears around AI is overblown. People do not understand that this moment is largely algorithmically driven.

These sorts of leaps in performance are not limited to linear algebra. In optimization a 1988 study showed a 43,000,000 times improvement in performance over a 15 year period. Of this improvement 1000 was due to computer improvements, but 43,000 was due to better algorithms. Another example is the profound change in hydrodynamic algorithms based on transport methods. The introduction of “limiters” in the early 1970’s allowed second-order methods to be used for the most difficult problems. Before the limiters the second-order methods produced oscillations that resulted in unphysical results. The difference was transformative. I have recently shown that the leap in performance is about a factor of 50 in three dimensions. Moreover the results also compare to the basic physical laws in ways the first-order methods cannot produce.

How do algorithms leap ahead?

“This is the real secret of life — to be completely engaged with what you are doing in the here and now. And instead of calling it work, realize it is play.” ― Alan Watts

Where do these algorithm breakthroughs come from? Some come out of pure inspiration where someone sees an entirely different way to solve a problem. Others come through the long slog through seeking efficiency. The deep analysis yields observations that are profound and lead to better approaches. Many are pure inspiration coming out of giving people the space to operate in a playful space. This playful space is largely absent in the modern business or government world. To play is to fail and to fail is to learn. Today we have everything planned and everyone should know that breakthroughs are not planned. We cannot play; we cannot fail; we cannot learn; breakthroughs are impossible.

“Our brains are built to benefit from play no matter what our age.”

– Theresa A. Kestly

The problems with algorithm advancements are everywhere in today’s environment. Lack of fundamental trust leads to constrained planning and lack of risk taking. Worse yet, failure is not allowed as the essential engine of learning and discovery. This sort of attitude is pervasive in the government and corporate system. Basic and applied research is both lacking funding and that funding is not free to go after problems.

In the corporate environment the breakthroughs often do not benefit the company where things are discovered. The transformer was discovered by Google (Brain), but the LLM breakthrough was made by OpenAI. Its greatest beneficiary is Google’s rival microsoft. A more natural way to harness the power of innovation is the government funding. There laboratories and universities can produce work that is in the public domain. At the same time the public domain is harmed by various information hiding policies and lack of transparency. We are not organized for success at these things as a society. We have destroyed most of the engines of innovation. Until these engines are restarted we will live in a fallow time.

“There is no innovation and creativity without failure. Period.”

― Brene Brown

I see this clearly at work. There we argue about whether to keep using 30, 40 and 50 year old algorithms rather than invest in the state of the art. They then convince themselves that it is good because their customers like the code. The code is modern because it is written in C++ instead of Fortran. The results feel good simply because they use the most modern computing hardware. Our “leadership” does not realize that this approach is getting substandard return on investment. If the algorithms were advancing the results would be vastly improved. Yet, there is little or no appetite to develop new algorithms or invest in research in finding them. This sort of research is too failure prone to fund.

“Good scientists will fight the system rather than learn to work with the system.”

– Richard Hamming

Page, Lawrence. The PageRank citation ranking: Bringing order to the web. Technical Report, 1999.

Vaswani, A. “Attention is all you need.” Advances in Neural Information Processing Systems (2017).Vaswani, A. “Attention is all you need.” Advances in Neural Information Processing Systems (2017).

Margolin, Len G., and William J. Rider. “A rationale for implicit turbulence modelling.” International Journal for Numerical Methods in Fluids 39, no. 9 (2002): 821-841.

Boris, Jay P., and David L. Book. “Flux-corrected transport. I. SHASTA, a fluid transport algorithm that works.” Journal of computational physics 11, no. 1 (1973): 38-69.