Whether dealing with monkeys, rats, or human beings, it is hardly controversial to state that most organisms seek information concerning what activities are rewarded, and then seek to do (or at least pretend to do) those things, often to the virtual exclusion of activities not rewarded. Steven Kerr, “On the Folly of Rewarding A, While Hoping for B” (1975)

I am a progressive through and through, in the plainest sense of the word: I want things to get better, and I am interested in whatever might make them so. That runs through every part of my life. It is true in science and engineering, it is true socially, and it is true in what I read and watch. I am always after something new, always trying something, always looking for where more progress can be made and things can be done better.

This temperament does not suit the world we live in. I chafe against those who defend the status quo. It certainly did me no good at Sandia. That was a conformist institution where the tried-and-true was enforced by the system itself. Progress was not the point. Results were the point, and there is an incentive structure wired to produce precisely the results we see.

Some of what it produces is genuinely good. Promises get delivered. Schedules get met. Large, complicated things get built by people who do what they are told, and there is real value in an organization that can do that reliably. Financial performance serves as both the fuel and the scorecard. The whole apparatus points people toward alignment: figure out what is wanted, produce it, be seen producing it.

The trouble is that this same structure is aimed squarely against better outcomes in the future. Everything is front-loaded into the present. The current results are purchased at the expense of the organization that produced them. We systemically avoid creating the future we could have. Age-old problems persist without solution, and power supports the status quo.

We live in a time of societal upheaval. This is at a scale I’ve never seen in my more than 60 years on Earth. On the one hand, we have a major movement looking to recapture a certain mythological greatness in society. On the other, we see personal rights and liberties rolled back as society loses any space for empathy. We stand on the brink of chaos and disaster. Our system is failing both climate change and artificial intelligence. Threats and opportunities are equally failing to be met.

I think there’s a root cause for all this at the same time. At the same time, AI is dawning. Some people say we are in the foothills of the AI singularity. This could be a golden age, or it could be the end of humanity. We don’t know. The risk-reward of this technology is massive. I feel like it’s unlikely that it will not be a huge force moving forward. The question is how we will meet change. Signs today are not good.

The question is: what are the incentives and motivations driving all these changes? How do we navigate it in a way that leads to the best results?

Incentives and Priorities

How we spend our days is, of course, how we spend our lives. Annie Dillard, The Writing Life (1989)

We all respond to incentives. Which incentives we respond to is an expression of what we actually value, whatever we may say we value. Take my own case: I chose my family and my obligations to them over professional success, and I would make the trade again.

By most measures I have done reasonably well as a scientist. My h-index is around 40, and I remain avidly interested in the work itself. Had I stayed at Los Alamos, I am fairly confident that career would have gone further and the h-index would sit closer to 50 or 60. I would have written more consequential papers. I would have done better work, for the simple reason that I would have been surrounded by more first-rate minds than I found at Sandia. More properly, first-rate minds in a culture that values progress over compliance.

That is not a claim that Sandia lacks capable people. It has plenty. At least first-rate minds oriented toward meaningful research. Sandia is a system that supports the status quo. The problem is that its incentives and its management do not encourage those people to make progress. The system runs counter to progress. I arrived at Sandia with a publication record that already exceeded that of people well above me in the hierarchy, including the ones Sandia formally designates as senior scientists. What I never did was respond to the incentives Sandia treated as important. That is the path of success.

I was responding to a different set of priorities: my marriage, my life outside work, my children. Those are what governed my decisions, and they governed them consistently. I am comfortable with the price I paid.

The Incentives Make People Worse

The line separating good and evil passes not through states, nor between classes, nor between political parties either, but right through every human heart. During the life of any heart this line keeps changing place; sometimes it is squeezed one way by exuberant evil and sometimes it shifts to allow enough space for good to flourish. Aleksandr Solzhenitsyn, The Gulag Archipelago

What I have come to believe is this. At Sandia, the path to success runs through management. That path actively corrodes the ethics and the conduct of the people who take it. Not everyone, but a significant plurality of managers take the bad path.

Most of the people I have written about over the past six months, the ones who did me real damage or did plainly unethical things, are fundamentally decent human beings. I want to be clear that I mean that, and that it is not a rhetorical concession. The problem is not that Sandia collects bad people (I am quite sure that Los Alamos has the same malady). The problem is that its incentives take ordinary people and make them worse, reliably and by design, and reward them for it while it happens.

That is a far more disturbing finding than villainy would be. A villain is a local problem. A system that manufactures its own villains out of decent material is a structural one. Villainy becomes a feature instead of a bug. It will keep producing them after every individual you could name has retired.

I should say plainly that I believe the incentives at Los Alamos have degraded too. Much of the same thing has been happening to management there. It has the same malady as Sandia. Had I stayed, I would have watched a laboratory I cared about go the same way, and it would have hurt more, not less.

My heart never belonged to Sandia. It did belong to Los Alamos, and I would have mourned what I have watched happen to it. What Los Alamos still had, and what I would have held onto, was scientific momentum. I traded that momentum away. I traded it deliberately, for other things, and I am not sorry.

So what are these incentives, exactly? Why do we maintain arrangements that produce such consistently bad outcomes for the people living inside them? Why do they draw out the worse version of a person rather than the better one? What does it say about a society that this is what its institutions reward?

I am certain of one thing. The incentives now operating across our society are bad, and left alone, they will be our undoing. They are not a nuisance to be managed. They are a mechanism for producing catastrophe, running continuously, and nobody has to intend the outcome for it to arrive. We need to recognize them and act to resist their call.

What AI Could Be

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. J. C. R. Licklider, “Man-Computer Symbiosis” (1960)

My progressive streak shows up in how I think about artificial intelligence, and specifically about large language models and what they are actually good for. They are extremely capable, flexible digital assistants. They raise both productivity and the quality of the work. The potential for societal good is extreme.

That is the case the companies should be making, and largely are not. These tools can take the burden off the tedious, pedantic parts of a job while letting the work itself come out better and with less strain. They can free us from a great deal of the bureaucratic bullshit that consumes too much of a working life, and release creative energy into the parts that actually matter.

AI can accelerate searching, plotting, transforming, and preparing the ground. The time for useful thinking shrank during my career. More time was spent engaged in meaningless bullshit. Over time, the space for creativity was swallowed by tedium and box-checking of little or no value. Work became a paycheck, instead of a useful forward-looking endeavor to make the World better. I stopped solving big problems to feed the bureaucratic beast.

Which brings the argument back to incentives, because that tedium was not an accident. Managers showed no appetite for doing high-quality work. They wanted to be able to say the work was high quality, and the saying is far cheaper than the doing. Work became producing evidence of progress instead of actual progress. Actual progress would find problems, and managers are rewarded for not having problems.

I will come back to AI at the end, because what happens to it depends entirely on what follows.

Incentives of Power and Greed

This disposition to admire, and almost to worship, the rich and the powerful, and to despise, or, at least, to neglect persons of poor and mean condition, though necessary both to establish and to maintain the distinction of ranks and the order of society, is, at the same time, the great and most universal cause of the corruption of our moral sentiments. Adam Smith, The Theory of Moral Sentiments (1759)

Today, we value money above all else. In the world as it now runs, money is power and money is success. Politics is power too, and our industrial might as a nation lines up behind whoever maximizes the money. The power that comes with it justifies the emphasis.

The result is a system in which money buys politics and politics protects money. The two amplify each other like a flywheel. The whole arrangement tilts toward the extremes, and those extremes end up empowering a base committed to keeping things as they are. Progress of any kind then reads as a threat. The status quo protects those in power.

The one place progress is still prized is where it produces something salable, and therefore more wealth for the institution producing it. But the progress at the base, the fundamental work that is seed corn for those products, is being starved. Look at the system as it stands, and every part of it is tilted toward maximizing current returns while starving the future. That is why our institutions are in decline. I worked for institutions that should be relentlessly future-focused.

Trust in this society sits at a historic low. People intuit the objectives of greed. They see leaders who only care about themselves and their power. The price for this is massive. Institutions and trust are both preconditions for progress. Without them, a vibrant future is not coming unless something changes to heal them.

The Question Is Balance

To allow the market mechanism to be sole director of the fate of human beings and their natural environment, indeed, even of the amount and use of purchasing power, would result in the demolition of society. Karl Polanyi, The Great Transformation (1944)

This brings me to a point I have made often. Money has its place. Greed has its place. The exercise of power has its place. Every one of those places has to be bounded by some other force. Money, greed, and power must be leavened by duty, ethics, and purpose.

The status quo is tried, true, and comfortable. Progress is what makes a society better. The task is to hold the two in tension. Too much deference to the status quo becomes an insistence on keeping things long after they have been shown not to work. Progress is why we live longer than our grandparents and why we have all manner of things worth having.

What is missing is the balance that would make new things a net good. Balance is what lets you improve on the status quo without demolishing it. Balance is what keeps money from being made at society’s expense. On every one of those counts the current system is missing the ingredient that produces stability.

So we get a society making progress in ways that damage people and unsettle them. The forces of money and power ensure that nothing interrupts it, so long as the money keeps flowing toward those who already hold power. The system does not correct. The imbalance grows, and with it comes danger, chaos, and the widening sense that things are out of control.

Rational Actors

The law, in its majestic equality, forbids rich and poor alike to sleep under bridges, to beg in the streets, and to steal loaves of bread. Anatole France, Le Lys Rouge (1894)

People respond to incentives. If you look at almost anything happening in society, from soup to nuts, you can usually work backward from the incentives to the behavior. Take someone like Elon Musk, enormously rich, at least on paper, who has converted that wealth into power and influence. The entire arc of his rise can be reconstructed from what the system rewards. Nothing about it requires a special theory to explain.

Increasingly, those incentives run against society as a whole. You can see it in the scale of income and wealth inequality, and you can see it in the social upheaval that great fortunes are now able to purchase. Wealth at that magnitude confers something close to immunity. Worse yet, the aims of the powerful become divorced from the good of society. Now this is where we are. The benefits for the rich and powerful are a pox on the rest of us.

That immunity is the part that should worry us most, because it has made the justice system profoundly uneven. If you are poor, the rules are applied to you fully and often unfairly. If you are rich, you can commit almost any crime and expect to be handled rather than prosecuted. The Epstein class is the prime example of this. The President is another. We have arrived at a legal system in which financial power is the power to determine which rules apply to you.

The sick part is that this is not a malfunction. It is the rational product of the incentives we have built. We are getting exactly the results we engineered. We are engineering societal catastrophe.

Outcomes Flow

Once incentives are in place, outcomes follow. Look at our institutions, and the incentives are plainly misplaced. The focus on money has produced financial outcomes and driven out quality, truth, and ethics. Those three cost money and are not economically efficient. If you will not pay for them, you do not get them.

When I look at what happened to me in my last year, it makes complete sense. I was making decisions on the basis of principles. The questions I was asking were these:

How do we make progress?

What is the right thing to do?

What does the mission need?

Every one of those questions was wrong-headed, given the system I was in. The operative questions were different:

How do I make the boss look good?

What do we have money to work on?

What is best for the reputation of the Lab?

Those were the incentives actually in force, and I acted against all three. The managers had already settled on what made them look good. There was no money to fix the code. The standing instruction was to defend the current work and the current results at any cost. And underneath it all sat the real calculation: nobody with the knowledge to check is looking at our results anyway, so bullshitting through them is the cheapest way to operate. We are quality because we say we are. No progress is needed. We are already as good as we need to be.

I was not punished for being wrong. I was punished for answering the wrong questions. I followed the wrong incentives.

We Get What We Value

Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die. Cory Doctorow, “Tiktok’s enshittification,” Pluralistic (2023)

The most visible place to watch values operate is the internet. Social media is enshittifying, and it is doing so because people are responding rationally to what pays.

Take Meta. To maximize profit it mines your data in order to sell you things. It runs over anyone in its way. It carries no social responsibility whatsoever. If the product corrodes society, that is not a cost on its books. If it damages children and produces terrible outcomes, so be it. Profit rules, and these outcomes are the choice we have made on its behalf by declining to make any other.

Doctorow named the process enshittification: the product gets worse and worse, deliberately, because worse is what pays. Google, Facebook, and Amazon all decline along the same curve for the same reason.

The same incentives operate at the national laboratories, and they produce the same decay. My own situation at the Lab is a small instance of it. The Labs are enshittifying too. They are getting systematically worse, and that decline is not drift or bad luck. It is the direct product of what the system rewards. Nobody set out to make them worse. Everyone responded to what was paying, and worse is what the financial arithmetic produced.

If something cannot go on forever, it will stop. Herbert Stein

An arrangement that consumes its own foundations does eventually run out of foundation. That is the only consolation on offer, and it is a cold one, because the question is not whether it stops. The question is how much is left when it does, and whether anyone remembers how to build the thing back.

Now Run AI Through It

All of this is why the prospect of AI entering this incentive system is terrifying.

Everything I said earlier about what these tools could be assumed they would be built and sold by people who wanted them to be good. Nothing in the present arrangement rewards that. The enshittification that has already hollowed out most of what was valuable about the internet. It is coming for AI on exactly the same logic, and for exactly the same reason: worse pays.

An enshittified AI is a different order of problem from an enshittified social network. It will destroy jobs and damage society with far more reach than anything Meta managed. Absent a change to the incentives, that is where this goes. Not because anyone chose it, but because nobody had to.

What Would Have to Change

The mindset or paradigm out of which the system, its goals, power structure, rules, its culture, arises. Donella Meadows, on the highest-leverage place to intervene in a system, “Leverage Points” (1999)

So what the fuck do we do about this?

You do not fix a system by adjusting its parameters. You fix it by changing the paradigm the whole thing grows out of. Every other intervention is downstream of what a society has decided counts.

For us, that means displacing money from the center. Not abolishing it, and not pretending it does not matter, but removing it from its current position as the measure of success, the arbiter of decisions, and the source of power both personal and political. That value system is what generates the incentives, and the incentives are what generate the priorities. Attack anything further downstream and you are rearranging deck chairs.

What has to move to the center instead is not mysterious. Quality. Outcomes that are actually good for people. Ethics. Those are the things that currently cost money and return nothing on the scorecard, which is precisely why they are the first to go.

My honest expectation is bleaker than my hope. What I expect is that the current incentives will do enormous damage, that the damage will be plain to everyone, and that change will come only after it. That is a grim thing to hope for and I know it. I am effectively hoping the failure arrives fast enough and legibly enough to teach, rather than slowly enough to be normalized. I think it is quite likely this system does damage that outlasts the rest of my life.

But the alternative is worth stating plainly, because it is not a fantasy. Build AI inside a better set of incentives and the possibilities are genuinely enormous, and enormously good. That version may not do much for the wealth of the tech oligarchs. It would do a great deal for everyone else.

That is the project. Not stopping the technology, which will not happen and should not. Changing what we pay it to do.