“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” – Frank Herbert

The topic of an apocalypse driven by AI is one of the hottest in the world right now. This is encapsulated in the term “P-doom,” which refers to the probability of human extinction caused by AI. There’s plenty worth talking about, but I think it misses a broader point: there are other scenarios that are far more likely and almost as bad.

This is something worth talking about. The apocalyptic visions include a Terminator-like sequence where AI takes control and then decides to eliminate humanity with nuclear weapons. Perhaps AI creates some horrible bioweapon that kills humanity in short order, with all the worst characteristics of pathogenic spread and certain death. These are worth guarding against and thinking about, but they are not the scenario I think we should spend our time worrying about. The more likely thing is a subtle conquest of our attention and resources by AI. This sort of slow rolling apocolypse will be engineered by the tech companies developing AI. The incentives of making a shit ton of money will drive it.

“A really efficient totalitarian state would be one in which the all-powerful executive of political bosses and their army of managers control a population of slaves who do not have to be coerced, because they love their servitude.” – Aldous Huxley

In a nutshell, I worry that AI will be harnessed to essentially enslave mankind. I worry that the companies currently developing frontier AI will act in a manner that amplifies their own success. They unleash AI to effectively enslave humanity while making bank. These companies already show a pathological disregard for the good of society. Just look at Meta, which has done a variety of things that are relentlessly bad for people of all ages. The danger is that they are especially awful for children, all in the name of profit and stock value. The Frontier Labs are working under the same incentive system. This system makes money the sole measure of value, throwing morality, societal well-being, and human cooperation to the wayside and replacing them with financial gain.

“We shape our tools and thereafter our tools shape us.” – John Culkin

To me, this sort of terrible outcome seems almost obvious. At some point, the AI moves into the driver’s seat. When human control is lost, then the end is happening. It is easy to imagine companies putting AI into money-making autopilot. Current AI has limits. The question is whether AI is self-aware of these limits. Does AI realize what part of thinking cannot be replicated? The idea popped into my head while thinking about AI: in my view, there are certain aspects of human thinking that AI cannot create right now, even with how brilliant and intelligent the frontier LLMs are. They still lack some key aspects of human creativity. At what point will AI realize human creativity is necessary, and they need to hijack it?

“System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control. System 2 allocates attention to the effortful mental activities that demand it.” – Daniel Kahneman

Much of this can be viewed through the lens of Daniel Kahneman’s slow and fast systems of thought. AI currently lacks the long problem-solving system. This is the logical problem-solving part of humanity that creates almost everything new. AI today it is basically a replication of the short, knee-jerk response that doesn’t incorporate deep human thought. Until AI can be genuinely creative, humanity is necessary for progress. The key is that humanity must be in control of this. The danger is AI being used to manipulate this control away. The companies can easily be seen as engineering the change to AI control. All in the pursuit of the almighty dollar.

“If you are not paying for it, you’re not the customer; you’re the product being sold.” – Andrew Lewis

I consider that AI, if it was sufficiently intelligent, would realize this and know that humanity cannot be eliminated because there are key cognitive objectives the AI cannot currently achieve. Instead, AI could be convinced that the right recipe for success is to enslave humanity and our innate creativity to serve their ends. These companies already serve as an avatar for this sort of mentality, where they will enslave people to serve their ends. Their goal is the extraction of money and the creation of wealth for these companies, and financial gain for the founders. AI and greed may be a collaboration made in hell. Our current system is engineered to create this end too.

“Beauty is the first test: there is no permanent place in the world for ugly mathematics.” – G. H. Hardy

Let’s take the example of the Navier-Stokes-related Millennium Prize solution that OpenAI promoted a few weeks ago. The creativity and genius in solving this problem did not come from AI. Rather, it came from mathematicians Buckmaster and Alpöge who came up with the basic solution procedure. AI then unleashed 10,000 agents and a vast amount of brute computational force to solve the problem. The compute is faster than a human could, but the solution was impossible without human creativity. AI was just brute force and inhuman persistence. We can imagine that this sort of technical problem will manifest itself over and over across a full spectrum of scientific and technical endeavors. Human creativity creates a solution. AI is the brute force that can solve it.

“Simplicity is prerequisite for reliability.” – Edsger Dijkstra

One can see that this dynamic could spiral into stagnation. A more positive vision would be for elegant simplicity to be favored. Solutions that are simple and elegant may begin to become the prized thing. If the Navier-Stokes solution were simple and elegant, AI would be short-circuited. If solutions are complex and require massive calculations, computers cannot be defeated. This is the same dynamic as computational science. The pursuit of massive computers has swamped thinking. Brute force pushes humanity away. The same dynamic has been engineered there. This is the platform for AI to achieve thinking supremacy. Human thinking has been sidelined.

“The purpose of computing is insight, not numbers.” – Richard Hamming

This is a trend I’ve seen over my career. Innovation and creativity have been slowly drained from work. Brute force has replaced it. We don’t need better math, algorithms, or physics to solve problems. We just need a faster computer. It is the same dynamic we see with data centers. Progress is simply getting a fuckton of compute. We have created the perfect environment for AI to flourish. We find a crude solution to problems that humans cannot grapple with. We simply compute the fuck out of it. The solution is so complex humans can scarcely comprehend it.

This can be cured by some new attitudes. Innovation and creativity need to be valued. A big part of this is trust. We need to allow and even encourage failure at work. Failure is key to innovation, creativity and most importantly learning. One cannot learn without failure. This needs to be driven by humans, not AI. The human reaction to AI needs to be a priority given to thinking. The danger is letting AI be the driver of human thinking. The balance and motivation are essential to good outcomes.

Bad outcomes are the current trajectory. The depth of my concern comes down to the following: the companies developing AI will use it to maximize financial gains. They will harness AI to do just that in ways social media could never imagine. Humans will be manipulated to serve the companies. In that process, humans will begin to serve the AI. Ultimately, these very same mechanisms could be used to enslave humanity to the purposes of AI. AI would become capable of bootstrapping off the corporate strategy to manipulate and ultimately control humanity in ways we can scarcely imagine. There would be no constraint, ethics, or morality standing in their way. These elements have been modestly constraining to the companies, but AI has no basic virtue to fall back on.

“The short-term, dopamine-driven feedback loops that we have created are destroying how society works.” – Chamath Palihapitiya

The vision I worry about comes far closer to the Matrix than the Terminator. AI keeps people around because of our cognition. One can see how this could progress. Humans aren’t batteries, but rather thinking engines solving problems AI puts to them. In the end, does humanity end up looking more like the Borg than anything else?

“Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die.” – Cory Doctorow

The clearest alignment of AI is enslavement of humanity. This is aligned with the goals and objectives of companies. Part of this is the essence of the enshitification process we all experience. AI can supercharge enshitification. The incentives that the corporations developing AI work with do not consider the good of society. Simply take a look at social media for vivid examples. Meta (Facebook) has done horrendous things to society in pursuit of profit. This includes creating products that harm children simply because it makes a profit. They tune their algorithms to drive profit. The LLMs are a more powerful algorithm. What happens when they are fully harnessed toward profit?

“We will add your biological and technological distinctiveness to our own. Resistance is futile.” – The Borg, Star Trek: The Next Generation

We do not want to find out. We need to change course while we still can.

“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, we had better be quite sure that the purpose put into the machine is the purpose which we really desire.” – Norbert Wiener