The AI doom talk from ‘insiders’ is getting tiresome

Another week, another AI researcher announcing on social media that the end of humanity is somewhere between six months and two years away.

Jacob Coxon, a 27-year-old former pre-training researcher at Anthropic, told the BBC this weekend that people working on the technology were “genuinely frightened” by the pace of progress. “I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said.

Coxon joined Anthropic in May and resigned on Tuesday (9 September), a whole four months! Yet, his colleagues rushed to agree with him before the metaphorical AI-generated ink on his resignation post had dried.

“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” said Evan Hubinger, alignment stress-testing lead at Anthropic. “Many AI developer staff desperately want to slow down,” added Samuel Marks, scalable oversight lead at the same firm.

Their boss had already set the tone and continues to push a ‘gentle gentle’ approach while taking investor money ahead of a record-setting stock market debut. Dario Amodei, Chief Executive of Anthropic, published an essay on Saturday calling for the entire industry to slow down, and the bosses of OpenAI and xAI promptly said they agreed with him.

To be clear, I think the AI industry kind of sucks and there are genuine concerns with AI worth being angry about.

Workers are being replaced, the cost of building and running these systems is soaring, misinformation is getting cheaper to produce every month, and the amount of money sunk into the sector is categorically a bubble.

The AI tech industry itself is a problem, filled with the worst Silicon Valley types which are far less interested in building something than grafting and creating FUD.

However, none of those problems require a pause, a treaty, or a bunker. They require honesty, which is exactly what this new ‘extinction narrative’ lets all the decision makers avoid.

Look at who is talking and when they started talking. The people now warning of the apocalypse spent the past three years insisting that the only responsible thing to do was build faster than everyone else.

This immense scaling story was the whole pitch from the start. Bigger models, bigger data centres, bigger funding rounds, and a valuation of $852 billion for OpenAI built on the promise that whoever spent the most would win.

Hyper scaling was never about genuine demand. It was about winning the AI race by spending competitors into the ground, and seeing which models were left standing. Increasingly it’s looking like just one or two American models and the Chinese.

That leaves the big American labs with a problem. They need a credible off-ramp from a story they have been selling to investors, governments, and the public for months, and “we are pausing because we might kill everyone” is a far better headline than “we overbuilt and our unit economics never worked”.

Safety is the perfect justification for the pivot as it also justifies moving the whole enterprise into a closed, Manhattan Project-style setting, funded by the government, with the frontier labs as the only approved participants.

Amodei has already asked for mandatory third-party testing with the power to block or revoke deployment of frontier models. Hugging Face Chief Executive Clement Delangue and others have pointed out that this would leave Anthropic and OpenAI as a duopoly, which is a strange outcome for a proposal supposedly about protecting humanity.

The pattern of departures matters, too, and it increasingly reads like a pre-written script that comes with your stock options. There is now a recognisable arc where a researcher joins a lab, collects the compensation that comes with a frontier role, leaves, and immediately becomes a prophet of doom on social media.

Nvidia Chief Executive Jensen Huang dismissed Coxon’s warnings as untrue at a Goldman Sachs conference last week, and he has previously called the idea of AI ending humanity “complete nonsense”.

Huang sells the chips that power the boom, so his motives are no purer than anyone else’s, but his read on the theatre is correct if only because it’s being driven by pure capitalisim.

Anthropic is reportedly preparing a record-setting stock market listing. Its senior staff are simultaneously telling the BBC there is a double-digit chance the technology kills everyone.

Both of those things cannot be true in the way the people saying them want them to be true. A company that genuinely believed its product had a 10% chance of ending civilisation would not be preparing to sell shares in it to your pensions fund. They would be quickly co-opted by the government and/or nuked from fucking orbit – even by the AI-friendly Trump regime.

The real risks from AI are boring, grinding, and already here. Job losses, a capital bubble, rising costs, and a flood of synthetic rubbish on every platform deserve serious coverage and serious policy.

The AI-based end of the world does not need another doomer press cycle. It needs honesty, and the people ‘speaking out’ now to explain why they only discovered the problems on the way out the door.

Now read: Bad news for Brits stuck talking to AI chatbots