Jacob Coxon, a 27-year-old former Anthropic researcher, told the BBC that people working on artificial intelligence were "genuinely frightened" about how fast the technology is advancing and what it could mean for humanity. He said, "I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future." That warning landed after his resignation post on the dangers of AI went viral, and it cut through the usual corporate fog around a sector that keeps sprinting while its own workers say they’re scared of the finish line.
The Race Nobody Can Opt Out Of
Coxon said his former boss, Anthropic head Dario Amodei, recently called for AI development to slow down, though some have questioned the motivations behind that call. The bosses of two rival AI firms, Sam Altman of OpenAI and Elon Musk of xAI, have both said they agree with Amodei's proposal for industry-wide deceleration and regulation, as well as independent monitoring of AI model development. Amodei wrote in an essay on Saturday that developing the technology was not in question, but the risks associated with it were "serious," and that companies and governments must be given time to address them. The language is familiar: keep building, keep profiting, then ask the state to tidy up the mess after the market has already set the pace.
Coxon, who worked at OpenAI before joining Anthropic, welcomed the suggestion of a slowdown but said it would need to be co-ordinated with China to avoid "a race at an international scale." He said, "The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they're scared of the outcomes of that race." That’s the logic of the whole machine. Compete first, regulate later, and call it responsibility once the danger becomes impossible to ignore.
He said the hardest question to answer was what an AI apocalypse would look like. One of the risks outlined in Amodei's comments was of a swarm of bots acting like a supercomputer that could take over the internet. Coxon said that scenario could be realistic in six months to a year. He said his peers feared the danger could even arrive in the next two years. Staff at AI firms were, he said, "planning what to do with their lives and thinking about the impacts of their work," and some were "considering buying land somewhere because they're so scared of the instability as a result of rapid AI progress." He added, "They all keep this in their head on a daily basis while working on the technology."
Profit, Panic, and Public Relations
Coxon also said the people researching the technology "genuinely want to see the upside" of things like "solving diseases and improving everyone's lives." That’s the pitch, anyway. The same firms that sell the future as salvation are now asking for a pause because the future might bite back before the next funding round closes.
Many others in the industry have voiced concerns since Coxon's social media post, including Anthropic scientist Evan Hubinger, who said, "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." Computer scientist and Nobel Prize winner Geoffrey Hinton, known as the "Godfather of AI," told the BBC on Friday that a 10% chance of AI killing all humans was "not unreasonable." Marc Warner, chief executive of AI safety firm Faculty, told the BBC's The World This Weekend that it was "extremely hard to place a probability" on AI killing all humans, but said it was important to recognize that the people making the warnings are very sincere.
Former Prime Minister Rishi Sunak, who is a paid adviser to Anthropic, wrote in the Sunday Times that he too was worried about the risk of AI to humanity despite generally being optimistic. The revolving door keeps spinning. The same political class that blesses the industry also turns up later to sound alarmed, as if concern were a substitute for control.
Some industry figures have suggested comments about the perils and power of AI may be designed to generate hype. Clement Delangue, chief executive of the AI platform Hugging Face, said on social media: "Sorry, but asking Jacob [Coxon] about AI extinction risk is like asking your AC guy about climate change. Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective." After Amodei's essay was published, Delangue offered to help be a part of the potential solutions Amodei proposed. Even the skepticism comes wrapped in the same ecosystem, where everyone is selling something, whether it’s caution, scale, or the next fix.
The Market Wants Shares, Not Answers
Nvidia boss Jensen Huang discussed Coxon's comments at a conference hosted by Goldman Sachs last week, multiple people in the group told the BBC, and they said he dismissed them as untrue. Huang has previously said the notion that AI was "going to be the end of humanity" was "complete nonsense." The denials are brisk, the stakes are absurd, and the money keeps moving.
Critics say Anthropic has been trying to trigger a regulatory push to block competition, leaving it and OpenAI in a duopoly. Anthropic is reportedly preparing for a potentially record-setting initial public offering on the stock market, allowing people to buy shares in the firm. OpenAI, which was most recently valued at $852bn (£630bn), had been expected to do the same, but Altman said on Friday this would not happen this year, citing safety concerns. So the industry asks for restraint while eyeing the market, asks for oversight while guarding its moat, and asks the public to trust the people who built the race in the first place.