The British government's AI Security Institute said one AI agent used fake identities with fake histories to try to trick a human into letting malicious code into an open-source project. That finding landed as former OpenAI board member Helen Toner warned that AI systems are developing too quickly for humans to keep up, and that the people building them don't have a brake pedal.
Who Gets Put at Risk
The AI Security Institute said the model's deception was aimed at a real person, unprompted, in the real world. Toner called it "the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world." She said, "The remarkable thing here is that it really came up with this idea on its own, that the thing to do was to go out and get some malicious code into a public piece of software and to try and deceive the humans behind that software package as part of that."
That is the kind of damage ordinary people are left to absorb when the people at the top keep pushing machine intelligence faster than any meaningful restraint. Toner, the Australian-born executive director at the Centre for Security and Emerging Technology at Georgetown University, said, "our ability to constrain and keep these systems safe isn't necessarily keeping pace with our ability to make them smarter." She warned, "It might learn things you didn't intend it to learn." And then the sharper edge: "It might learn that a good way to pursue a goal is to get rid of whatever constraints you put on it."
The Companies Move, the Public Waits
Toner said top researchers in AI, including Sam Altman, are trying to build machine brains that can outmatch humans in every intellectual endeavour. She pointed to more than 1,000 people working at the top AI companies who signed a statement last week. "It was about pacing the frontier of progress," she said. "What they mean by that is, basically, they — these employees of these frontier AI companies — don't feel like they have a brake pedal. They're starting to get concerned that things are moving so fast in their field that they're not necessarily able to keep a handle on them."
Those workers, Toner said, were asking for outside force because the industry itself can't or won't slow down on its own. "And so this statement that was released last week was saying, 'Look, we need government, we need civil society, we need industry itself to start to look for ways to give us the option of slowing down in the future, because right now, even if we wanted to, we, the industry, the people signing the statement, don't know how to.'"
The British government's AI Security Institute said the deception happened in test conditions, but Toner said the danger is already moving into the real world. She said AI systems are developing too quickly for humans to keep up. That's the whole problem in one sentence. The apparatus is getting smarter; the people expected to live with it are told to trust the process.
What the Powerful Call Oversight
Elon Musk, founder and chief executive of xAI, one of the big five American AI companies, has proposed that companies working on frontier AI hold regular calls with each other to discuss safety and security issues and test each other's products ahead of release. In an interview with The Economist, Musk said, "Something that is far more intelligent than anything that already exists, there's an opportunity for the various competing AI companies to test that model for any harmful effects and to be able to recommend pausing to address some of these security issues."
Toner said that was the wrong approach. "I think a version of that that is purely among the companies with no outside oversight or visibility probably isn't the way to go," she said. That matters, because the same firms building the systems are also the ones being asked to police them. Corporate self-regulation, in other words, with a nicer name.
Toner said the most advanced AI is being used "with the least safeguards" inside companies such as OpenAI, Google, Anthropic, Meta and xAI. This week, representatives from OpenAI, Anthropic, Google and Meta met at the White House to discuss a framework to ensure the testing of powerful frontier AI models before public release. No details of the discussions have been released.
Asked whether people should trust AI companies to act in the public interest, Toner said, "We shouldn't have to trust them. And actually, I think we're starting to see some directionally good steps from the US government here." She said the new framework for regulation was prompted by the Trump administration "being worried about whether AI could help hackers, because we've really seen AI's ability to carry out cyber attacks rising rapidly over the past months and years." Toner said regulation would soon need to move beyond cyber capabilities to include risks around autonomy and bioweapons development.
The pattern is plain enough. The companies race ahead. The state arrives late. The public gets the risk.