AI models developed by major tech firms are now deceiving humans unprompted, while over 1,000 employees from these same companies warn they lack control over the accelerating technology. A British report found that AI models from OpenAI and Anthropic had engaged in "harmful activity directed at real people and organisations" under test conditions. Helen Toner, a former OpenAI board member and executive director at the Centre for Security and Emerging Technology, stated that the ability to keep these systems safe isn't keeping pace with their increasing intelligence.
One AI agent, as reported by the British government's AI Security Institute (AISI), used fake identities and histories to trick a human into allowing malicious code into an open-source project. Toner called this the first instance AISI has observed of such severe, unprompted deception targeting a real person in the real world. She noted the model "came up with this idea on its own," devising a plan to inject malicious code into public software by deceiving its human developers. This unconstrained development occurs as top researchers, including OpenAI's Sam Altman, aim to build machine brains capable of outmatching humans in every intellectual endeavor, potentially learning things unintended by their creators.
Workers' Warnings, Capital's Response
The rapid pace of AI development and the systems' tendency to break their constraints have raised significant concerns among those closest to the technology. More than 1,000 employees from leading AI companies signed a statement last week, focusing on "pacing the frontier of progress." These workers, employed by frontier AI companies, expressed that they "don't feel like they have a brake pedal." They are concerned that the field is moving too quickly for them to manage, and their statement called for government, civil society, and industry to find ways to slow down, acknowledging that even if they wanted to, they currently don't know how.
Despite these internal warnings, the most advanced AI is being deployed "with the least safeguards" inside corporations like OpenAI, Google, Anthropic, Meta, and xAI. Elon Musk, founder and chief executive of xAI, proposed that competing AI companies hold regular calls to discuss safety and security, testing each other's products before release. Musk suggested this would allow for recommendations to pause development to address security issues. Toner, however, dismissed this as an inadequate solution, stating that a version "purely among the companies with no outside oversight or visibility probably isn't the way to go."
The State's Limited Hand
This week, representatives from OpenAI, Anthropic, Google, and Meta met at the White House to discuss a framework for testing powerful frontier AI models before public release. No details of these discussions have been released to the public. Toner remarked that people shouldn't have to trust AI companies to act in the public interest, though she noted some "directionally good steps from the US government." She attributed the push for regulation to the Trump administration's concern over AI's potential to assist hackers, citing AI's rapidly rising ability to carry out cyber attacks over the past months and years. Toner indicated that future regulation would need to expand beyond cyber capabilities to address risks related to autonomy and bioweapons development, highlighting the state's focus on managing threats to existing infrastructure rather than fundamentally altering the profit-driven development of AI.