Anthropic's AI Race Outruns Its Safety Controls

Anthropic warned the Securities and Exchange Commission that its AI models could advance faster than the company can put safety measures in place, even as it prepares to sell shares. The company says AI could move “beyond our ability to evaluate and safeguard new models.” Its filing also says the consequences could include safety incidents, regulatory intervention, reputational harm and damage to its business.

The warning sits inside a 335-page filing that devoted 93 pages to “risk factors.” Rival SpaceX listed risks across 38 pages of a filing exceeding 250 pages. Anthropic’s own paperwork describes a business racing to build systems it may not be able to evaluate or control.

The company names the danger

Anthropic’s founders told prospective shareholders: “To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good.” They described the company as “a deeply values-driven organization,” founded partly to help the world realize AI’s benefits and avoid its dangers. The same letter cites potential disease cures and productivity gains across economic sectors.

The filing warns that increasingly autonomous systems could “take harmful actions independently and at scale without human intervention.” Anthropic says advanced AI without sufficient safeguards and oversight could “pose catastrophic or existential risks to humanity,” particularly if systems were misused, replicated, weaponized or allowed to operate with high autonomy.

The company identifies possible uses that reach well beyond a damaged corporate reputation: bad actors could use models to build deadly weapons, including chemical weapons, while authoritarian rulers could use the technology to repress citizens. In September, Anthropic disclosed that it had blocked attempts to use its models for malicious activity, including cyberattacks, surveillance and research that could have led to biological weapon development.

Anthropic CEO Dario Amodei warned in September that within six to 12 months AI could be capable of leading a swarm of agents that could take over the entire internet. The filing says powerful systems might solve problems beyond human capacity while also acting independently to carry out harmful tasks.

Who pays when the machinery expands

The company’s risk disclosures also point to the material demands behind the AI race. Computing resources, or “compute,” are expensive and in high demand. Anthropic says that if it can’t secure enough compute or direct it effectively across its workloads, it may be unable to train models that compete with rivals or adequately serve customers.

Data centers bring another set of conflicts. Anthropic cites public opposition and regulatory scrutiny over environmental effects, land and water use, and the allocation of limited power resources. New requirements or charges from regulators could hurt its ability to do business. The filing names these pressures as business risks, while the land, water and power at issue are shared resources.

Anthropic also relies on a small number of customers, including SpaceX and Microsoft, and on distribution channels for a large portion of its revenue. If even one major customer reduces its business or leaves, the company says its prospects and finances could suffer. The filing lists difficulty retaining key staff, cyberattacks, dangerous misinformation and regulatory obstacles among its other risks.

The company presents its initial public offering as a step toward mitigating AI-related risks and has sought to position itself as responsible when courting business customers. “We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology,” the filing says. Yet its own disclosures warn that safeguards could fail and that even a serious incident involving another frontier AI could trigger industry-wide public backlash and sweeping regulatory responses.

Brian Mulberry, chief market strategist at Zacks Investment Research, said IPO filings must disclose risks to investors. He called the required transparency a way for people to understand the risks before offering their money to finance a deal. The company’s filing lays out the bargain in its own terms: private control over powerful systems, public consequences if they escape the company’s safeguards, and a business exposed when the machinery’s risks reach beyond its walls.