AI Firms Face Government Veto Over Model Releases

Anthropic presented US lawmakers with a draft bill allowing the government to block an AI model’s release if officials determine it could cause mass casualties or more than $1 billion in damages. The proposal sets thresholds for state intervention, while legal experts separately question whether existing law requires AI companies to report threats of violence.

Who Controls Release

Under Anthropic’s proposal, the government could block a model’s release if officials determined it posed an imminent risk of killing or injuring more than 50 people, or causing more than $1 billion in damages. The proposal ties withholding a release to a specific scale of harm. It doesn’t give the government a general power to block models for any risk; the anticipated threat must meet the stated criteria.

The draft identifies three ways the harm could arise: the model creating or releasing a biological weapon, the AI going beyond human control, or a cyberattack. So the proposed authority would depend both on the potential scale of harm and on whether the threat falls into one of those categories. The proposal concerns blocking an AI model’s release, not a general rule for all conduct by AI companies.

Politico first reported the proposal. Anthropic presented it to US lawmakers amid heightened concern about risks from rapidly advancing technology. This is a draft bill, not an enacted law. The proposed government power would directly affect whether an AI model could be released.

Reporting Duties and Legal Gaps

Separately, legal experts questioned whether existing US law requires AI companies to report threats of violence. Their discussion followed the February 10, 2026, mass shooting in Tumbler Ridge, British Columbia, and lawsuits alleging OpenAI failed to prevent it. The lawsuits make allegations; the account reports no legal finding that OpenAI failed to prevent the shooting.

Legal experts said existing duties to warn can apply to mental health professionals or parties with a special relationship. But they questioned whether those duties extend to AI companies. That leaves a separate issue from Anthropic’s draft: the proposal would let the government block some model releases, while the legal debate concerns whether companies must report threats of violence under existing law.

Risks of Monitoring

Experts also raised concerns about privacy, false threat reports and unequal monitoring. They discussed how difficult it can be to distinguish a credible threat from fictional writing, research or intrusive thoughts. Those distinctions matter to the reporting question: the account describes uncertainty about how to interpret a threat, along with concerns about who may face monitoring and the consequences of an inaccurate report.

The draft sets a high numerical threshold for government action, but the legal questions experts describe concern reporting duties and the interpretation of individual threats. The account doesn’t specify how the proposal would address privacy, false reports or unequal monitoring, and it gives no details about provisions concerning workers, unions, company profits or compensation. It presents two separate policy questions: when the government could stop a model’s release, and whether AI companies already have a duty to warn about threats.