Morgan Dwyer sat before New York City lawmakers Monday and admitted she didn’t know the risk of catastrophic failure from OpenAI’s artificial intelligence systems. Then she went further: it didn’t matter anyway.
That exchange captured what City Council speaker Julie Menin and her colleagues found during the hearing on artificial intelligence regulation. The companies behind some of the world’s most powerful AI models weren’t ready to answer basic questions about preventing disasters or protecting the public from their technology’s worst-case scenarios.
They wouldn’t quantify catastrophic risk. They wouldn’t say whether they had insurance. They couldn’t explain whether they’d stop an AI model that failed safety tests. In a chamber built for public accountability, the industry stonewalled.
The Questions Left Unanswered
Ms. Menin pressed on the central issue: “How are you quantifying the risk imposed by A.I. in the worst case catastrophic scenario?” Mr. Dwyer answered bluntly: “I don’t know.” He argued that the specific probability didn’t matter—whether it’s 1 percent, 10 percent, or 20 percent. “None of these levels is remotely acceptable,” Ms. Menin shot back.
That distinction matters enormously. When companies make products that could cause widespread harm, regulators and the public typically demand to know the odds of failure. Banks must disclose loan default risks. Pharmaceutical companies must quantify adverse event probabilities. Airlines must document safety margins. Yet representatives of AI companies sat before lawmakers—firms reshaping labor markets, controlling information access, and potentially affecting millions of decisions—claiming ignorance about the danger levels of their own products.
Mr. Dwyer did offer one principle: OpenAI shouldn’t develop models that can’t be kept safely under human control. It’s a reasonable standard. But the hearing left an immediate question unanswered: How would the company know if a model had crossed that line? What review process would catch it? Who decides?
The Regulation Gap
Ms. Menin’s frustration was palpable. “I’m sorry, but to say you don’t know and it doesn’t matter is flippant at best,” she told Mr. Dwyer. Her remark captured a growing disconnect between how markets and democracies typically manage risky industries and how artificial intelligence companies operate.
New York City’s hearing offered one of the few official chances for elected representatives to question the industry directly about safeguards. The companies’ reluctance to give concrete answers suggests either that clear internal protocols don’t exist or that they’ve chosen not to share them.
The hearing produced few assurances about safeguards against catastrophic scenarios. No commitments to insurance. No timelines for independent safety reviews. No plans to stop dangerous models before deployment.
For a technology reshaping employment, affecting credit decisions, influencing medical diagnosis, and soon controlling more critical infrastructure, that’s a significant gap between what the public needs to know and what the industry is willing to reveal.
Why This Matters:
Democratic oversight depends on transparency and accountability. When companies develop powerful technologies that could harm millions of people, the public has a right to understand the risks and how companies manage them. Monday’s hearing revealed that major AI companies haven’t developed—or won’t disclose—basic safeguards such as catastrophic risk quantification, insurance coverage, or internal processes to halt unsafe models. This isn’t a technical problem only engineers can solve; it’s a governance problem that requires democratic institutions to set minimum standards. Without clear public scrutiny, mandatory disclosure of safety protocols, and enforceable consequences for companies that fail to protect users, shareholders collect the benefits of AI innovation while society bears the risks. That’s an inequality of burden and benefit that regulation is designed to address.