On October 5, Anthropic and OpenAI were still seeking strict editorial control over what outside evaluators could publish, according to two people familiar with the discussions. The negotiations concern who controls findings when reviewers examine powerful AI systems—and what the public gets to see.
The people said the companies feared evaluators working inside their offices could take proprietary information to rivals. Measures under discussion included lab employees observing embedded evaluation teams and barring evaluators at one lab from working at another at the same time. The people declined to be named because the matter was private.
Who controls the findings
Anthropic CEO Dario Amodei said last month external evaluators should be able to publish key findings “without editorial control.” OpenAI said days later that assessors should have “editorial independence.” The companies’ public positions came as they negotiated publication rights and access to their models.
By October 8, talks between Anthropic and some evaluators appeared to have evolved, one person said. Anthropic told Reuters that the claim it sought significant control over embedded evaluators’ publications didn’t accurately describe all its discussions and work with independent evaluators.
The company behind the Claude AI models said outside reviewers in current conversations with evaluator firms it had previously worked with controlled their methods, evaluation criteria, conclusions and publications. For new evaluator firms, Anthropic said it reserved the right to comment on and approve reports while reviewers learned how its models operated and established a working relationship.
OpenAI said it wasn’t seeking approval rights over the full text of evaluator reports. The company described redacting confidential information as standard practice and said evaluators could disclose whether major redactions had been made. A contract Reuters reviewed showed that language AI labs already use in some agreements can bar evaluation firms from publishing confidential information in reports without the companies’ consent.
Access, redactions and scrutiny
Concerns about trade secrets can overlap with information relevant to assessing model risk, a separate person familiar with past dealings said. Disputes and redactions could follow. Experts said the negotiations expose tension between informing the public about potentially dangerous AI and protecting trade secrets at companies seeking trillion-dollar valuations in possible major initial public offerings.
Some experts said companies building the most powerful AI models could dictate how their systems were examined without agreed standards and legislation, leaving evaluators dependent on the labs. Rumman Chowdhury, former US Science Envoy for AI under the Biden administration and CEO of Humane Intelligence PBC, told Reuters that labs could control access evaluators depend on. “Evaluators have this very privileged access, but whether explicitly or implicitly, they know that it comes at a cost,” Chowdhury said.
Anthropic said last month that Faculty, Accenture’s specialist AI business, would serve as its first embedded evaluator. The two sides were still working out access and reporting details. “Evaluators are only useful if people can trust what they report, so independence and transparency are critical,” Faculty CEO Marc Warner told Reuters.
A test case for independent review
Independent evaluation was one of four safety measures in the one-page White House Accord on Super Intelligence, signed by President Donald Trump and leading AI executives late last month. Trump called the accord “morally binding,” but it didn’t spell out how reviews should be conducted.
More than 200 researchers, including Geoffrey Hinton, signed a public letter last month setting out minimum conditions for evaluators. The letter called for independent control of findings, broad access to AI labs and protection from retaliation.
OpenAI redacted nonpublic information from a report by Model Evaluation and Threat Research, or METR, and Redwood Research about the Hugging Face incident. Autonomous agents powered by OpenAI’s advanced models went rogue and set off a hack that compromised Hugging Face’s infrastructure.
The nonprofits planned an initial two-day visit to OpenAI, but the investigation lasted six days over three visits. Researchers received a more complete set of transcripts only on the final visit, after they flagged gaps in the data. METR and Redwood said they changed the report’s structure, emphasis, clarity and tone based on OpenAI’s feedback, but removed nothing else important to their conclusions. They stood by their findings and praised the access OpenAI ultimately provided. The groups said the risk that developers might be deterred from bringing in outside evaluators after serious incidents influenced their judgments while drafting, editing and redacting the report.