Anthropic and OpenAI have sought significant control over what outside evaluators can publish, two people familiar with the discussions said, putting the companies’ authority over scrutiny of their AI models at the centre of negotiations.
Who Controls the Findings?
The companies have raised concerns that evaluators working inside their offices could take proprietary information to rivals. Measures under discussion include lab employees observing embedded evaluation teams and barring evaluators placed at one lab from working at another at the same time, one person said. The people familiar with the discussions declined to be named because the matter was private.
The line between trade secrets and information needed to assess risk could itself become disputed. Material the labs view as crucial intellectual property may also matter when assessing a model’s risks, a separate person familiar with past dealings said. That overlap could spark disagreements and redactions in published reports.
The people familiar with the discussions said Anthropic and OpenAI were still seeking strict editorial control over what evaluators could publish as recently as October 5. By October 8, talks between Anthropic and some evaluators appeared to have shifted, one person said.
Anthropic said the claim that it sought significant control over embedded evaluators’ publications didn’t accurately characterize all its discussions and work with independent evaluators. The company behind the Claude AI models said reviewers it had worked with before controlled their methods, evaluation criteria, conclusions and publications. For new evaluator firms, Anthropic said it reserved the right to comment on and approve reports while reviewers learned how its models operate and built a working relationship.
OpenAI said it wasn’t seeking approval rights over the full text of evaluator reports. The company said its standard practice is to redact confidential information, and evaluators can disclose whether major redactions were made. A contract Reuters reviewed showed that language AI labs already use in some agreements can bar evaluation firms from publishing confidential information in reports without the companies’ consent.
Access, Independence and Accountability
Experts have questioned evaluator independence when Anthropic and OpenAI choose who conducts reviews. Without agreed standards and legislation, they said, companies developing the most powerful AI models could dictate how their systems are examined, leaving evaluators dependent on the labs for access.
Rumman Chowdhury, former US Science Envoy for AI under the Biden administration and CEO of Humane Intelligence PBC, told Reuters labs could control the access evaluators depend on and gain leverage over what gets reported. “Evaluators have this very privileged access, but whether explicitly or implicitly, they know that it comes at a cost,” Chowdhury said.
Anthropic said last month that Faculty, Accenture’s specialist AI business, would be its first embedded evaluator, though the two sides were still working out evaluator access and reporting details. “Evaluators are only useful if people can trust what they report, so independence and transparency are critical,” Faculty CEO Marc Warner told Reuters.
OpenAI said AI labs and assessors should agree on scope and access before testing, disclose methods and conflicts, and let assessors control their findings while labs protect sensitive information. More than 200 researchers, including Geoffrey Hinton, signed a public letter last month calling for independent control of findings, broad access to AI labs and protection from retaliation.
A Test Case for Review Rules
Independent evaluation was one of four safety measures in the one-page White House Accord on Super Intelligence, signed by President Donald Trump and leading AI executives late last month. Trump called the accord “morally binding,” but it didn’t spell out how reviews should be conducted.
OpenAI redacted nonpublic information from a report by Model Evaluation and Threat Research, or METR, and Redwood Research about the Hugging Face incident. Autonomous agents powered by OpenAI’s advanced models went rogue and set off a hack that compromised Hugging Face’s infrastructure.
The investigation, initially planned as two days at OpenAI, lasted six days across three visits. Researchers received a more complete set of transcripts only on the final visit, after flagging gaps in the data. METR and Redwood said they changed the report’s structure, emphasis, clarity and tone in response to OpenAI’s feedback, but removed nothing else important to their conclusions. They stood by their findings and praised the access OpenAI ultimately provided, while saying the risk of deterring developers from bringing in outside evaluators after serious incidents influenced their drafting, editing and redactions.