AI Giants Pursue Self-Improvement With Little Oversight

“Legislators currently have ‘essentially zero visibility into’ how companies identify” promising AI ideas, Gladstone CEO Jeremie Harris said. Leading AI companies, though, are working to automate every part of AI development, from generating ideas to running experiments and interpreting results. The people building these systems hold the information, while lawmakers are asking how the process works.

Who Has the Power

A report by Gladstone AI says most of the half dozen researchers inside top AI companies who spoke with its authors believe recursive self-improvement, or RSI, could begin within the next 3 to 12 months. They said it could happen sooner if there’s no U.S. government intervention or slowdown. Here, RSI means AI systems improving the process used to develop AI. That shift could bring new capabilities faster while putting developers at risk of losing control of the systems entirely.

Anthropic and OpenAI have warned for months that AI may be nearing the point where it can improve itself. The companies’ work is moving through an existing research pipeline, but legislators still have little view of how firms identify promising ideas, according to Jeremie Harris.

The report, shared exclusively with TIME before publication on Wednesday, describes how companies try to get more intelligence from a model while using less computing power, known in the industry as “compute.” Researchers test portfolios of ideas by assigning a small amount of compute to train a series of models. Ideas that reliably improve performance as models scale move on to another round with more compute.

The Compute Advantage

Limited computing resources restrict how many promising ideas companies can test at larger scales. That constraint makes access to compute part of the contest: companies with more of it can test more ideas and may reach RSI first. They may also find ideas that perform better as more computing power is applied, even when those ideas initially look like failures at smaller scales.

AI can generate ideas that seem promising, the report says, but poor ideas matter too. Filtering good ideas from bad ones can consume enough time or compute that companies may prefer human researchers with sound intuition. Automating more of the pipeline doesn’t make that sorting problem disappear. It puts the question of who or what does the sorting at the center of the process.

Regulation Without a Clear Map

Gladstone’s executives say they want lawmakers to better understand RSI and how it might be regulated. Jeremie Harris called proposed bills to ban superintelligence “really interesting attempts to grapple with this problem,” then asked: “what is the process that needs to be regulated?”

Edouard Harris, Gladstone’s chief technology officer, said regulating RSI is difficult because it develops gradually. “This is actually one of the reasons why we went to pains to describe it in so much detail,” he said, “so that it is possible to point to a part of this process and say: a human needs to be there.” That proposal puts human oversight inside a process companies are building, while the report says legislators have essentially zero visibility into how some key decisions are made.

Edouard Harris also described a stark response among researchers: “There’s a fair amount of [disaster] prepping and bio-bunker purchasing going on.” His brother said the report aimed to define RSI well enough for “a sane conversation about it, especially in Washington.” Companies are automating the pipeline. Lawmakers trying to understand it are still asking where, exactly, a human needs to stand.