
Anthropic and OpenAI are among the frontier AI companies actively working to automate AI R&D.
Anthropic, for example, created the Anthropic AI R&D Automation Index, and as of August 2026, it states that Claude ‘leads’ 26% of Anthropic’s AI R&D work and that the share of work at or above ‘AI collaborates’ is above 90%.
OpenAI recently announced that it has reached the goal of having an “automated research intern.” It now aims to create an automated AI researcher by March 2028.
It sounds interesting and even fascinating until we learn about the serious implications of what researchers call an “intelligence explosion” as a result of AI R&D automation.
A recently published paper titled “What If Automating AI R&D Triggers an Intelligence Explosion?”, authored by Alan Chan et al. (among the co-authors are Geoffrey Hinton and Yoshua Bengio, two of the “godfathers of AI”), addresses the topic.
It defines the intelligence explosion, driven by the automation of AI R&D, as “a dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less.”
If that happens, the paper states, AI systems would rapidly eclipse human experts across most domains and radically accelerate technological progress.
From a societal perspective, this would have two main implications:
The extremely rapid development of highly capable or superhuman AI systems;
The likely deployment of those systems to develop new technologies and act in the world.
And this would lead to three main societal risks, each deserving close attention because it could have serious and systemic consequences:
Capabilities growth outpacing society’s capacity to steer and adapt;
Loss of oversight and control;
Erosion of checks on power.
From an AI governance perspective, these implications are extremely serious and far too risky. Their plausibility would demand an urgent global call to stop any practice that could lead to this “intelligence explosion.”
To understand why, consider what loss of oversight and control means in practice: AI would have become fully ungoverned and would have agency over humans and human society.
It would be the effective embodiment of sci-fi-inspired dystopian scenarios of AI domination.
The paper suggests three main avenues or policy actions:
Obtain visibility into companies’ automation of AI R&D;
Develop ways to steer and constrain an intelligence explosion;
Prepare to adapt to an intelligence explosion’s impacts.
This is an extremely timely and important paper, and a great read before the week starts. Download it here.
*Given the importance of the topic, we will discuss this paper in depth during our next AI Governance Workshop. Join us!




It’s hard to see what good an “intelligence explosion” would do in the world if it does not relate to raising the intelligence of any people, but simply raising the intelligence of machines.
Does anyone really want a world filled with increasingly smart machines and increasingly dumb people? Only those that think they would benefit because they control the smart machines, and through them, the dumb people. After all, people bereft of critical thinking and learning skills are much easier to control.
OpenAI now requires documented risk assessments and safety cases before reinforcement-learning training runs, and each of three senior leaders can veto a run. It adopted the rule only after a 20 September run in which the automatic stop failed and staff halted training by hand 2.5 hours later.
That's oversight built after an incident, not ahead of one, and on my read it stays internal: I've seen nothing in the rule that gives outsiders visibility into when the veto gets used or why. The paper's first policy avenue, visibility into automation of AI R&D, is this gap. If an intelligence explosion is the risk, the near-term test isn't whether labs have internal brakes, it's whether anyone outside the lab can see them being pulled.