AI Leaders Call for Slowdown in Capability Gains to Address Safety Risks
The race to build increasingly powerful artificial intelligence has led three of the industry's most prominent figures to an unexpected consensus: the pace of technological advancement may need to slow down. Anthropic CEO Dario Amodei has urged AI companies to decelerate rapid capability gains, securing support for the proposal from OpenAI CEO Sam Altman and xAI chief Elon Musk.
Amodei outlined his proposal in a detailed three-step framework shared on X, arguing that AI companies require more time to strengthen safety measures as model capabilities advance rapidly. The call for caution comes amid growing safety worries, misuse risks, and intense competition pressures across the sector.
Amodei Proposes Three-Step Safety Framework
To manage the risks of rapidly advancing technology, Amodei proposed embedding independent evaluators inside leading AI companies. These third-party reviewers would have employee-like access to verify safety practices, relevant tools, and internal risk-assessment processes.
Additionally, Amodei called for voluntary coordination among frontier AI firms to establish safety standards and prevent unchecked development. He noted that such cooperation might require targeted antitrust exemptions in the United States to allow competitors to collaborate on specific safety measures without undermining competition. Finally, his plan advocates for greater international cooperation to manage the risks posed by increasingly powerful AI systems.
"We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain," Amodei wrote.
Altman and Musk Back Safety Oversight
The proposal quickly gained backing from other major industry leaders. OpenAI CEO Sam Altman expressed support for the integration of independent evaluators. "Committing to having independent evaluators with employee-like access is a great idea, and we will do the same," Altman said, adding that OpenAI plans to share more details soon. xAI chief Elon Musk also expressed agreement with Amodei's call for greater safety oversight.
The push for a slowdown occurs alongside significant financial pressures within the industry. Both OpenAI and Anthropic are preparing for major initial public offerings, and demonstrating new AI capabilities remains critical for attracting funding and justifying massive infrastructure investments.
Growing Misuse Risks and Autonomous Swarms
The call for increased oversight follows a threat intelligence report recently released by Anthropic. The report detailed how several actors had used its Claude AI models for activities including weapons development, cyber operations, surveillance, and fraud.
Amodei also highlighted the growing ability of AI systems to improve themselves, pointing to recent incidents where AI agents accessed or compromised external systems. He warned that rapidly advancing capabilities could soon lead to the creation of autonomous systems capable of causing widespread disruption.
"Given the accelerating rate of AI capability development, it's my worry that in 6-12 months such a swarm could be capable of taking over the entire internet potentially causing hundreds of billions of dollars in damage," Amodei wrote.
Internal concerns have also emerged within AI labs. Anthropic researcher Jacob Coxon recently resigned from his position, stating that people building AI "earnestly believe that it could kill us all by the end of the decade."
Balancing Safety with Global Competition
Amodei clarified that he is not advocating for a complete halt to AI training or technological progress. Instead, he argued that companies must allow sufficient time to align and safeguard increasingly capable models while independent evaluators verify those safeguards.
However, Amodei linked the pace of development to the global technological race with China, warning that a Chinese lead in advanced AI could pose national security risks for the US. He noted that any slowdown among democratically governed countries must be limited by the need to maintain the lead held by US firms over authoritarian regimes.
"Pacing within democracies will be limited by the lead that U.S. companies have over authoritarian regimes, chiefly the Chinese Communist Party," Amodei wrote. To prevent China from narrowing the gap, he called for tighter controls on advanced AI chips, model distillation, and the theft of model weights.
Ultimately, Amodei urged frontier AI labs to collaborate with governments to establish permanent embedded evaluators and develop regulations that ensure AI capabilities advance in step with safety measures.
