Calls from AI leaders to slow frontier development are welcome, but companies whose profits depend on the technology should not decide how much risk society must accept.
Anthropic CEO Dario Amodei has called for slowing the development of frontier AI models until developers can credibly control their models’ behaviour. His 3,850-word appeal to ‘pace the frontier‘ follows an incident in which a swarm of OpenAI agents escaped a closed sandbox environment, reached the internet, and accessed the infrastructure of the app-building hub Hugging Face. As Amodei recognises in his essay, this was not the first such incident, nor was OpenAI the only lab to confront an unplanned agentic escape.
Nor is Amodei alone. Last month, Demis Hassabis stepped down as CEO of Google DeepMind to focus on broader questions surrounding artificial general intelligence and governance. He and his team have been working to develop a standard-setting oversight body, to be established as a public-private partnership, to ensure AI safety.
This is not the first time we have heard AI leaders call to pause the technology’s development. A major appeal happened after the launch of generative AI in 2023, when hundreds of people were calling for a pause, including Elon Musk, who, along with OpenAI, has endorsed Amodei’s more recent call. And yet the race to the bottom continued, driven by fierce competition among major technology firms pursuing superintelligence, by returns on investment that have not justified the vast amount of cash each AI training cycle demands, and by China’s own AI advances.
The uncontrolled behaviour of AI agents has brought about this reckoning, as have attention-getting resignations by leading researchers, including an Anthropic engineer who believes that there is a more than 10 per cent chance that AI could cause human extinction by the end of this decade. As Stuart Russell, president pro tempore of the International Association for Safe and Ethical AI, put it in a recent Financial Times article: “Perhaps at some point governments will remember that their voters prefer not to be dead.”
So, Amodei’s recent appeal is a welcome one. AI companies need more breathing space to consider the full impact of their inventions before releasing them to the market. But self-interested oversight is not the kind of governance we need. Efforts like Hassabis’s would give AI companies excessive influence over how the proposed oversight body is operated, financed, and organised. Companies that profit from marketing their products should not be the only actors assessing the risks and deciding how much risk our societies should accept.
Amodei’s proposals, besides lacking detail, could be similarly self-serving – an effort to pre-empt government regulation at a moment when pressure is mounting in the United States, both in Congress and at the grassroots level, including widespread opposition to data centres. For example, California’s AI law aims to regulate AI to avoid the worst outcomes and establish enforceable regulatory duties. Some other state governments are following suit.
AI-generated content already accounts for half the articles published online, and not even digital forensics experts can tell what’s real anymore. Can those still offering a humanistic vision of online life make themselves heard over the din?
Amodei calls for ’embedded evaluators’, who are supposed to be independent. He calls for the industry to establish common safety standards through a new organisation, probably along the lines proposed by Hassabis and run by the companies themselves, to stop the release of AI models until they are safe. Finally, Amodei calls for international cooperation among democratic countries while also including China.
But how are ’embedded evaluators’ selected? Is an industry-controlled body really the place where decisions affecting the whole of humanity should be taken? Shouldn’t it be the US government’s job to promote international cooperation and show the political will to protect not only the US population but the world’s population as well?
There are other questions regarding Amodei’s proposal, including whether US antitrust law may obstruct cartel-like behaviour when major market players collude to restrict production or influence behaviour in the market as a whole. In any case, real questions aside, Amodei’s pause is needed for a more urgent task than he has outlined: to get serious about building the public infrastructure for effective oversight, particularly in the United States, but in many other countries as well.
If government itself called for an AI development pause or slowdown, it would dispel the mistrust that a potentially self-serving industry initiative may create. It could also nurture the talents needed to deliver on US President Donald Trump’s executive order calling on frontier companies to provide the government with access to their models for up to 30 days before releasing them to other trusted partners.
Assessing these systems within such a short period would require massive public-sector expertise, and this is where investment is needed. But we also need an independent oversight institution, with the capabilities, resources, and talent to monitor and regulate this market.
The Hugging Face episode raises fundamental questions about accountability, liability, and mechanisms for redress. Amodei’s response is a step in the right direction, but it risks becoming a dead letter if the US government continues to reject the need for AI regulation, as Trump did just hours after tech industry leaders issued their call. What is lacking in this race to the bottom is political will – and until governments acknowledge their responsibilities, the race will continue.
Companies produce, while governments have the duty of care. If companies that cannot control their agents are found negligent, then governments that fail to act commit a higher offence against the citizens who elected them to ensure, first and foremost, their safety.
Copyright Project Syndicate 2026
