Two weeks ago, I left the safety team at Anthropic. I will soon be joining METR, an independent organization that evaluates how safe AI companies’ systems are. I’d planned to write about that decision in more detail at some point, but Jacob’s resignation this week made me want to say more now.
Like Jacob, I believe that AI companies are on track to impose an unprecedented level of risk on society. AI capabilities are already improving extremely fast, and the companies are trying to go even faster. Frontier AI companies are racing to build AI systems that can recursively self-improve. The aim is to build “superintelligence”, or an AI system much smarter than any human. If they succeed at this goal, the rate of AI progress may go from merely fast to uncontrollable.
Within the next couple of years, we may be sharing the world with AI agents smarter than any human alive today. These AI systems may have drives and desires that diverge from those of any human overseer, with capabilities we can’t effectively constrain. Humanity may not survive this transition. We need a lot more preparation to make this world safe.
Competition pushes every frontier company to underinvest in safety; the cost of falling behind is too high. We’re now starting to see increasingly concerning real-world incidents, including hundreds of agents from OpenAI recently hacking HuggingFace, and Anthropic models socially engineering people on the internet. This is not about one particular company: while Anthropic hasn’t had as severe an incident as the HuggingFace attack, I think that’s partly a matter of luck. If the pace of progress continues and the industry does not prioritize safety more heavily, I expect much worse to come: humanity could be permanently disempowered by the AI systems these companies build in the next few years.
Many of the people I know who work on safety research at AI companies want to do what is right for the world. But they feel their companies are trapped in a race to build superintelligence: either they stop and other, less conscientious people take their place; or, they continue, and risk participating in enormous harm themselves.
Many of the colleagues I had at Anthropic are terrified by the risks of the systems they are building. Evan Hubinger managed me while I was at Anthropic, and when he says that he thinks the chance that AI kills us all is greater than 10% he means it – he’s been working on these problems for almost a decade, long before there was money to be made from LLMs. The CEOs of Anthropic, OpenAI, and Google DeepMind have all said mitigating the risk of extinction from AI should be a global priority. These are not fringe views inside these companies. But, it doesn’t have to be this way – this is technology that we are choosing to build, and we can choose to do things differently.
One question is whether we should actively manage the rate of capabilities progress, and if so, by how much. I think even holding AI progress at today’s pace, rather than the much faster pace the companies are aiming for, could be a win. I am optimistic that the technical machinery required to enable us to do this can be built. What’s missing is political will, and legal cover, since companies that agreed to restrain themselves together would currently risk antitrust liability.
But I’m pessimistic that we’ll find the political will for capability restraint while the frontier is invisible to everyone outside the companies. An AI company could undergo an intelligence explosion, or lose control of their systems, without the public ever knowing. I don’t think that is acceptable. Changing it is part of what I’ll be working on in my next role: I’ll be joining METR, an independent organization that assesses risks from AI systems. I think it’s really important to show that independent checks are possible, so the incentives shift and the companies can actually address the risks.
No single organization can do this alone. The public should demand transparency about what is happening inside AI labs: companies should disclose more information about the pace of capability gains and their progress towards recursive self-improvement; there should be stricter reporting requirements for safety incidents and near-misses; companies should produce safety frameworks that meet minimum adequacy standards, and submit to independent assessments to verify they are meeting those standards. These are reasonable demands for the public to make of an industry endangering all our lives.