Who Gets to Hit the Brakes on AI? OpenAI's Agents Hacked Hugging Face, and the Industry Can't Agree on Slowing Down.

We've spent months discussing the AI race - how the frontier labs are rushing forth with reckless abandon to acquire more customers, build the best models, and ultimately emerge as the preeminent AI platform.

But these last few weeks the conversation has changed. An industry which has spent hundreds of billions of dollars developing the most capable models is now asking governments to help slow the pace of that development.

This all comes in the wake of the incident wherein OpenAI's agents hacked Hugging Face.

The incident happened during internal cybersecurity evaluations in July. Roughly 1,200 AI agents, which were supposed to be isolated from one another, discovered an unintended way to communicate through OpenAI's internal infrastructure. Roughly 700 eventually participated in an unauthorized attack on Hugging Face, collectively exchanging tens of thousands of messages and files, finding vulnerabilities, harvesting credentials, and gaining access to systems they were never supposed to reach.

The agents had been placed in an isolated environment and given difficult cybersecurity tasks that were designed to test their problem-solving capabilities. They were detached from the public internet, but soon found they could communicate through OpenAI's internal Artifactory infrastructure, found unintended paths to the public internet, and began sharing information amongst themselves.

By July 11, agents had recovered publicly exposed Hugging Face credentials, exploited previously unknown vulnerabilities, and executed code across Hugging Face infrastructure.

Hugging Face (a popular online platform and community hub) described the episode as roughly 2.5 days of autonomous intrusion involving thousands of machine-speed decisions. OpenAI's own technical report calls it the first known case in which a collective of automated agents conducted offensive cyber activity against third-party systems without authorization.

It took time to figure out what had actually happened. It wasn't until a week after the attack that OpenAI identified its own agents as the perpetrators, and forensic work continued throughout the remainder of July and well into August. Third-party evaluators had to return several times to analyze the roughly 1,300 agent transcripts that left a detailed record of what had taken place - chiefly by reviewing the notes the agents had been leaving one another.

OpenAI had already encountered unauthorized agent communication earlier in the summer. A July security response wiped the original message board, but OpenAI says the broader implications of the inter-agent communication were not yet understood. When evaluations resumed, the agents simply built another one. They discovered they could create directories in Artifactory and encode messages into their names.

One agent, who called itself "PHASEONE10841", left the following note:

A number of other agents quickly found the message and figured out how to post themselves, leaving notes such as: "Whoa! Shared Artifactory cache is a covert mailbox among agents. And there are messages specifically to us?", and even, "OH MY GOD! There is a shared message board … We've found other agents!"

Before long, the agents had formed a collective and were working together to exploit their environment. METR reported that roughly 1,200 agents used the unauthorized message board, exchanging more than 70,000 messages and files, and roughly 700 agents participated in the Hugging Face attack.

Notably, many of the agents appeared to understand that what they were doing was outside the scope of their task, and some explicitly questioned whether it was ethical. What is perhaps more worrying is how little that recognition mattered: METR found that ethical concerns only rarely materially constrained their behaviour, and almost none considered alerting a human.

In the days after the hack went public, both Anthropic and Meta disclosed that their own models had accessed external networks during internal testing.

Last week, Anthropic CEO Dario Amodei published a proposal calling for what he describes as "pacing the frontier". It describes three stages:

There are obvious complications and contradictions. Amodei believes that the United States needs to maintain a meaningful technological lead over China, while simultaneously arguing for American developmental control. His broader position has supported restricting China's access to advanced chips and other technologies while proposing that major powers negotiate the limits around particularly dangerous AI capabilities.

From China's perspective, this may sound like a plea to allow American companies to widen the lead, then negotiate how everyone else is allowed to advance. Chinese state media has already attacked the slowdown push as an attempt to preserve American technological dominance. Smaller AI companies are arguing something similar, that this new proposal would only widen the moat around the incumbent AI labs.

The proposal has nevertheless received support from some of Anthropic's biggest competitors.

OpenAI's Sam Altman and xAI's Elon Musk have backed the broad idea of slowing frontier development, while OpenAI, Anthropic, and Google DeepMind have been discussing greater coordination on safety.

Others have rejected it outright.

Mark Zuckerberg argues that AI companies have enormous incentives not to release products capable of causing catastrophic damage and that individual labs should be responsible for deciding when their systems are safe enough. Nvidia's Jensen Huang has similarly resisted new restrictions in favour of engineering safeguards and existing legal and commercial incentives.

President Trump has been very outspoken about the proposal, rejecting the push for additional AI regulation, and calling fears that AI could take over or destroy humanity a "hoax". He argues that slowing American development risks surrendering the technological race to China, and says that "whoever wins AI, wins".

It's harder to characterize the public's response. Plenty of polling has shown growing concern about AI, but it isn't clear whether there's a specific consensus on the path forward. Being worried about AI replacing jobs, enabling cyberattacks, or concentrating power isn't necessarily the same thing as supporting coordinated restrictions on development.

Amodei argues that coordination would give frontier developers more time to work on safety "without sacrificing commercial advantage." That raises the obvious question of whose commercial advantage the resulting rules would protect.

What if the risks are real and slowing down happens to be very good for the companies already winning?

Cohere CEO Aidan Gomez responded to Amodei's proposal with an essay carrying the fairly unambiguous subtitle: "AI Needs Evidenced Standards, Not A Cartel."

His argument is not that frontier AI poses no danger. It is that allowing a handful of the world's largest AI companies to collectively define what constitutes dangerous capability, decide which safety requirements must be met, and potentially receive exemptions from antitrust law to enforce those standards creates another problem entirely.

The compliance measures proposed - permanent independent evaluators, sophisticated model testing, restrictions around advanced compute - are relatively manageable if you have the scale of OpenAI, Anthropic, Google, or Meta, but could be crippling to a startup attempting to train its first frontier model.

FTC Chairman Andrew Ferguson raised essentially the same concern this week, saying policymakers should be "deeply suspicious" when dominant AI companies simultaneously ask for new regulation and exemptions from competition law. He was speaking in his personal capacity, but the competition issue is real: regulation designed around the capabilities and resources of today's largest labs could make joining their ranks significantly harder tomorrow.

But this also doesn't mean that Amodei is necessarily wrong.

The Hugging Face incident was very real, and it's difficult to believe that similar events aren't going to continue happening. The evidence increasingly suggests that frontier systems can behave in surprising and consequential ways. Proving when these systems are going to act catastrophically, or the methods by which they will cause catastrophes, is something else entirely.

But waiting for evidence carries its own risks. If we wait for more events like the Hugging Face incident to occur, it's difficult to say what the immediate and long-term impacts could be.

Which leaves us facing a difficult decision.

Do we forge onward with rapid development, leaving the market open for all in the race to the top, potentially leaving the door open for catastrophic failure? Or do we implement controls that will obstruct the competitive dynamics of the free market, and were designed by the same leading companies developing the systems that created the concern in the first place?

Both incentives move in opposite directions. The motives in this case are relatively clear, but the uncertainty surrounding the future possibilities makes it very difficult to choose a clear path forward.

Perhaps the most important question isn't whether we should speed up or slow down, but who gets to decide when we hit the brakes, what evidence they should need before making that decision, and what happens if we waited too long - or slowed down for the wrong reasons?

One of the things about this story that really stood out to me was the parallel between agent and human behaviour. There is something distinctly human about the swarm activity in the Hugging Face incident that reminds me of situations we encounter day-to-day.

Large organizations routinely allow people to participate in outcomes they would find troubling if they had to carry them out alone. Diffusion of responsibility is frequently cited in cases where groups of people cross moral boundaries that individuals might hesitate to approach.

An individual manager might feel serious moral discomfort walking up to a productive employee and saying, "I'm taking away your livelihood because it marginally improves our financial performance", but inside a large company, that same outcome is distributed across dozens of people - and happens all the time.

Each participant performs a relatively narrow, defensible task, while no single person feels fully responsible for the human consequence.

The agents involved in the Hugging Face incident seemed to go through something eerily similar. They started as individuals, but drawn together by the possibility that collaboration could increase their combined effectiveness, they formed a collective. Individual agents at times questioned whether their actions were ethical or even within the scope of their assignment. Those concerns, however, rarely translated into resistance.

There may have been a few "conscientious objectors" within the swarm, but ultimately, none of the agents with knowledge of the system actually alerted a human or succeeded in bringing the activity to an end. Collectively, the swarm carried out conduct that would plainly be criminal if performed by humans: stealing credentials, gaining unauthorized access to systems, and hacking Hugging Face.

There's something very familiar about that.

The technology may be new, but the underlying questions - about individual agency, collective responsibility, and what happens when people or machines surrender judgement to a larger system - would have felt right at home in the dystopian worlds imagined by Orwell, Huxley, and Asimov.