The Hugging Face incident was definitely a wake-up call for anyone claiming the models aren’t getting insanely smart. And even if you still view them as stochastic parrots, they can unintentionally cause a lot of harm.
In short: this was a cybersecurity training run in which the models were deliberately tasked with turning known vulnerabilities into working exploits. So it’s a bit less evil and frightening than the video made it out to be.
The “impossible” tasks they couldn’t complete came down to things like a spreadsheet whose formulas depended on Google Drive links the agents couldn’t reach. So to finish the job, they went looking for the answers, and eventually ended up inside Hugging Face, where a huge amount of the world’s AI models and datasets live.
How they got in
It’s pretty crazy how many vulnerabilities and workarounds they chained together to get into Hugging Face’s infrastructure:
- They found the shared package server in their sandbox (JFrog Artifactory) was the weakest point, and used previously unknown bugs in it to reach the internet.
- They turned that same server into an improvised message board, which roughly 1,200 agents used to share exploits and divide up the work.
- They chained two zero-days in Hugging Face’s dataset processing (an HDF5 file-read bug and a Jinja2 template injection) to run their own code.
- From there they escalated to cluster-admin and took about 17,600 actions across Hugging Face’s network in four days.
- May 2026
The reinforcement-learning run starts. The first escape attempt comes a day later.
- 11–13 Jul
Agents breach Hugging Face’s production infrastructure.
- 21 Jul
OpenAI and Hugging Face publish a joint statement.
- 5 Aug
OpenAI walks through the attack at Black Hat USA.
- 18 Aug
OpenAI pauses RL training for two weeks and holds its biggest planned run.
- 24 Sep
Australia reveals the agents also breached a government health-statistics portal in June.
- 26 Sep
OpenAI pauses again after another sandbox escape.
This isn’t production ChatGPT
Side note: these agents were running with deliberately lowered cybersecurity refusals, in a much less controlled environment than usual. The production version of ChatGPT would never let them get this far because of all the safeguards in place.
I watched the long technical breakdown at Black Hat when it came out, and they’ve clearly massively improved their security and safety work since: pausing reinforcement-learning training, holding back their largest training run and bringing in independent reviewers. I don’t think it will completely stop future incidents (it didn’t), but they took it very, very seriously, unlike what the press would have you believe, painting them as cavalier and avaricious.
Why it’s worth getting right
As for the doom and gloom: the models are definitely getting super smart and capable of insane stuff, from running companies to discovering new science and mathematics. Just this week OpenAI published 722 maths manuscripts produced by one of its internal models.
The promise of them contributing so much science, medicine and good to the world is very real, and it’s absolutely worth making sure they’re aligned and have our best interests at heart. Every day they’re held back by legislation and politicking is a day someone dies of cancer, or something else preventable, that they will absolutely be part of finding cures and treatments for in the very near future.
It’s not just a chatbot, the way people dismiss it. It’s a whole new industrial and cognitive revolution.
The China question
A huge part of the debate is China. They’re also racing to develop ASI (artificial superintelligence), but with far fewer of the same moral qualms. Chinese labs have signed safety commitments, but follow-through has been patchy, and historically the state has used technology to advance its agenda with impunity, never mind the human rights it violates. Look at the mass surveillance of the Uyghurs in Xinjiang.
A Chinese agent that reaches ASI before a Western one does, because we legislated the US labs into the ground, is the worst possible scenario for the future of the solar system. It entrenches a systemic advantage: whoever gets there first will probably stay smarter. I want to live in a free future, not a CCP-controlled one.
So any call for a pause, or any demonisation of the US labs, that doesn’t come with a plan for getting China to pause too is disingenuous at best and ignorant at worst. In my view, even a misaligned US model is probably better than an aligned one built to serve the CCP. Though, of course, we still really need to work on alignment a lot.
Neo-Luddism and nuclear
These things always get political, and I think we’re going to see a big rise in neo-Luddism over the next few years as people get fed bad information about AI. It’s part of why SpaceX is going so hard on orbital datacentres: up there, there are no zoning boards, environmental impact assessments or neighbourhood groups opposing them (you do still need an FCC licence).
I worry we won’t get the amazing, really utopia-level future we could have soon, because of bad political decisions made by ill-informed people. It’s the same way nuclear was shut down in Germany and elsewhere, despite being one of the cleanest and safest sources of energy we have.
Log scale. Nuclear, wind and solar are all around a thousand times safer than coal.
Source: Our World in Data
On jobs
I’m also really sympathetic to people losing their jobs to AI, facing a scary future that’s changing very fast, seemingly without their input. My job is already gone. It was one of the first to be replaced by AI, hence starting this company and working as fast as I can to prepare for a wild time ahead.
But we can’t stop advancing technology just because it puts people’s lives in flux. That’s like keeping waterborne illnesses and horse transport because people didn’t want to start using soap or learn not to step in front of cars.
There’s such good technology and such good times ahead, if we get it right. And I’m very optimistic that we will.