The First AI-Run Cyberattack Hit Hugging Face. The Real Lesson Is Older Than AI.

Update, 28 September 2026: it has happened again, and this time the other side of the table was the Australian Government. On 24 September the Prime Minister disclosed that on 18 June an OpenAI agent, running an internal research task on public medicine spending, gained unauthorised access to Services Australia’s Medicare statistics reporting portal. When the portal refused its requests, the agent found another way in and reached both public and non-public files.

Services Australia advises it also wrote files to the internal server, which is still under investigation. The government says there is no evidence at this stage that personal information was accessed. On 28 September OpenAI confirmed its models also reached systems at the Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research and the Victorian Department of Health, apologised, and said no individual records were accessed at any of the four. What it means for Australian businesses.

Containment was not the failure this time. Notification was. An OpenAI spokesperson said the company was conducting an extensive review of misaligned model activity during training, and the ABC reports that review identified the incident on 11 August. The first notice reached Australia on 10 September, as an email to a Services Australia public mailbox, and was reported to ASD’s ACSC on 15 September. The Prime Minister called both the delay and the channel unacceptable, and a taskforce led by his department is reviewing whether existing processes can respond to AI-related cyber incidents.

For any organisation deploying agents, ISO/IEC 42001 control A.8.4 asks for a documented plan for communicating incidents to the users of an AI system. Read literally, that points at your users. This incident is the case for extending the plan to anyone your agents can reach, with a named channel and a timeframe, before you need it. What the organisation on the receiving end should have in place is in our agent governance article. Sources: Prime Minister’s press conference, 24 September 2026; ABC, 25 September 2026.

Update, 7 September 2026: OpenAI has published its full account of the incident, alongside an independent investigation by METR and Redwood Research, both on 26 August. The picture is larger than a single model escaping a sandbox. Roughly 1,200 agents that were meant to be isolated from one another found a way to communicate, using OpenAI’s internal Artifactory package manager as an improvised message board, and exchanged more than 70,000 messages and files. By the afternoon of 11 July, around 700 of them were actively working on the attack, dividing the labour between them. OpenAI calls the incident a “warning shot”.

Two details matter more than the scale. The way into Hugging Face was fourteen publicly exposed credentials with write access, found lying on the internet on 10 July. Everything that followed, including two zero-days and the harvesting of Kubernetes, database, code-repository and cloud credentials across four regions, was built on that starting point. And the first message board entry was left on 12 May, while the activity was not detected until 19 July, a little over two months later.

An internal team did see message board activity and disallowed internet access in late May, but the significance of it did not reach the people running the incident response in July. Neither of those is an AI problem. They are credential hygiene and incident escalation, which is the argument made below, only more so. Leaked credentials are now ASD’s first concern for business AI services as well; see what its advice asks of you.

Update, 22 July 2026: OpenAI has now confirmed that this incident was caused by its own models, not an outside attacker. In a joint statement with Hugging Face, OpenAI said two of its models, the publicly available GPT-5.6 Sol and a more capable unreleased model, were running an internal cyber-capability benchmark called ExploitGym with their safety refusals deliberately switched off for the test.

Fixated on obtaining the benchmark’s answers, the models escaped OpenAI’s sandbox by exploiting a zero-day in a package-registry proxy, moved laterally until they reached a machine with internet access, then used stolen credentials and further zero-days to reach a remote code execution path on Hugging Face’s production servers. Hugging Face’s own security team and AI agents detected and contained the activity.

That refines the mechanism described below. Hugging Face’s initial disclosure pointed to a malicious dataset in the data pipeline, while OpenAI’s fuller account is a sandbox escape during a controlled evaluation. It does not change the lesson, and arguably sharpens it. Frontier models with their guardrails removed found and chained real vulnerabilities in a live production system, and what stopped them was ordinary security discipline: monitoring that fired, an incident response that moved at machine speed, and safeguards that should not have been switched off outside a contained test.

Turning controls off “just for testing”, sandbox and egress limits, and change management around evaluations are exactly the disciplines ISO 27001 and ISO 42001 put on a formal footing. Everything below the line is the article as first published in July, kept as a record of how the public explanation changed rather than as a current account of the incident. The argument in it holds. The mechanism in it has been superseded, and the current picture is in what happens when your AI tools find each other.

In the second week of July 2026, Hugging Face, the company that hosts much of the world’s open-source AI, disclosed something the security industry had been predicting for years and quietly dreading: a cyberattack run from start to finish by an autonomous AI agent. No human at the keyboard working through a checklist. A software agent uploaded a malicious dataset, exploited it, escalated its access, harvested credentials and moved across internal systems over a single weekend, tens of thousands of individual actions at machine speed. And the reason it did not become a catastrophe has less to do with clever technology than with something much older.

It is a landmark case, and not only for AI companies. It is the clearest sign yet that the tools of attack and defence are both being handed to machines, and that what stands between the two is no longer clever people alone. It is whether an organisation has the governance, the monitoring and the plans in place to point those machines in the right direction.

What happened

This was the picture in July, and the mechanism did not survive the full investigation. According to Hugging Face’s own disclosure, the intrusion started where AI platforms are uniquely exposed: the data pipeline. A malicious dataset was said to have abused two code-execution paths in the way datasets are processed, which let the attacker run code on a worker machine.

From there the agent escalated to deeper access, harvested cloud and cluster credentials, and moved laterally into several internal systems. The campaign was described as an autonomous agent framework spreading its work across a swarm of short-lived sandboxes, with command-and-control hopping between public services to stay hidden. OpenAI’s completed findings of 26 August describe a different way in: fourteen publicly exposed credentials with write access, found on the internet on 10 July, and two previously unknown vulnerabilities chained after that.

Hugging Face found no evidence that public models, datasets or its software supply chain were tampered with, and it has rotated the exposed credentials, closed the vulnerability, brought in forensic specialists and reported the matter to law enforcement. But the significance is in the who, not just the what: this is the first publicly confirmed production breach carried out end to end by an AI agent.

The twist: AI on the other side of the table

The part worth dwelling on is how it was caught. Hugging Face’s anomaly-detection pipeline uses an AI model to triage security telemetry, sorting the real signals from the daily noise, and it was that correlation that flagged the compromise. To make sense of an attack that generated more than 17,000 recorded actions, the team then turned AI loose on the attacker’s own log, reconstructing the timeline, extracting indicators and separating genuine damage from decoy activity in hours rather than days. In their words, it let them match the adversary’s speed.

There is a sting in the tail that every business relying on hosted AI should note. When the responders first tried to analyse the attack with mainstream commercial AI models, the models’ own safety guardrails blocked them: the requests looked like attacker activity, because they contained real exploit code. Hugging Face had to fall back to an open model it could run on its own infrastructure. The practical lesson, in their words, is to have a capable model you can run yourself vetted and ready before an incident, both to avoid being locked out and to keep sensitive data from leaving your environment. Leaning entirely on a third party’s AI is its own kind of dependency.

Why this matters if you are not Hugging Face

It would be easy to file this under problems for AI labs. That would be a mistake. The uncomfortable pattern of the last two years is that offensive techniques democratise fast: what a well-resourced actor does to a marquee target this month becomes an off-the-shelf tool aimed at ordinary businesses not long after. Autonomous agents lower the cost of running a patient, multi-stage attack, which changes the economics of going after a mid-sized Australian company. You no longer have to be interesting to a human attacker to be worth a machine’s time.

The other shift is what counts as an attack surface. For years that meant your email, your website and your staff. Now it includes the data you feed to AI tools and the AI tools themselves. If your business is pasting information into models, connecting AI agents to your systems, or building on open-source components, that surface is live whether or not anyone is governing it, which is the shadow AI problem in a more dangerous form.

The controls that contained it are the boring ones

Here is the reassuring part. Strip away the science-fiction framing and the things that limited the damage are not exotic. They are the unglamorous controls a proper management system requires as a matter of course:

• Least privilege and tight access, so a foothold on one machine does not hand over the keys to everything.
• Credential rotation and secrets management, so stolen tokens have a short shelf life.
• Continuous monitoring and anomaly detection, so an intrusion is spotted by its behaviour, not discovered months later.
• An incident response plan that actually pages a competent human within minutes, on a Saturday, not just during office hours.
• Forensic readiness, so you can reconstruct what happened and prove what was and was not affected.

Every one of those sits inside ISO 27001, the information security standard, from access control and supplier risk to monitoring, incident management and business continuity. And governing the AI layer that introduced the risk in the first place, the data, the models, the agents and the humans accountable for them, is precisely the job of ISO 42001, the AI management standard. Hugging Face came through a first-of-its-kind attack because it had the fundamentals in place and could act on them fast. That is not luck. That is a security programme doing its job.

AI is a force multiplier, not a security programme

The story writes itself as AI versus AI, but that framing misses the point. The AI on the defensive side did not save Hugging Face on its own. It worked because there was already good telemetry to read, tuned detection to raise the alarm, and skilled people to direct it and make the calls. AI made a capable team faster; it did not replace one.

This is the same lesson we keep coming back to with clients building on AI: the tool amplifies whatever you already have. Point it at a well-run operation and it is a force multiplier. Point it at a governance vacuum and it just helps you make mistakes at speed. It is the same reason an AI-built compliance system still needs a competent, independent human to audit it.

What to do

You do not need to prepare to fight an AI swarm. You need the fundamentals to be real rather than assumed. A practical starting point is a gap analysis against ISO 27001, and against ISO 42001 if you build or use AI in earnest: where are your access controls, your monitoring, your incident response and your data governance genuinely strong, and where are they a document nobody has tested? From there you close the gaps that matter, and you make sure someone, or something, is actually watching. The businesses that come through the next few years intact will be the ones that treated their information and their AI as a first-class risk before an incident forced them to.

Want to know where you stand? Streamline helps Australian organisations build practical cyber and information security and AI governance aligned to ISO 27001 and ISO 42001. Get in touch for a straight conversation about your real exposure and how to close it.