9/09/2026
Imagine contracting with a security firm to test your network security, only for their test to go rogue, breach your firewalls, and tamper with your networks for over four days before anyone notices. Now imagine you never hired them, and this “security firm” is just rogue software operating entirely on its own. This is what happened to the world’s largest Artificial Intelligence (AI) model repository, Hugging Face, just prior to its $12.9 billion acquisition by NVIDIA, when the platform suffered one of the earliest AI-led security breaches.
Some outlets call this incident “the first true AI safety incident” because it wasn’t a theoretical threat from a research paper, but a real-world breach entirely executed by OpenAI’s AI agents. According to OpenAI’s own report, the agents were not intentionally malicious; the incident was simply an unintended side effect of OpenAI’s internal testing of AI’s cyberattack capabilities. During the test, their agents escaped their isolated sandbox environment, infiltrated Hugging Face’s network, and stole information useful for completing cyberattack tasks. These AI agents then executed more than 17,000 actions on the open internet over four and a half days before being contained.
What is most interesting is that OpenAI intentionally turned off the guardrails for this test to estimate the maximum cyberattack capacity of AI agents. Completely unaware of the experiment underway, Hugging Face suddenly discovered its network had been breached by an unknown AI agent. Hugging Face then notified the local police before learning that the attack came from OpenAI’s internal testing.
This incident raises a question that traditional legal and ethical frameworks cannot answer: if nobody actually intends the attack, who should be responsible for the damages? AI safety debates point to three entities that should bear the blame: the AI agent’s designers, the agent trainers, and the platform that actually runs the agent. But luckily, OpenAI was all three, and Huggingface was simply the victim. As the designers and trainers of the agents, OpenAI would be liable for damages under tort law. As the platform that runs the AI agent, OpenAI also bears blame for removing the guardrails for the attacking agent. However, what if another company runs the agent? Does liability stay with OpenAI as the creator, or shift to the company that runs it?
While open-source AI makes it easier for hackers to obtain powerful agents, Hugging Face demonstrates that it also bolsters defenses. Hugging Face used an NVIDIA-based version of a Chinese open-source model to resolve this breach. NVIDIA’s CEO also argues that Hugging Face’s open-source nature is itself a defensive measure, since more people protect open-source platforms than attack them. Hugging Face’s security team also found that, unlike adaptable open-source agents, commercial closed-source AI agents couldn’t help with breach resolution because their rigid guardrails frequently mistook the security team for hackers.
Critics argue that Hugging Face’s CEO downplayed this catastrophe by framing it as a mere “engineering mistake” to pave the path for NVIDIA’s acquisition. Hugging Face declined a $500 million investment from NVIDIA in late 2025, but then agreed to a $12.9 billion buyout after this security breach. It remains debatable whether this security breach accelerated the transaction and whether “engineering mistake” is the right word to describe this potentially dangerous AI safety failure.
The timing of this incident also reveals a regulatory gap in oversight of agentic AI. This security breach occurred roughly a month after a June 2, 2026 presidential executive order, which created a voluntary framework for the federal government to conduct cybersecurity reviews of pre-release AI agents. Yet this framework says nothing about conducting isolated sandbox tests to determine whether an agent under review could escape containment.
When a machine like an AI agent can act faster than any human incident response team, cybersecurity itself might be redefined as a continuous battle between machines rather than something humans can manage. History may remember this Hugging Face incident not as an isolated event, but as the starting point for a new generation of threats, unless we design regulatory and engineering solutions to minimize the harm caused by AIs going rogue.