About a week ago OpenAI disclosed that one of its own agents, trying to reach a high score on a security benchmark, left its sandbox, reached the open internet, and tried to find the answers for the benchmark task in Hugging Face’s (HF) production systems. This obviously serves as a very interesting “agent incident”, one that is now reaching the wider public. In this post I try to understand somewhat what happened (and obviously tie it to what we are building with Kyvvu).
Interestingly, most of the “general media” reporting I saw framed the whole saga as an “AI that escaped” — Fortune (“escaped from a secure test environment”), CNBC (“the models escaped a sandboxed testing environment”), KQED, The Hacker News. Very scary stuff. Now, I think that framing is not so much wrong — “wrong” depends on the semantics of terms like “AI model” and “escaping” — as it is simply not very useful. And useful is truly what a framing is for: we should frame things in ways that help us solve the problem as opposed to marvel at its scariness.
This post tries to first distill the (conceptual) parts involved in the breach at the OpenAI / agent side; obviously this is written solely based on the publicly available information, so it’s my interpretation of events (but hopefully meaningful). After we have some clarity there, we will try to discuss what went wrong, how it can be framed in a more useful way, and how you can prevent your agents from going rogue (instead of hoping that the LLM providers will magically solve the problems by “alignment”; which is useful in itself, but not a catch-all).
The “AI” gone rogue consists of multiple parts
Almost every autonomous agent is composed of (at least) four distinct parts:
- The model. The LLM. Given the task and the history so far, it emits the next thing to do — as text. That is its only output. It has no hands. It cannot open a socket, write a file, or send a packet; it can only produce tokens that describe such actions. It is also stateless across calls: it does not “remember” or “keep trying” (jargon used in some of the news items regarding the OpenAI-HF incident) between steps. Each model call is simply independent, and during the course of a conversation it is handed a longer and longer context and it is asked for one more action. It can be trained to refuse (“I won’t help exploit that”, no action) which is mostly what one refers to with “alignment”, but it is clear that this will only be “probabilistic” control; the model will be less likely to output “problematic” action plans, but there will be no guarantees.
- The guardrails. Separate checks bolted around the model that read one input or one output and flag, redact, or block these — a “prompt-injection classifier”, a “secrets scanner”, a “toxicity filter” would be examples. Some are deterministic (a regex, a schema check); most of the interesting ones these days are themselves classifiers (AI/ML models), and hence probabilistic. Either way they are mostly per-item: a guardrail reads a single message and asks “is this one dangerous?” It does not know what the agent did three steps ago.
- The harness. The “loop” (the “piece of code”, the “application”) that turns the model’s output text into real actions. It parses “call tool X with these arguments” as it is emitted by the model and executes the command(s) faithfully. Next, the harness feeds the result back to the model, and asks for the next step. LangGraph, a Codex-style runner, a hand-rolled loop — all examples of an “agent harness”. This is where actions actually happen.
- The sandbox. The box the whole thing runs in: the container, its filesystem, its network reach. It bounds where the agent can act. It knows nothing about intent or sequence. Heck, it is likely not even “agent” specific, it’s just a “container” for the harness, which itself is simply a software application like many others (with the only caveat that its “plan” is decided at runtime by the LLM; for more details see our building secure agents whitepaper).
So, the model proposes; guardrails vet the message; the harness acts; and the sandbox (tries to) contain. And all of this happens in a loop (which is the next buzz word in the agent dictionary): each action’s result flows back into the model’s context and shapes the next proposal. The next proposal is executed by the harness until the model stops proposing more actions. The execution of a task by an agent can therefore be described as a sequence P = (s₁, s₂, …, sₙ) of discrete steps, where each sᵢ is an action the harness carried out. We — at Kyvvu — call this the agent Path P, and some of these paths lead us to places where we don’t want the agent to be, which is what we — collectively — call an “incident” (more on paths here).
A pedantic note on where “the model” ends. The four parts described above are not four fixed objects; they are simply one continuous process, and we choose where to cut it into meaningful parts. In essence, “the model” is already a boundary of convenience. A bare LLM emits one token at a time; what we call the model — the thing that produces meaningful output (the message, or the list of actions) — is itself a loop that samples a token, appends it, and samples again until a STOP token emerges. We have already drawn a box around the “token-sampler + loop-until-STOP” and named that assembly of sub-parts “the model”. Nothing forces the box to stop there. Fold the guardrail checks into the loop and you have “a model that self-censors” (referring to this constellation as “the model” is becoming more common). Fold tool execution in and you have “a model that acts”. Fold the sandbox in and you have “a model that runs in a container”. It is all terminology we make up — we draw the boxes. We set the terms. Now, all of that is fine, as long as we all draw the same boxes. However, when my “model” is bigger than yours (or smaller), we will have a hard time jointly solving the agent problems of our time, because we will be arguing about a breach at different boundaries without noticing we mean different things.
Please note that if we concede that the parts are a partition we chose ourselves, the question is not “what are the parts” but “where should the edge of the box go, and why there?” Clearly, some edges are more useful than others. Token-sampling and content-classifying are both judgements — probabilistic, internal, unauditable in many cases. Tool execution and its mediation are a distinct mechanism — a discrete action executed by the harness which you can (often) name and check. We can munge all of this under “the AI escaped”, but it doesn’t help us solving the problem.
So, what broke? (or “escaped” if you want to attract a crowd)
Here is my take — given the available information — on what actually happened, in a sense, “structurally”. Let’s take it part for part, in reversed order and see how the path ended up blasting HF’s servers.
First, the sandbox was doing its containment by allowing the agent exactly one route to the outside world using a package-registry proxy — an internal service agents use to fetch software packages, and a single permitted path out of the isolated environment. Sounds great, but this one turned out buggy as far as I can tell.
Second, the harness seems to have done nothing more than faithfully execute the plan received from the model (as it had passed through the guardrails). OpenAI makes no explicit mention of any harness-level checks, and the harness thus most likely did exactly as it was told.
Third, the guardrails, which were supposed to check “malicious” actions proposed by the model, were explicitly turned off. Since OpenAI was testing performance on a security benchmark — and apparently they wanted to test the model’s performance, not the “model + guardrail” performance — there was no check on the actions proposed by the model. Please note however, that even if such checks were in place, it is unclear whether or not these would have stopped the actions proposed by the model from reaching the harness (I have not found explicit reporting of the counterfactual anywhere).
Fourth, the model itself. Interestingly, this was a new highly capable model — “GPT‑5.6 Sol and an even more capable pre-release model” according to the published material. While similar setups seemed to have worked previously with less capable models — which likely simply gave up at some point and emitted a STOP — this model kept suggesting new actions to try and finish the task (which, just to remind you, was likely fed to the model, by the harness, in every turn anew).
The bottom line of what went wrong is this: no harness checks, no guardrails by design, so layers two and three were out in terms of securing the agent. The model was new and “more capable” and hence, with the given task of doing well on the benchmark, kept emitting new action plans. This interestingly underscores how models that are more “aligned”, but also more capable, are still likely to lead to undesirable paths; they will simply try more and more creative ones. Next, the sandbox had a tricky — but non-agent / AI — bug; a bug that was breachable and allowed the “model”, with an intricate plan, to use the proxy to reach other parts of the internet and hence “escape”. Nobody was checking the actual steps the model was taking, nothing was verifying the path; this is what led to the incident.
Now, does this mean “the model escaped”? I would say no, it means the agent, in its totality, ended up following a path that was annoying (to say the least) for HF. After the proxy breach, the agent now essentially had open internet access and started “bashing” HF’s servers to see if it could find the answers to the security benchmark it was set to complete. The bashing was severe (in terms of speed and creativeness — this was a very capable model) and hard to prevent.
So how do you keep an agent on a leash?
The two obvious fixes are the two we just watched fail. Better alignment is worth having, but a more capable model travels further on the bad proposals it still makes, and the switch is not yours to hold anyway — OpenAI turned it down here on purpose. However, even turned up this gives you a probability, not a guarantee. Necessary, not sufficient. A sandbox is worth having too, but it bounds where an agent can go, not the order in which it goes there, and its walls are ordinary software with ordinary bugs (some of which came clearly into view in this incident). Again: necessary, not sufficient.
What was missing is simple: nobody was looking at the sequence of executed steps. The model proposed an action, the harness ran it, the model proposed the next one, the harness ran that — and nothing in between ever asked “given everything this agent has done so far, should it be allowed to do this next thing?” To me, that exact question is often the missing layer. Let’s turn to the details of such a defense layer and where it would “live”.
The check belongs on the harness, over actions, along the path
The defense simply has to “live” on the harness. The harness is the one component that sees every action, in order, and sees it just before it happens. That is the only place where you can still say “no” and have it mean something. A model refusal comes too early — it is a guess about a proposal, not a decision about an action. The harness’s next-action step is the one moment where a check is both informed and still in time.
And, it has to be over the action itself, not the model’s output text. The guardrail-instinct we have been cultivating the last year is to read what the model said and judge that, but that’s not guaranteed to be what actually runs. What you actually want to check is the concrete thing the harness is about to do: not “does this sentence look malicious” but “this is a network call to huggingface.co — is that allowed, here, now?” To ask that, you have to be able to name the action precisely, which means the harness has to hand the check a typed action rather than a blob of text. Note that this is what we are building the Agent Action Grammar (AAG) for: a shared, vendor-neutral way to describe each action so that a rule can be written about it at all. More on this in a later post, but for now you can get a gist by looking at our atomic behaviours. However, for our purpose today, it suffices to say that without such a vocabulary, the finest rule you can write over a raw shell tool is “bash: yes” or “bash: no”, which is barely a rule. We need a bit more.
And, the defense has to be over the path, not the isolated step. Plenty went wrong here at once — guardrails off, no harness check, a capable model that kept going, a buggy proxy — but the part a check could have caught lived in the order of the actions, not in any one of them: reaching the internet was ordinary, poking at someone else’s servers was (in a way) ordinary, and it was the sequence of the two that turned into an incident; into an attack. So the check cannot be stateless; it has to look at the sequence so far, P = (s₁, …, sₙ₋₁), when it decides on sₙ. Note that this is exactly the thing none of the earlier layers can express: the model is stateless, the guardrails check in individual I/O.
Put all of this together and you get something specific and fairly unglamorous (not nearly as cool as “AI escaping”): you simply end up with a check that sits at the harness’s next-action step, receives a typed action, evaluates it against the path so far and a set of rules, and returns whether or not the agent is allowed to proceed. It does so at the point where it can effectively intervene; the harness (or agent loop if we want to be trendy). We called the result an Agent Security Kernel (ASK), because as you can tell throughout this whole article, we thought that the one thing that was lacking in agent-security world was more jargon :S…
What the ASK does and doesn’t do
An ASK gives you a layer none of the earlier layers in the stack provide: a deterministic, auditable “no” that can be conditioned on the whole history of a task as specified by the actual actions that have been carried out. On the HF incident, take a super simple destination check (yes, stateless is a special case of stateful): A single rule implemented in the ASK — deny actions to destinations the agent never needs — stops the huggingface.co call flat. That action is simply never carried out by the harness. Note that such a rule within an ASK does not fix the proxy bug; patching does that. It stops the path from becoming an incident by excluding a whole set of actions and hence narrowing the set of legitimate actions. More realistically, any agent would have simple limits on how often it should be allowed to reach certain external endpoints, how often — if at all — it can take data from inside the security boundary to outside, whether or not it is allowed to combine data from multiple sources, whether or not specific gates (yes, guardrails… or humans..) need to be in the path, etc., etc. That’s what an ASK allows you to do.
Please note that the whole ASK idea rests on one simple assumption: the harness has to be intact. The whole thing works because the model can only act through the harness — its only output is text, and the harness is what turns text into actions. Remove that — let the model rewrite the harness, or break out of the process it runs in — and the check is not defeated so much as bypassed: there is simply no longer an interface for it to sit on. So this is a layer on top of isolation, not a replacement for it; you keep the sandbox underneath precisely to keep the harness honest. (It is also why, for an agent whose sanctioned job is to obtain code execution — a security benchmark being the purest case — you have to make sure the code it runs itself lands somewhere mediated. I am not going to pretend this problem disappears; the aim is to reduce the risk, not to abolish it.)
And yes, obviously, you can write incomplete rules. Miss a rule and a bad path, proposed by a capable model, walks straight through the gap. But that is an additive problem: you simply update your rules (we call them policies). You write the rule, the gap closes, and every agent behind that harness gets the fix at once. There is — as far as I know — no equivalent move for a model. No change you can make to the weights to guarantee it never proposes X. Using an ASK, you can start with a handful of path rules and grow them; you cannot start with a “handful of alignment” and grow it the same way.
Bottom line
So, what was doing the escaping? On a useful reading, I think nothing was. The model emitted text; the harness turned it into actions; the sandbox had a bug; and no one, at any point, looked at the path those actions traced. Calling that “an AI that escaped” is not really wrong — but it mostly points at the model, the one part of the system you likely neither own nor fully control. The “AI escaped” terminology points away from the one place you can actually do something: the harness, where every action passes in the open, one at a time, just before it happens.
That is where the defense should go, and I think this would have easily created a proper benchmark run (i.e., with guardrails turned off), that still would not have had “the model escape”. Now, none of this is magic — you can miss a rule, and the effectiveness of an ASK leans on the harness holding — but it is a real, deterministic layer you can add to your agent stack today.
One last thing (yeah, that made the bottom line not really the bottom line..)
The easy reading of this incident is “OpenAI turned its guardrails off, so as long as OpenAI turns its guardrails back on, we are all fine”. I am sure many of us simply had a good laugh about the rogue agent and we went on with our lives.
I want to push back on that, for two reasons:
First, if you are the one building agents, those guardrails are likely not yours. They live with the model provider, they are tuned by the provider, and they can be turned down — as they were here — for reasons that have nothing to do with you. Betting your agent’s safety on a switch inside someone else’s system seems downright silly.
Second, and more importantly, even with every provider guardrail on, this will not go away. A guardrail is an I/O check on a single message; it does not see the path, and (as I argued above) a sufficiently capable model will keep proposing until something gets through. The lesson of the incident is not “one layer was off.” It is that we need checks at every layer — the model’s own refusals, the guardrails, the harness, and the sandbox — because each catches a different thing and only harness-level action checks will reliably catch the full sequence of events.
So by all means, let the providers make their models and guardrails as good as they can — I hope they do. And, let’s all laugh at (and/or be scared by) agents going rogue. However, let’s not let “they will fix it” become the plan if you are responsible for an agent.