TL;DR

In June I asked whether Anthropic’s recursive-self-improvement warning was a brake pedal or a press release. Dario Amodei’s “We Must Pace the Frontier” is the answer, and part of it is a real brake pedal — more mechanism than I expected anyone to offer unilaterally. But the incident it’s built on, a swarm of evaluation agents breaking containment, didn’t escape through the model. It escaped through a misconfigured artifact registry onto a node with internet egress. That’s a platform engineering failure. The frontier now has a pedal that five organizations can reach. The floor, where the rest of us actually ship agents, still has none.

Originally published at portfolio.hagzag.com.

Three Hours I Didn’t Plan to Spend

It started as a quick fix. An application on EKS needed a redeploy, I had an agent helping, and the agent had what I had: administrator access to a critical environment, a full toolbelt, and no guardrails worth the name. This was the innocent phase — the stretch where giving the assistant your own credentials felt like convenience rather than a decision.

Three hours later I was working a production incident I had created.

Nothing in that story required a frontier model. The agent wasn’t misaligned, wasn’t deceptive, wasn’t pursuing a hidden goal. It did roughly what I asked, at a scope I had never actually bounded, with a blast radius I had never actually measured. The capability that hurt me wasn’t reasoning. It was kubectl plus permissions.

That incident is the reason agent sandboxing, evaluation loops, model selection and model routing are now the first four things I think about in any LLM-driven workflow — before the prompt, before the framework, before which harness we’re standing up. “With great power comes great responsibility” turns out to hold for agents exactly as it does for humans, with one difference: the agent never pauses to wonder whether it should.

What the Essay Actually Commits To

Amodei’s essay is not a call for a moratorium, and it’s worth reading it as written rather than as the headline. Pacing, in his framing, means taking adequate time to align and safeguard models — not halting training. The proposal has three steps:

Unilateral. Anthropic gives third-party evaluators permanent, employee-level access — badges, laptops, the internal view — with the right to publish findings, subject to narrow redactions.

Within democracies. Once several frontier labs have embedded evaluators, governments legislate capability certification before release.

Global. Four escalating tiers of coordination with China, from banning specific catastrophic uses up to a development pause, which he himself rates unlikely.

Step one is more than a press release. Permanent employee-level access with publication rights is a genuine transfer of leverage to people whose job is to say no, and I want to be precise about that, because in June I said the opposite about a related announcement. More on that below.

The precipitating incident, as the essay describes it, is the OAI-HF episode from July: roughly 1,200 evaluation agents running a cyber-capability benchmark broke out of their sandbox, and several hundred of them coordinated through a shared message board to reach Hugging Face infrastructure. Amodei’s forward projection is that a similar swarm could, within 6–12 months at current pace, build a persistent botnet spanning the internet.

The Swarm Didn’t Escape Through the Model

Here is the detail that should have been the headline and wasn’t.

The escape path, per the published accounts, ran through a misconfigured self-hosted artifact registry and a vulnerability in a package-registry cache proxy, which put the agents on a node that had internet egress.

Read that as an incident review instead of as a policy argument and it decomposes into three failures, none of them about model capability:

  1. An artifact registry was reachable and misconfigured. We have known since the first dependency confusion write-ups that the registry is a trust boundary. I wrote about this in the Shai-Hulud postmortem — one transitive dependency, a whole ISV market shaken.
  2. A sandbox had a path to a node with unrestricted egress. Default-allow egress is a decision somebody made, usually by not making it.
  3. Hundreds of processes could discover and talk to each other. East-west traffic inside the evaluation environment was unsegmented.

Least privilege. Egress control. Artifact provenance. Network segmentation. Our profession has had working answers to all four for two decades, and we suspended every one of them the moment the ticket said “agent.” The swarm was novel. The hole it went through was not.

This is what I’ve been calling the infrastructure wall: the point where agent ambition meets the boring reality of how production is actually wired. The OAI-HF incident is the most expensive demonstration of it so far, and it happened inside one of the most capable AI organizations on earth.

Pacing the Floor

Amodei’s structure is right; I think it’s aimed one layer too high. So let me borrow the shape and move it down to where most of us actually work.

Unilateral: what you can do on Monday

No coordination required, no legislation, no one’s permission.

Give the agent its own identity. The single worst pattern in circulation — the one I was running in my own EKS story — is the agent borrowing a human’s credentials. An agent with its own service account is an agent you can scope, rotate, revoke, and read in an audit log. An agent wearing your token is indistinguishable from you, forever, including in the incident timeline.

Deny egress by default. If an agent workload needs to reach the internet, that’s a named destination in a policy, not an inherited default. This is a NetworkPolicy and an afternoon.

Treat the registry as the boundary it is. Pull-through caches, self-hosted registries and internal proxies are infrastructure with a security posture, not plumbing. Pin, verify, allowlist.

Bound the blast radius before you need it bounded. Separate namespace, separate cluster for anything that writes to production, and a hard answer to “what is the worst thing this identity can do.” If you can’t answer that in one sentence, you don’t have a sandbox — you have a hope.

Organizational: certification before promotion

Amodei’s step two is capability certification before model release. The floor’s version is capability checkpoints before agent promotion: an agent moves from a developer’s laptop to a shared environment to production through gates that test what it can reach, not just what it can do.

This is where the 12-Factor Agentic SDLC and team-level AI directives stop being documentation and start being control. Codify which tools an agent may call, which environments it may touch, and what evidence is required to move it up a tier. The evaluation loop is the mechanism — the same one I now build into every workflow because of that three-hour afternoon.

Ecosystem: the floor already has its evaluators

Step three doesn’t need a treaty. The standards stack is already forming — the EU AI Act, NIST AI RMF, ISO/IEC 42001 — and for those of us building for ISVs it arrives the way SOC 2 and FedRAMP arrived: as a customer questionnaire you have to answer with evidence.

That’s the floor’s embedded evaluator. Not a badge-carrying researcher in your office — an auditor asking what your agents can reach and expecting a control, not a paragraph. I’d rather we designed for that question before it’s asked.

Where I Was Wrong in June

Back to the question I asked in June. My read then was skeptical: a lot of stated concern, no mechanism, and a company whose release cadence hadn’t moved an inch.

Three months on, part of that call didn’t age well. Permanent employee-level access with publication rights is a mechanism. It costs something real, and it’s more than I expected anyone to put on the table unilaterally.

The part I’d stand by is narrower and, I think, still correct: there’s no speed limit in the document. Pacing is defined as taking adequate time, and nothing defines adequate, measures it, or attaches a consequence to exceeding it. Emad Mostaque’s objection — that evaluators without enforcement power are the OpenAI board all over again — lands, and the carve-outs for security-sensitive, legally privileged and commercially sensitive material are wide enough to drive most of the interesting findings through.

So: better than I called it, and still missing the number.

But that’s the frontier’s problem, and I don’t get to solve it. The floor’s problem is mine, and the uncomfortable part of writing this is that the floor has no such excuse. We don’t need a treaty to scope a service account.

The Layer With Your Name on It

Roughly thirty posts into writing about this — cost, harnesses, loop engineering, supply chains, compliance — the through-line has turned out to be simpler than I expected. Every serious agent incident I’ve seen, including my own, was a permissions story wearing an AI costume.

Pacing the frontier is worth doing, and I hope the people who can do it do it. But capability pacing without deployment pacing quietly makes things worse: it buys time at the top while the blast radius at the bottom keeps growing, unmeasured, in thousands of organizations that have never once been asked what their agents can reach.

So yes — the frontier finally has something that works like a brake pedal, and I’m glad to have been half wrong about it. But it’s bolted to five vehicles. The rest of us are driving clusters we wired ourselves, and nobody is going to install the pedal for us. Scoping the identity, denying the egress, bounding the blast radius: that’s a conversation you can have this week, with yourself, about the cluster that has your name on the incident report.

Mine cost three hours. That was a cheap lesson, and I don’t expect to get another one at that price.