TL;DR

Every team building AI agents hits the same wall: the prototype works beautifully on a Mac mini or a local Python loop, and then productionization becomes a nightmare. The hard part isn’t harness engineering or prompt tuning — it’s infrastructure. Understanding that distinction changes how you architect from day one.

Introduction

I’ve watched this pattern repeat itself across a dozen consulting engagements in the last 18 months. A team spins up an agent in a weekend — Claude running in a loop, a Python file, maybe a couple of tools wired up. It does the thing. Everyone’s excited. Then someone says: “Let’s put this in front of customers.”

That’s when the real work begins.

In a recent conversation between Anthropic’s platform leads Angela and Caitlyn and AI+I’s Dan Shipper, they named something I’ve seen firsthand but hadn’t heard articulated so cleanly: people think harness engineering is the hard part, but the wall they actually hit is infrastructure.

This post is about that wall — what it’s made of, why everyone hits it, and what the platform landscape is doing about it.

What the Demo Conceals

The agent demo is seductive precisely because it hides everything. You have:

  • A persistent local environment (your laptop or a Mac mini)
  • An always-on file system
  • A browser with your session intact
  • A Python runtime that doesn’t evict your process

None of those things are free in production.

When you move your agent to a real cloud environment — especially for long-running, async, multi-turn interactions — you immediately start asking questions you didn’t need to ask on day one:

  • What happens when the sandbox dies mid-task?
  • Where does the transcript live between sessions?
  • How do I spin sandboxes up and down without paying for idle compute?
  • How do I handle secure credential injection into an ephemeral environment?
  • How do I recover from partial failure without rerunning the whole task?

These aren’t ML problems. They’re ops problems. And most teams building agents aren’t staffed for them.

The Harness Trap

Here’s why teams get caught off guard: early in the agent-building process, the visible complexity is harness engineering. You’re thinking about:

  • How to structure tool calling
  • How to maximize context window efficiency
  • How to do prompt caching right
  • Whether your memory architecture is the right shape

These are real concerns. And the ecosystem has delivered real solutions — the Anthropic SDK, agent frameworks, pre-built memory primitives. So teams invest there, get comfortable, and declare themselves “ready to ship.”

Then they try to ship.

A sandbox loses connection and the whole agent dies. The server they stood up at 2 AM is still running and burning money at 3 PM. There’s no retry logic because who writes retry logic for a demo? The transcript is stored in memory and disappears on restart. Scaling from one concurrent session to a hundred is a rewrite, not a config change.

The harness was never the wall. The infrastructure was always the wall.

Lessons from a Production Voice AI Platform

I ran into this exact sequence building out a voice triage and call management platform — multiple microservices, real-time audio handling, async session management. Local dev on k3d was smooth. ArgoCD GitOps wired up to EKS worked well in isolation. The moment we started pushing async, long-running voice sessions through the stack, every assumption we’d made about statefulness, connection persistence, and sandbox lifetime got stress-tested.

We had solid foundations — Helm umbrella charts, Terraform/Terragrunt for IaC, GitHub Actions with OIDC auth. But the agent loop itself had to be rebuilt to handle: dropped telephony connections, failed sub-agent spawns that needed cleanup, secrets injection into ephemeral containers at runtime, and multi-cluster ArgoCD state that had to stay consistent across three AWS accounts.

None of that is in any agent tutorial. All of it is in every production agent deployment.

What Good Infrastructure for Agents Actually Looks Like

Based on what I’ve seen work across engagements — and what the Anthropic platform team is building toward — here’s what production-grade agent infrastructure needs to cover:

Sandbox lifecycle management. Agents need ephemeral compute that spins up fast, persists the right state, and terminates cleanly. If your agent dies mid-task, you need a recovery path — not a rerun.

Transcript and state persistence. The conversation history, tool outputs, and intermediate results need to live somewhere durable. In-memory is fine for demos. It’s a ticking clock in production.

Credential and secret injection. Agents need to call external services. That means OAuth tokens, API keys, and credentials that must be injected securely at runtime — not hardcoded, not passed through environment variables in cleartext.

Async-first design. Long-running agents can’t block a synchronous request. If your architecture assumes a response in under 30 seconds, you’ll redesign it the first time you run a deep research task.

Human-in-the-loop hooks. Not every step should be fully automated. You need clean pause points, approval flows, and audit trails — especially in regulated industries.

Observability at the agent level. Standard metrics (CPU, memory, latency) are necessary but not sufficient. You need to know: which tool calls failed, which sub-agent stalled, what the model decided and why.

The Model Lock-In Question

One concern I hear constantly: “If I build on Claude-managed infrastructure, am I locked in? What if I need to swap to GPT-5 or Gemini?”

It’s a valid concern. But the framing is shifting.

The old mental model was: generic harness + hot-swappable models. That made sense when models were roughly interchangeable and the differences were marginal. Today, the gap between model-specific behaviors — how each model uses file systems, how it handles tool call patterns, how it responds to different harness architectures — is wide enough that the generic harness approach leaves significant performance on the table.

The emerging model is: harness + model as a paired unit. You pick your primitives carefully, you hill-climb the harness for your chosen model, and you accept that the agent you build is somewhat specific to the model it runs on. The tradeoff is worth it — the performance delta between a well-tuned harness and a generic one can be dramatic.

If you need model diversity, build it at the agent level (different agents for different tasks, each optimized for its model), not by making every agent model-agnostic.

The Platform Direction

What Anthropic’s platform team described — and what I think is the right trajectory — is a future where the parameters you care about compress to two: outcome and budget. Claude figures out the model selection, the sub-agent topology, the harness architecture. You specify what you want done and what you’re willing to spend.

We’re not there yet. But we’re closer than most people realize. The current managed agents infrastructure is already handling sandbox lifecycle, transcript persistence, and credential vaulting — the pieces that eat the most engineering time in DIY implementations.

A year from now, the expectation is that the platform scales to support agents that constantly recreate themselves, run long-duration tasks, and operate across multi-agent topologies — without the infrastructure ever being the bottleneck.

For teams building today: don’t prototype your way into a production rewrite. Design for the wall before you hit it.

Conclusion

The infrastructure wall is real. It’s the same wall across every team, every company, every agent use case. The good news is it’s a solved problem — the solutions just haven’t been packaged cleanly until recently.

If you’re building agents: spend a day mapping your production requirements before you spend a week on your harness. Ask yourself whether you’re designing for the demo or for scale. The answer changes your architecture from day one.

And if you’ve already hit the wall — welcome to the club. Most of us have been there.

Resources