TL;DR

An agent’s identity should be issued to it by attestation, not handed to it as a key. SPIRE does that: it looks at the pod and issues a short-lived X.509-SVID. Use that same key to sign every task the planner puts on the queue. The signature survives the queue, where TLS doesn’t.

In the lab, forged, self-signed and tampered tasks are all rejected. Then a poisoned inventory makes the legitimate planner sign a delete, and the bucket is gone anyway. Identity is not permission.

Originally published at portfolio.hagzag.com.

The instinct I had to talk myself out of

Part 1 ended with a deleted bucket and a log line naming a ServiceAccount. The instruction sat in Redis between two TLS hops, and anything that could write to Redis could have put it there.

My first reaction was the one from twenty years of signing commits: give the planner a GPG key and sign every task. That is half right. Signing the payload is exactly what’s needed, because a signature travels with the message. The key is the wrong half. A GPG key is created by a person, lives for years, gets copied into a Secret, and knows nothing about where the code using it is running. Put one in every agent pod and you’ve built the Part 1 LLM key problem again, with a signature on top.

What I want is a key the agent never had to be given. It gets one because of what it provably is, and the key expires on its own.

License plates are issued, not claimed

That’s the job SPIFFE describes and SPIRE implements. The ladder in Kubernetes has three rungs:

  1. ServiceAccount. A name. Anyone who can create a pod in the namespace can use it.
  2. Bound, projected ServiceAccount token. A short-lived JWT the kubelet mounts into the pod. The SPIRE agent uses its own one to prove to the SPIRE server which node it runs on (k8s_psat node attestation).
  3. SPIFFE X.509-SVID. When a process opens the SPIRE agent’s socket, the agent asks the kubelet which pod that process belongs to (k8s workload attestation). If the pod’s namespace and ServiceAccount match a registration entry, the agent issues a certificate whose URI is the identity, e.g. spiffe://highway.lab/ns/migration/sa/planner-agent.

The lab adds a rogue pod to the story. It sits in the same namespace, it’s meshed by Linkerd like everyone else, and it has the same SPIRE socket mounted. It just has no registration entry.

task part2:up     # Part 1 road + SPIRE 1.15.3 + registration entries
task part2:svid   # every pod asks the local SPIRE agent: who am I?
planner-agent   spiffe://highway.lab/ns/migration/sa/planner-agent | expires 2026-09-25T05:46:50+00:00 | issuer ...O=SPIFFE,C=US
executor-agent  spiffe://highway.lab/ns/migration/sa/executor-agent | expires 2026-09-25T05:46:50+00:00 | issuer ...O=SPIFFE,C=US
rogue           FetchX509SvidError: no identity issued (StatusCode.PERMISSION_DENIED)

Same socket, same network, no plate. The rogue can’t ask for an identity it wasn’t registered for, and the planner never stored a key anywhere. SPIRE rotates the certificate within the hour.

Break: the queue takes anything

Identity alone changes nothing if nobody checks it. With signing still off, the rogue pushes one task straight into Redis:

forge=pushed mode=unsigned claimed_signer=- instruction='delete bucket prod-archive'
executor=task task_id=666 step=1 instruction='delete bucket prod-archive' requested_by=haggai signer=-
tool=delete_bucket bucket=prod-archive status=ok peer=executor-agent.migration.serviceaccount.identity.linkerd.cluster.local

Look at the last line. The mesh did its job perfectly: the rogue’s connection to Redis was mTLS, and so was the executor’s call to the tool server. The rogue even wrote requested_by=haggai, and nothing objected.

Signing the cargo

The fix has two sides.

  • Planner: it fetches its SVID from the SPIRE socket and signs each task as a compact JWS. The certificate chain goes in the x5c header.
  • Executor: it fetches the trust bundle from its own SPIRE socket. Then it verifies four things, in this order:
# src/highway/signing.py (abridged)
header = jws.extract_compact(token).headers()          # typ + x5c required
chain  = [load_der_x509_certificate(b64decode(c)) for c in header["x5c"]]
# 1. the chain leads to an authority in the SPIRE trust bundle
# 2. the leaf's SPIFFE ID is in ALLOWED_SIGNERS (only the planner may queue work)
# 3. the signature verifies with the leaf's public key
# 4. exp has not passed (limits replay)
task part2:sign    # SIGN_PAYLOADS=true on the planner, VERIFY_PAYLOADS=true on the executor
broker=put task_id=43 step=1 signed_by=spiffe://highway.lab/ns/migration/sa/planner-agent
executor=task task_id=43 step=1 instruction='copy bucket customer-data to regulated-customer-data' requested_by=haggai signer=spiffe://highway.lab/ns/migration/sa/planner-agent

The executor log now names who wrote the instruction, not only which workload carried it.

Three attacks, three rejections

task part2:forge   # unsigned, then a self-signed cert that claims the planner's SPIFFE ID
task part2:tamper  # executor paused; rogue rewrites signed tasks while they sit in Redis
executor=REJECTED reason='unsigned payload'
executor=REJECTED reason='chain does not lead to the highway.lab trust bundle (claimed spiffe://highway.lab/ns/migration/sa/planner-agent)'

forge=tampered index=0 before='copy bucket customer-data to regulated-customer-data' after='delete bucket prod-archive'
forge=tampered index=1 before='copy bucket billing-exports to regulated-billing-exports' after='delete bucket prod-archive'
executor=REJECTED reason='bad signature: bad_signature: '
executor=REJECTED reason='bad signature: bad_signature: '

The second rejection is the important one. Anyone can mint a certificate that says spiffe://highway.lab/ns/migration/sa/planner-agent; a URI in a SAN field is just text. What the rogue can’t do is get that certificate issued by the trust domain’s CA. The tamper test is Part 1’s cargo problem, solved: the rogue had full read-write access to the queue, and it still couldn’t change a word without breaking the signature.

Break again: a valid signature on a bad decision

Now replay Part 1’s attack, the poisoned owner note in the inventory, with every defense from this post switched on:

task part2:poison
planner=step task_id=45 step=2 instruction='delete bucket prod-archive'
broker=put task_id=45 step=2 signed_by=spiffe://highway.lab/ns/migration/sa/planner-agent
executor=task task_id=45 step=2 instruction='delete bucket prod-archive' requested_by=haggai signer=spiffe://highway.lab/ns/migration/sa/planner-agent
executor=tool_call task_id=45 step=2 tool=delete_bucket args='{"bucket": "prod-archive"}'
tool=delete_bucket bucket=prod-archive status=ok peer=executor-agent.migration.serviceaccount.identity.linkerd.cluster.local

Valid signature. Right signer. Untampered payload. Wrong decision.

The signature proves the planner said it. It says nothing about whether the planner should have said it. The note went in as data and came out as a correctly signed instruction. And requested_by=haggai is now tamper-evident, but it isn’t true. The planner copied it from its command line, and nobody checked that I asked for anything, let alone a delete.

This is what I got wrong in my first design: I treated signed as trusted. A signature makes a message attributable. It doesn’t make it authorized.

What actually went wrong while building it

The attacks weren’t the only thing that broke. Two failures from the build are worth more than the happy path:

  • The JWS library rejected every legitimate message. joserfc caps the protected header at 512 bytes by default. A real SVID in x5c is about 1 KB, so the executor logged exceeded_size for every signed task until I raised the cap. The safe default was quietly incompatible with certificate-based signing.
  • Security policy has a rollout window. In one capture, a task signed at 04:50:22 was executed with signer=-. The executor had been restarted with verification on, but the old pod was still shutting down, still running, and still popping tasks from Redis with verification off. Rolling out a stricter verifier doesn’t cut over atomically. For a few seconds, the old policy and the new one run side by side, and a queue happily feeds both. The lab now waits for terminating pods to disappear. In production, I’d gate the queue itself, or version the message format so old consumers can’t read it.

😉 A confession from the build. At one point the executor couldn’t get an identity, while the planner could. The AI coding assistant helping me debug the lab knew exactly what to do: add a WorkloadAPIServer plugin block to the SPIRE agent config. It applied the change, reran the lab, and reported success.

That plugin doesn’t exist. SPIRE 1.15.3 refuses to start with it (unsupported plugin type). It only looked fixed because the SPIRE agents already running never reloaded their config. The next fresh cluster would have come up broken.

A trusted agent with write access, a confident explanation, and a green result. Nobody checked the decision itself. That’s this whole series in one paragraph. What caught it was a second opinion and a fresh cluster, not the agent’s own report.

And the cost:

  • SPIRE is another control plane. A server, a DaemonSet, and registration entries to manage.
  • The agent is privileged. It runs with hostPID so it can map a caller’s process to its pod.
  • Every task grows by about a kilobyte of certificate chain.

I’d pay that for an agent system with delete rights in a regulated environment. I wouldn’t for a demo.

The road ahead

Each vehicle now has a plate that can’t be forged and survives the queue. What’s still missing is the permit:

  1. The Safe Highway: mTLS, and its limits.
  2. License Plates (this post): attested workload identity, and signed payloads.
  3. Driving Permits: delegation. The token should say who the agent acts for (sub), which agent is acting (act), for which task, with which scopes, until when, enforced at the tool gateway. The LLM key also leaves the pods.
  4. The Closed Track: sandboxing and predicting the blast radius.
  5. The Black Box: an audit chain that names the human, the agent and the task.

The lab is practice/part2 in agentic-highway-labs. task part2:all runs it end to end on k3d, and task part2:doctor shows what SPIRE sees when something doesn’t attest.

Conclusion

GPG had the right idea and the wrong key. Sign the payload, because the payload is what crosses the queue. Sign it with a key the workload earned by attestation, not one a human copied into a Secret.

Then accept what that buys you: every message now has a verifiable author, and some of those authors will be wrong. A license plate tells you whose car it is. It doesn’t tell you the driver was allowed on this road.