TL;DR

A signed image that nobody verifies is an unfinished sentence. This post walks through wiring the Wolfi-based hagzag/tools image from Part 3 into a real Terraform modules pipeline, verifying signatures at both CI time and at deploy time with Sigstore’s policy-controller, and closing the gap with digest-pinning via Renovate. This is Part 4, the finale of the “Rebuilding for Compliance” series.

Where We Are in the Series

Part 1 established the vocabulary — SBOM, provenance, SLSA, cosign — and the regulatory pressure behind FIPS and FedRAMP. Part 2 made the case for Wolfi OS and the daily-rebuild model. Part 3 walked through actually building and signing the hagzag/tools toolchain image. This post is where the work stops being about the image and starts being about the pipeline around it.

The Two Halves of the Verification Loop

Supply chain security has two halves, and most teams only build one of them:

  • Producer side: build reproducibly, generate SBOMs, sign artifacts. This is what Part 3 covered.
  • Consumer side: verify those signatures and attestations before using the artifact — in CI, in admission control, and ideally both.

If the consumer side is missing, the producer side is theatre. An unsigned image and a signed-but-never-verified image have the same security posture. The goal of this post is to close the loop.

The tf-modules Migration — Before and After

The hagzag/tf-modules repo is the perfect stress-test consumer. It runs terraform fmt, renders docs, and fires semantic-release — all of which need a consistent, auditable toolchain.

Here is what the old pipeline looked like, before the toolchain image existed. Every job pulled and installed its own tools from the internet on every run:

# BEFORE — host-based, drift-prone
jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.14.8
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Install semantic-release + plugins
        run: |
          npm install -g \
            semantic-release \
            @semantic-release/commit-analyzer \
            @semantic-release/release-notes-generator \
            @semantic-release/changelog \
            @semantic-release/git \
            @semantic-release/exec \
            @semantic-release/github
      - name: Run semantic-release
        run: semantic-release

Every problem from Part 1 is in that snippet. There is no SBOM, no signature, no provenance. Tool versions drift silently between runs. The npm install -g step is a public-internet dependency on every job execution. If registry.npmjs.org hiccups, the release pipeline fails for reasons that have nothing to do with the code being released.

Here is the same job after migrating to the Wolfi-based toolchain image. No installation step, no version drift, and the image itself is signed:

# AFTER — pinned, signed, reproducible
jobs:
  release:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/hagzag/tools:latest
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run semantic-release
        env:
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN || secrets.GITHUB_TOKEN }}
        run: |
          git config --global --add safe.directory "$GITHUB_WORKSPACE"
          semantic-release -r ${{ github.server_url }}/${{ github.repository }}.git

The diff is smaller than you might expect and that is exactly the point. The complexity moved out of the consumer workflow and into an auditable, signed artifact that every consumer can share.

Container-Mode Gotchas Worth Knowing

Moving consumer workflows into a container: job sounds trivial, and usually is, but there are three traps that tripped us up and are worth calling out:

The first is private registry auth. GHCR treats pulls from container jobs the same as any other pull — you need credentials on the job, not just on a login step elsewhere in the workflow:

container:
  image: ghcr.io/hagzag/tools:latest
  credentials:
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

The second is the safe.directory git warning. When the container runs as a different UID than the checkout, git will refuse to operate on the workspace with a “dubious ownership” error. The fix is one line but it has to go inside the container job, not outside:

- run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

The third is smoke-gating. You want a cheap preflight that proves the image actually has the tools it claims to have, but you do not want to run it on every push — that is what the image’s own CI does. We gate it behind workflow_dispatch so it runs on demand when bumping the image tag, and let downstream jobs key on (success || skipped).

Verification in the Consumer Workflow

So far the consumer trusts the image because it is on a private registry and pulled with a PAT. That is not a trust anchor — it is a perimeter. The real trust anchor is the cosign signature from Part 3, and we can verify it before running anything:

- name: Install cosign
  uses: sigstore/cosign-installer@v3

- name: Verify tools image signature
  env:
    IMAGE: ghcr.io/hagzag/tools@sha256:<pinned-digest>
  run: |
    cosign verify "$IMAGE" \
      --certificate-identity-regexp '^https://github.com/hagzag/tools/' \
      --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
      | jq '.[0].optional.Bundle.Payload.logIndex'

Two things to notice. The --certificate-identity-regexp pin means the signature is only valid if it was produced by a workflow in the hagzag/tools repo — someone who steals a GHCR push token still cannot forge a valid signature from the wrong identity. The jq at the end extracts the Rekor log index, which is the bit that makes this falsifiable: the signature is recorded in a public transparency log, and anyone can audit it after the fact.

Beyond CI — Admission Control at Deploy Time

CI verification is good but it is not sufficient. CI is the path where we build. Production is the path where everything runs — including images someone on another team pulled, or an operator installed from a Helm chart, or a vendor shipped with their product.

The place to enforce “only signed images from known identities run here” is the cluster admission controller. Sigstore’s policy-controller is purpose-built for this. A ClusterImagePolicy that gates the whole cluster on signed images from our hagzag/tools identity looks like this:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-hagzag-signature
spec:
  images:
    - glob: "ghcr.io/hagzag/**"
  authorities:
    - name: github-actions-keyless
      keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: "^https://github.com/hagzag/.+"
      ctlog:
        url: https://rekor.sigstore.dev
  mode: enforce

With this policy applied and the policy-controller webhook installed, an unsigned ghcr.io/hagzag/* image — or a signed one from the wrong identity — is rejected at admission. Not “flagged later.” Not “caught by a weekly scan.” Rejected, as in never admitted to the cluster. The webhook consults Fulcio for the certificate chain and Rekor for the transparency log entry on every admission request.

If policy-controller feels like too new a thing to adopt, Kyverno supports a very similar pattern with a ClusterPolicy that calls verifyImages. Pick whichever fits your existing policy story.

Digest-Pinning and Renovate

Tag-based pins like ghcr.io/hagzag/tools:latest are convenient and also terrible for supply chain integrity. “Latest” is mutable. A tag can be moved. A digest cannot — sha256:abc... is content-addressable and tamper-evident.

The mature move is to pin every consumer workflow to an immutable digest, and let Renovate handle the maintenance burden of keeping that digest current:

{
  "extends": ["config:base"],
  "packageRules": [
    {
      "matchDatasources": ["docker"],
      "matchPackageNames": ["ghcr.io/hagzag/tools"],
      "pinDigests": true,
      "groupName": "hagzag tools image",
      "schedule": ["before 6am on monday"]
    }
  ]
}

Renovate’s github-actions manager understands container: image@sha256:... syntax in workflow files and opens a PR whenever a new digest is available. Reviewing those PRs is the moment where a human confirms — ideally with a fresh cosign verify run against the new digest — that the tag has not been hijacked.

Measuring What Changed

Numbers from the actual tf-modules migration, the ones I would bring to a compliance review:

The old pipeline touched seven external services on every run (github.com, registry.npmjs.org, releases.hashicorp.com, terragrunt.gruntwork.io, awscli.amazonaws.com, plus Node and terraform setup registries). The new pipeline touches one: GHCR. That alone is the difference between “I can reason about our dependencies” and “I cannot.”

Every release job now produces a verifiable chain: the consumer workflow verifies hagzag/tools signature; the tools image’s own build produced an SBOM with 2054 packages, a grype scan with zero highs, and a cosign signature tied to a specific GitHub Actions workflow identity. When the auditor asks “where did this terraform binary come from,” we have a chain of receipts, not a shrug.

Total tool installation time across a typical tf-modules PR dropped from roughly 90 seconds of npm install -g + setup-terraform + setup-node, to zero. The image is already warm on GitHub-hosted runners by the time the job starts. The measured wall-clock improvement is less important than the reliability improvement: the pipeline stopped having flaky dependency-install failures entirely.

What I’d Tell a Peer

  • Do not stop at signing. If the signature is never verified, it is cryptographic performance art. Close the loop in CI and at admission.
  • Gate on identity, not just signature. cosign verify --certificate-identity-regexp is where the actual trust decision lives. A signature from the wrong identity is just as dangerous as no signature.
  • Move the complexity into an artifact, not a workflow. Every npm install -g in a consumer workflow is tech debt. Pay it down once, in an image, with a build pipeline that emits SBOMs and signatures.
  • Pin digests, automate the rotation. Tag-based pins in production workflows are an accident waiting to happen. Renovate plus cosign verification on the bump PR is the pattern that actually works.
  • Admission control is not optional for regulated environments. FIPS and FedRAMP reviewers ask “what stops an unsigned image from running here?” The answer had better be a policy, not a process.

Wrap-Up — Closing the Loop

Part 1 set the vocabulary: SBOMs, provenance, SLSA, cosign. Part 2 argued that Wolfi and the daily-rebuild model make the FIPS/FedRAMP image rebuild problem genuinely tractable. Part 3 built the signed toolchain image with all the receipts. This part made those receipts mean something by consuming them — verifying in CI, enforcing at admission, and automating digest rotation with Renovate.

The headline is not that any single step is difficult. The headline is that each step is boring, which is what “mature” looks like in this space. No heroic patch cycles, no mystery npm install failures, no “which version of terraform was this pipeline running six months ago” archaeology. Just a signed image, verified in two places, pinned by digest, rotated on a schedule.

If you have been meaning to start this migration and were not sure where the first concrete step was — start with the consumer workflow. Pick your noisiest pipeline, move it into a container: job against a signed image, add a cosign verify step, and work outward from there. Everything else in this series is something you can bolt on later.

If you are already down this path and seeing it differently, I want to hear about it.

📖 Further Reading