Green CI, stale prod: assert the outcome, not the exit code

The pipeline is green. Every step exited zero. Production is running last week’s build. Everyone has a version of this story, and every version has the same ending: a customer found it.

A green pipeline proves that the commands ran. It does not prove that the world changed. The fix is to make the last step of every deploy a question to the live system, and to fail the run if the answer is wrong.

How a green run ships nothing

The ways are boring, which is why they keep happening:

• A tag that did not move. The manifest says app:latest or app:1.4, the image was rebuilt under the same tag, and the node already had that tag cached. imagePullPolicy: IfNotPresent does exactly what it says.
• A cache hit at build time that produced yesterday’s layers with today’s commit message.
• A rollout that did not happen because nothing in the rendered manifest changed, so the orchestrator, correctly, did nothing.
• kubectl rollout status returning success because the old pods are, indeed, ready.

None of these are bugs. They are tools doing their job while the pipeline asks the wrong question.

Make the system able to answer

The live system needs to be able to say which build it is. Bake the commit into the image at build time and expose it where a probe can read it:

ARG GIT_SHA=unknown
ENV APP_VERSION=$GIT_SHA

# the same endpoint the load balancer already probes
@app.get("/healthz")
def healthz():
    return {"status": "ok", "version": os.environ.get("APP_VERSION", "unknown")}

Tag the image with the commit, never only with a moving name. app:1.4 is fine for humans; the manifest gets app:1.4-3f9c2d1, or better, the digest.

The verify step

The step itself is small. It polls the public URL, through the load balancer, until the reported version equals the commit, or gives up loudly:

# .github/workflows/deploy.yml (after the deploy step)
- name: Assert the outcome
  env:
    URL: https://app.example.com/healthz
  run: |
    for i in $(seq 1 30); do
      live=$(curl -fsS --max-time 5 "$URL" | jq -r .version || echo "unreachable")
      if [ "$live" = "$GITHUB_SHA" ]; then
        echo "live version is $live"; exit 0
      fi
      echo "attempt $i: live=$live want=$GITHUB_SHA"; sleep 10
    done
    echo "::error::deploy did not converge to $GITHUB_SHA"; exit 1

Two details matter. Probe through the same path the users take, not kubectl exec into a pod, because the thing that lies is usually between the pod and the user: an old replica still in the pool, a cache, a load balancer that has not drained. And poll with a budget, because a rollout takes time and a single check at the wrong second produces a flaky pipeline that people learn to re-run, which is worse than no check.

For services with several replicas, ask several times and require every answer to match; a rollout that is half done reports both versions. For queues and workers, publish the version as a metric and query the metric instead of an endpoint.

The same idea for infrastructure

Infrastructure has its own version of “green but stale”: a plan that would change things nobody meant to change, merged because the apply step exited zero. The gate is the plan’s exit code, which Terraform makes explicit:

# exit 0: no changes · exit 2: changes present · exit 1: error
terraform plan -detailed-exitcode -out=tfplan

# a zero-diff gate: imported or refactored code may not change the world
- name: Zero-diff gate
  run: |
    set +e
    terraform plan -detailed-exitcode -input=false -out=tfplan
    rc=$?
    set -e
    if [ "$rc" -eq 2 ]; then
      terraform show -no-color tfplan
      echo "::error::this change is supposed to be a no-op but the plan has diffs"; exit 1
    fi
    [ "$rc" -eq 0 ]

That gate is what makes importing a hand-built cloud safe: a resource is only “under code” once the plan for it is empty. The same trick verifies a refactor of deployment charts. When two charts are unified into one, the proof is not that the new chart deploys; it is that it renders byte-identically for every environment:

for env in dev staging prod; do
  helm template app ./charts/old -f values/$env.yaml | sha256sum
  helm template app ./charts/new -f values/$env.yaml | sha256sum
done
# only when every pair matches does the cut-over happen

What it costs

A minute per deploy, a version endpoint per service, and one uncomfortable week: the first time the gate is on, it fails on things that were always broken and nobody knew. That week is the return on the investment, not the cost. Once the pipeline can tell the truth, a lot of other habits get simpler, because “it is deployed” stops being a matter of belief.