Read-only production migration diagnostics, iterated into shape

When a production migration fails, you need to see what it saw

Statpro runs a Step Function migration pipeline in production. When an execution fails, the failure is in the network path between the state machine and the container image registry. It is hard to look at that from outside. The infra team needs a read-only way to diagnose what the execution actually saw before they can fix it.

Eight commits, eight refinements

The day started with a single read-only workflow that validates an exact failed execution and extracts evidence from the state machine. Then the work split into eight commits as the design toughened through what it actually needed to attest:

1. Split lookup sessions: broke the diagnostic into separate lookups by domain, so each step is scoped to one query.
2. Scope the task definition read: make sure the diagnostic only reads the exact task definition it needs, rather than the whole cluster.
3. Bind the state machine qualifier: ensure the workflow cannot accidentally query a different state machine.
4. Attest repeated image layer references: the container image has duplicate layers across regions; the diagnostic needs to verify it reaches the same layer each time.
5. Expose effective diagnostic facts: the final commit pulls the facts the infra team actually needs into the artifact output.

Each commit is a refinement of the previous one. The first version worked but was too broad. The scoping and attestation added in later commits are what make it safe to run in production without accidental reads or missing layers.

Pin every input before you touch anything

The key idea repeats throughout the day: each diagnostic step validates its inputs against exact expected values before it does anything. This is how you pin a read-only workflow to one execution at a time:

set -euo pipefail
  [[ "$APPROVED_SHA" =~ ^[a-f0-9]{40}$ ]] || fail
  [[ "$FAILED_EXECUTION" == "execution-12345" ]] || fail
  # stop here if anything does not match
  artifact_dir="$(mktemp -d "$RUNNER_TEMP/diag.XXXXXX")"
  mkdir -p "$artifact_dir/evidence"

The first guard pins the commit SHA to a 40-character hex string. The second pins the exact failed execution to a literal string. Nothing writes or reads until those checks pass. That is what makes this runnable in production without concern about accidental reads.

The result is a diagnostic that is scoped to one execution at a time, attests its facts by reading only the exact resources it needs, and hands the infra team an artifact they can trust.