Yesterday a Statpro game page said "Final" next to an empty box score. Nothing had crashed; the pipeline had simply failed quietly and the page rendered what it had, which was nothing. That gap between promised and rendered is the whole problem, so last night's session (41 commits in one push) built the thing that closes it: a data health control plane that is fail-closed by default.
Derive the promise from upstream, then compare
The design decision I'm proudest of is one sentence long: derive the expected-game universe from retained provider schedule observations, independently of the canonical games table. Why? Because a health report built from your own games table can never detect the game ingestion omitted entirely. It will happily certify the universe it can see. You need a second, independent inventory of what should exist, then a comparison, then a verdict.
Dry run by default, write behind two locks
The controller (a CLI a scheduler can invoke) has no flags. That's the whole trick. A scheduler cannot widen its own window, and it cannot turn a dry run into a write without a deployment-owned gate plus the controller's own identity. The shape, simplified from the real code:
export function runtimeConfig(env: ProcessEnv): RuntimeConfig {
const wantsWrite = env.HEALTH_CONTROLLER_WRITE === "enabled";
if (wantsWrite && env.SERVICE_IDENTITY !== CONTROLLER_IDENTITY) {
throw new Error("controller write requires the controller identity");
}
return { mode: wantsWrite ? "write" : "dry_run" };
}The comment in the real implementation is better than anything I'd have written: there's deliberately no reviewed invocation adapter yet, so the code refuses to claim a lease just because some ARN-shaped config happens to be present. Missing half the dispatch configuration becomes an explicit, typed limitation instead of a shrug.
What the public is allowed to see
Game pages now expose three honest states: available, pending, or incomplete. A final game never silently renders an empty box score again; if our copy is incomplete, the page says so. Internal provider details stay private; the states are sampled and deduplicated health events.
The slice landed heavy on tests, on purpose: the controller and its repository shipped with roughly 1,400 lines of test code against about 2,000 lines of implementation, and most of those tests are failure tests. The question they answer is "what does the controller do when the database is lying, the dispatch is half-configured, or the clock is wrong," because those are the only cases that matter at 3am.
Corrections leave a trail
One more design choice worth stealing: health verdicts are stored as append-only attestations that supersede earlier ones and reference the exact evidence components behind them. There's no "update the health row in place." If a certification later looks wrong, you can walk the chain and see which evidence produced it. That discipline saved us the same night it shipped: the last commits of the session were a small repair (an incremental probe had been discarding a stat it shouldn't), a fix, and then a reviewed certification of the rollout, all recorded as new attestations rather than edits. The history reads like a lab notebook because it is one.
The Effect pilot has to earn it
Alongside the control plane, each sport's transport moved to Effect (MLB, NBA, NBC, then NFL) with typed failures and deterministic failure tests, and the odds boundary got a shadow pilot: baseline and Effect resolvers decode the same captured response, so the comparison is free. The promotion gate is written down before the experiment: 14 days minimum, 1,000 provider events, 99% result parity, zero unexplained canonical-impacting disagreements, no p95 or cost regression. Framework adoption is a hypothesis here, and the pilot has to earn it with numbers.
Takeaway
If you run any pipeline that renders promises to users, two things transfer: build the expected-truth inventory from upstream observations so omission is detectable, and make every automatic writer fail closed with the write switch living in deployment config rather than CLI flags. The empty box score wasn't a rendering bug. It was an organization chart bug, and now there's a component whose only job is refusing to guess.