Corrections kept falling out of the database
Saturday was one long NFL data repair on Statpro. The symptom was maddening: a stat we had corrected on a player page would quietly revert a day later. Eleven commits and one failed attempt later, the root cause turned out to be architectural, and the fix changed how I think about mutable state. By dawn the player pages were stable, but the path there ran through three separate bugs that shared one root.
Certified roster facts were being projected from whatever the mutable current-state tables held at read time. Patch the table, and the next nightly pass could overwrite your patch with a stale upstream value. You were correcting a copy. The original kept disagreeing.
Rebuild from the decision log instead
The fix that landed (PR #1932, migration 0117) stops trusting mutable tables for certified profiles. Certified decisions are append-only. A player's canonical profile is now projected from that decision history at migration time, which means a certified correction stays certified. You cannot overwrite the past.
The new shape, where the canonical profile never reads the mutable layer directly:
flowchart TD
U["Nightly upstream update"] --> L["Certified decision log (append-only)"]
M["Mutable tables (reference only)"] -->|read-only| P
L --> P["Certified projection, migration 0117"]
P --> G["Public pages"]
The edge to watch is the read-only one. The projection may look at the mutable tables, but it may never take a certified value from them.
There is a subtle payoff beyond durability: the projection is a pure function of the log, so it can be rebuilt from scratch and compared row for row against what is live. That rebuild-and-diff is exactly how the repair was verified before anything shipped.
Chronology was its own bug
Fixing content exposed a second problem: order. The database knew a fact had carried over, but not the sequence of carries, and projections that assumed a tidy order produced subtle nonsense (PR #1930). Think of it as git without parent pointers: every blob exists, but good luck reconstructing history. The certified carry chronology is now preserved explicitly, so a profile reflects the order events actually happened in.
The same disease, different organ
Player news had it too. A player's identity across sources was resolved incrementally, and a news revision could arrive keyed to a stale identity, so a correction written for one version of a player would land on another (PRs #1938 and #1939). Same lesson, smaller scale: when identity is implicit, every downstream write inherits the ambiguity. Making the identity continuity explicit is what let the news keys survive revisions.
Two smaller traps on the way
The publisher rendering the public pages had drifted from the database itself: its facts and the rows' actual carry semantics disagreed, so a page could show a value the DB no longer backed. PR #1929 reconciled the two. Then the identity bootstrap turned out to crash on healthy rosters whenever one player's ID evidence was simply absent, taking good players down with the incomplete one. PR #1935 makes absence a logged deferral:
if (identity == null) {
warn(`no identity evidence for ${player.name}; deferring`);
return bootstrapWithoutIdentity(player);
}One missing file used to abort a whole roster pass. Now the roster ships and the gap waits for the next evidence pass. That pattern generalizes to any bootstrap that quietly assumes completeness: treat absent evidence as a state, not a crash.
Why I trust it
The repo keeps a coordinate-proof plan for every repair attempt, including the 01:17 failure before the 18:55 fix. Failed attempts are documented next to successful ones, and CI now fails loudly when a daily ingest phase dies instead of letting the schedule limp on. By morning the NFL pages show certified source statistics and the dupe-baseline tests confirm the projection is stable. If corrections in your own system keep reverting, stop patching current state. Make the log of decisions load-bearing and project from it. None of this required rewriting the app. It required changing what the database refuses to forget.