Certifying a season: the ratchet that tamed historical NBA data

Certifying a season: the ratchet that tamed historical NBA data

Twenty-six commits landed in Statpro yesterday, and at first glance they look like a pile of unrelated fixes: parse numeric minutes, trust boxscore team identity, repair anonymous player rows, backfill games started, correct appearances and rounds. They're not unrelated. They're one pattern applied over and over, and by end of day three seasons were certified green.

The pattern

Every historical season is a wall of messy source data: boxscores with placeholder rows, teams that played under different identities then, players swapped into the wrong roster slots, minutes logged as strings. You can't fix it all at once. What you can do is fix one class of defect, gate it with a test, and refuse to let that class regress while you move to the next.

That's the ratchet. The commit messages tell the story in order: guard the schedule merges, trust team identity, preserve team identities, repair anonymous rows, parse numeric minutes, backfill games started, certify the 1992 season. Each step closes a defect class and turns it into a green test. By the time you reach "certify," the season is locked.

The one bit of code worth stealing

The biggest speedup came from parallelizing box fetches. The old loop processed units serially inside a batch. The new one asks the source how much concurrency it tolerates and drains in slices:

const concurrency = Math.max(1, Math.floor(
  source.fetchConcurrency?.(resourceType, config) ?? 1
));
for (let i = 0; i < batch.length; i += concurrency) {
  const results = await Promise.all(
    batch.slice(i, i + concurrency)
      .map((unit) => processUnit(db, source, config, resourceType, unit))
  );
  fetched += results.filter(Boolean).length;
  failed += results.filter((ok) => !ok).length;
}

Two things make this more than a textbook Promise.all. First, concurrency is per-source, per-resource-type: historical NBA box endpoints tolerate four in flight, other lanes stay at one. Second, the failure accounting is honest: every unit in the slice is counted, fetched or failed, so a crashed batch never silently drops work from the queue.

What shipped for users

By end of day the 1990, 1991, 1992, and 1993 NBA seasons are repaired and certified, historical NFL schedules reconcile cleanly back through pre-1999, and the app's reference search and the Ask surface now hit one unified entity index instead of two parallel ones.