The stats site's NFL injury pipeline failed its weekly publish on four runs in a row last week, and not one of them paged me. The alarm I built to catch exactly this sat silent the whole time.
Three lies in one receipt
The fix touched three assumptions, each of which had quietly gone stale:
- The failure receipt said only
publish. No step, no message, no clue. Failed weeks now carry the step that died (parse, identity, write), the job's own message, and the database error code. - The alarm treated a "partial" run as a completed write. A week that hasn't been posted yet is genuinely not published, so if a run reports partial, something real failed. It now pages.
- The week selector started 72 hours in the past, so already-played weeks kept getting fetched and republished, 28 reports twice a day, for days after they mattered. Selection now starts at the run clock, so a week drops out once its last game kicks off.
There was a fourth bug under it all. On Wednesdays the league site lists every matchup for the new week but shows a "no injuries reported" notice where the report table will be. The parser demanded two tables per matchup, so the whole week failed to parse until every team's report existed. It now reads the notice as its own state and skips those teams rather than publishing an empty report.
The takeaway
The next time an alarm doesn't fire, the question isn't "is the alarm broken?" It's "what state did the alarm never imagine?"
One detail I like: the verify page's click race got fixed too, after a test discovered it had been clicking before the scroll settled. Tests that never click what users click don't count. Monitors deserve the same skepticism. If it has never fired, you don't know it works.