A 60-second timeout that worked until production

The migration that passed every test and then timed out

I shipped a big unification of Statpro's Trending and public News surfaces yesterday: 85 files, 24,000 insertions, a new migration (0035) that mints deterministic public routes for every item in the corpus. Every test passed. The acceptance pack was signed. Then production tried to actually run the backfill.

It timed out at 60 seconds.

The migration SQL had this line, copied from the fixture-sized test budget:

SET LOCAL statement_timeout = '60s';
-- ...the deterministic route backfill exercises the row-level
-- minting trigger for every currently public item

Sixty seconds is plenty when your test database has a few hundred rows. Production has a 24-month corpus across MLB, NBA, and NFL. The backfill fires a row-level minting trigger for every public item, and that trigger does real work per row. The original timeout was sized for the fixture rather than the corpus.

The fix was one line, plus a test that asserts the old value never comes back:

SET LOCAL statement_timeout = '15min';

assert.match(sql, /SET LOCAL statement_timeout = '15min'/u);
assert.doesNotMatch(sql, /SET LOCAL statement_timeout = '60s'/u);

I kept the operation bounded (15 minutes, not unbounded) because a runaway migration is worse than a slow one. The assertion locks it in: if someone shrinks the timeout back to a fixture-friendly value, the test fails. The lesson is boring and I keep relearning it: your test database is not your production database. Timeouts that feel generous in local runs are invisible until they aren't.