Every render was a cache miss
The night of October 4th, a crawler walked 290 binder-idea pages on Binderdex in about 70 seconds. No errors, nothing dramatic, every request answered. But each of those renders ran the same three catalog scans, and the related-lists panel had no cache in front of it, so all three hit Postgres every single time. Database CPU hit 95%, the connection pool started timing out, and the first thing users saw was nowhere near the pages the crawler visited: card search in the mobile app started failing.
Here is the part worth internalizing. A connection pool is a queue with a clock. Ours had 25 slots and a 10-second connect timeout, sitting on one small compute task. Three slow scans per render don't look dangerous until a burst multiplies them, and then every query in the system waits behind the crawler. The subject lookup the error tracker titled the incident was just the first query to give up. The scans behind it were the real cost.
The fix is boring on purpose. The related lists depend only on the page category and archetype, so they now sit behind a cache keyed on exactly that, with an hour of revalidate. Key the cache on what the data actually depends on, and a crawler stops being an incident:
const related = unstable_cache(
() => listRelatedSubjects(category, archetype),
["related-subjects", category, archetype],
{ revalidate: 3600 },
);The index that could not say no
That same night, the error queue served a second lesson, about column order. The card page was slow to show graded prices: a p95 of 1.2 seconds on one query. Prices live in a partitioned table, and the best available index was on (card_id, recorded_at). The query also filters by source, and Postgres can't use that index for the source filter. So every partition ran a backward time scan and threw its rows away on the filter. One source has only had data since mid-May, so a hot card meant sixteen older partitions scanned at roughly a thousand rows each, returning nothing.
The composite index below is declared in the schema and built per partition. Column order does the work: equality on card_id, equality on source, then sort order. A partition with no matching rows can now answer instantly instead of scanning.
-- column order: card_id (equality), source (equality), recorded_at (sort)
CREATE INDEX graded_prices_card_source_recorded
ON graded_prices_p2026_05 (card_id, source, recorded_at DESC);Receipts, because I don't trust vibes on this stuff. On scratch partitions seeded with 300,000 noise rows, the old plan removed about a thousand rows per stale partition at 1.29 ms; the new plan returns zero rows per stale partition at 0.34 ms. In production, EXPLAIN (ANALYZE, BUFFERS) on a hot card with roughly 25,000 price rows touched 16,291 buffers and reported 259 to 1,409 rows removed by filter on every stale partition. That run was cold-cache, so the 6.1-second absolute is inflated, but the shape is the whole story.
Three things to steal
- Key caches on real dependencies. The related lists never varied within a category and archetype; computing them per page paid for a difference that didn't exist.
- On partitioned tables, a missing index column is a per-partition tax. Column order decides which partitions can answer "no rows" and skip the scan.
- The alert names the innocent. The query in the error title was the first waiter for a connection, not the slow one. Read the plan, then believe the title.
Two database posts in a row now; the catalog pipeline one from last week is the sibling. The site is fast on good days, and this week was about the bad ones. If you run a small pool behind a render path, go read your access logs for crawler bursts tonight. The incident in one picture:
graph TD C["Crawler: 290 slugs"] --> B["290 renders, 3 scans each"] B --> P["Pool: 25 slots"] S["User traffic"] --> P P --> E["Mobile search 500s"]