Launching 10,000 public pages with a guarded migration, and finishing the news rollout

10,000 pages, zero rollback

I needed to put ten thousand public answer pages live on Statpro without a maintenance window and without a rollback button if something broke. The answer was a guarded migration with checkpoint resumption, HMAC-signed batch approval, and a frozen publication-proof contract that locks the catalog state at launch time.

The problem: atomic activation at scale

Ten thousand pages is too many to flip in one transaction. But you also can't just iterate and hope, because a failure midway leaves a half-live catalog where some pages resolve and others 404. The middleware caches, the CDN edges need invalidation, and the database generation counter has to advance atomically so cached query results don't serve stale data alongside new pages.

The approach: split the catalog into shards, sign each shard batch with an HMAC, and write a checkpoint set after every batch commits. If anything fails, you resume from the last checkpoint rather than from zero. A canary review gate between activation and publication means the first batch has to pass human review before the rest are allowed to proceed.

The frozen publication proof

Here's the part I'm proudest of. Instead of trusting that the catalog state at activation time matches what was reviewed, the migration writes a frozen publication proof. This is a set of inline SQL literals that record exactly which leagues, stat types, and year ranges are covered. The same immutable SQL is used by both the Drizzle generated-column definition and the strict parity checks, so the database itself enforces the contract.

The evidence keys look like this (this is catalog metadata rather than product code):

const COVERAGE_EVIDENCE_KEYS = [
  "ask-coverage:2026-08-11.5|nfl|player-season-regular|1990-2025",
  "ask-coverage:2026-08-11.5|nba|player-game-postseason|2000-2025",
  // ... 40+ entries covering MLB, NBA, NFL
] as const;

Each key encodes the coverage version, league, stat type, and year range. If a future coverage expansion needs different bounds, it has to add a v2 column and token rather than editing this frozen v1 contract. That sounds rigid, but it's the whole point: the publication proof is a tamper-evident record of what was live at launch.

Batch approval with HMAC signatures

Every shard batch carries an HMAC signature derived from the batch contents, the activation intent, and a versioned approval key. The activation script verifies the signature before committing:

const BATCH_APPROVAL_VERSION = "056d.activation-batch-approval.1";
const BATCH_APPROVAL_HMAC_DOMAIN = "question-activation-batch-approval";

// Verify the batch was approved by the signing script
const expected = createHmac("sha256", key)
  .update(BATCH_APPROVAL_HMAC_DOMAIN)
  .update(batch.canonicalForm())
  .digest("hex");
if (!timingSafeEqual(sig, expected)) {
  throw new QuestionShardActivationError("batch signature mismatch");
}

The timingSafeEqual matters here. A naive string comparison short-circuits on the first mismatched byte, which leaks information about the signature through timing. This is the kind of detail that doesn't matter until someone writes a paper about your auth flow.

Checkpoint resumption

After every batch commits, a checkpoint set is written. If the activation script crashes (or the database connection drops, or the canary review fails), the next run reads the checkpoint set and skips already-committed shards. An advisory lock ([1452, 1] in Postgres) prevents two activation runs from stomping each other.

This is boring infrastructure. But boring infrastructure is what lets you launch 10,000 pages at 4 AM and go to bed knowing that if it failed at batch 47, you'd resume from batch 47, not from zero.

Completing the news rollout

The same 26-hour window also saw the NBC player news rollout go from staff-only monitoring to fully public. That involved a production monitoring and operations layer, evidence and takedown contracts for rights compliance, a database migration (0034) adding execution capability, and fixes to the public news category query and the RDS IAM database user configuration.

The takedown contract is the interesting piece. It's not enough to publish news; you need a structured process for removing it if a rights issue surfaces. The contract defines what a takedown request looks like, how it's validated, and what the runtime does when one is approved. That's the kind of thing that's easy to defer and painful to build after you need it.

What I'd do differently

The frozen publication proof took the most design effort, and I'm glad it shipped before launch rather than after. But the batch activation script ended up at 1,057 lines, which is more than I'd like for a single file. The checkpoint logic, the HMAC verification, the advisory lock handling, and the canary review integration are all in there together. If I were doing it again, I'd split the signing and verification into a separate module from the activation orchestration. The frozen proof module is already separate and clean; the batch orchestrator should follow that pattern.