Workers that sleep through the off-season

Last night I noticed the stats site's game-sync workers had been running 24/7 since launch, burning Fargate capacity through every idle minute between games. The fix sounds boring (scale to zero when nothing's live), but the interesting part was everything that immediately conspired to undo it.

The design: each league puts itself to bed

Each league's worker runs a planning pass for its league every minute, tracks whether it actually did box work, and if the queue is empty and nothing's happened for 15 minutes, it scales its own ECS service to zero. A separate scheduler drops to a 10-minute tick and wakes a stopped worker 45 minutes before any game starts. Rollback is a single environment variable pointing at the old always-on stage.

Three things that kept waking the workers up

Shipping "scale to zero" is easy. Keeping it asleep took three separate fixes in the same night:

1. Our own deploys. Every deploy ships a new task definition, so CloudFormation calls UpdateService, and it was helpfully passing DesiredCount=1 along with it, restarting any worker that had just put itself to bed. The fix: stop declaring the count in the template entirely, and CloudFormation leaves the live count alone. (New services still start at 1, and rollback stages set it explicitly.)
2. Non-work that looked like work. The NFL injuries facet re-touches final games every few hours, and my "is this worker busy" check counted that as activity. The worker could literally never idle. The activity check now only counts box work on final games.
3. The idle bastion. The database bastion host sat up around the clock for maybe one SSH session a week. An hourly Lambda stops it after 45 minutes without an active session; the tunnel script starts it back on demand.

Plus some honest accounting: dropping 34 CloudWatch metric series that no alarm or dashboard read (about $10 a month), pinning tasks to one availability zone next to the database to kill cross-zone transfer charges, and taking the scheduled tasks onto Fargate Spot.

One part of the rollout I'm genuinely glad we didn't skip: months ago, a different change moved scheduled tasks off Fargate Spot because their invocations kept landing in the dead-letter queue. Nobody had revisited whether that was still true. Before re-enabling Spot, we ran a one-off scheduled task on Spot in production with a harmless dry-run command, watched it start, exit cleanly, and leave the DLQ at zero, then deleted the canary. Five minutes of proof beats an outage-shaped surprise, and the runbook paragraph that banned Spot now explains the fix instead.

The pattern worth stealing

The reusable piece isn't "scale to zero," it's the rule about who's allowed to wake you. The first version failed because the worker's own activity signal included maintenance noise. Define activity as the thing you'd actually charge for (box work on games that matter), and let everything else be invisible to it. Deploy tooling, housekeeping passes, and your own deploys should never register as demand.

Also on tonight: a measured run of the game-recaps lane came in at 211 October games for $0.31 (about $0.0015 a game), so that schedule is now on every 3 hours. And the Google Analytics tag is gone entirely; PostHog already captures page views with referrers, and GA's numbers turned out to be 93% three-second "Direct" bot visits anyway.