The projections on Statpro's props boards come from models we train and run ourselves, and for a long time they graded their own homework: forecasts were compared against forecasts. That changed last night. The new evaluation path scores past predictions against the market's closing line, and it caught two real bugs on its first run.
Grade against something outside yourself
The new path works through one command pipeline: build a history of past prop lines, replay what the market settled on, then score every projected distribution against that. It also prices combination markets like points+rebounds+assists from the joint simulation instead of stitching margins together by hand, and it recalibrates each stat's variance so the model's own settled prices score best.
The trick is that the grader is deliberately not the model. A model checked against itself can be confidently wrong and never notice. The market line is the opposite: it isn't always "right," but it's a stubborn fact about what the world believed, and beating it consistently is the honest definition of a good model.
One honesty rule is baked in: each recalibration only sees results settled before that moment's snapshot, so the calibration can't peek at the answers it's being graded on.
Bug one: the evaluator saw one season instead of three
The first run scored only 24% of the props it should have. Cause: the football evaluator loaded historical seasons up to the start of the window and nothing newer, so a "three-season" evaluation graded one season of props and silently ignored the rest. Priors were right; the evidence was truncated. It now loads every season the evaluation window covers.
Worse was hiding underneath. The evaluator handed the passing line to a single guessed starter, so every real start made by anybody else scored the over at exactly zero. Anyone who has watched a lineup sheet change an hour before kickoff knows how well that goes. The fix pooled every sampled lineup, made each quarterback eligible to take the pass, and now reads each player's simulated stats conditional on him actually being in the lineup.
And the projects stopped pretending every player plays: forecasts are now conditioned on the player being in the game, unlisted players get a fitted play chance, and recent weeks weigh more than old ones when sharing out touches. The basketball side got the same treatment: a confirmed starter now counts as playing, which is exactly the boring fact the projections were missing.
Small context beats big architecture
A separate change gave the basketball minutes model the player's own layoff length, games missed, recent minutes volatility, and how packed the schedule is around the game. Results across the January-to-March walk-forward window of held-out games: minutes error dropped from 4.89 to 4.81 without lineup confirmations, and from 4.64 to 4.57 with them. Points log loss improved the same way. And the evaluator now reads availability the same way the daily job does, so an offline grade and a game-night projection see the player the same way.
The takeaway is the lesson I keep relearning: a small honest context feature beats a bigger architecture. Also, add a free public source that lists named starters and bench players for games about to tip, checked every ten minutes on game days, because the model can only react to what it can see.
Why this beat more modeling
This bought more than the same hours spent on new features would have. It found real bugs, gave every future change a neutral referee, and moved a number that matters, minutes error from 4.89 to 4.57. The models now argue with the market instead of with themselves, and both sides are better for it.