Statpro's Ask feature learned to refuse questions it can't prove

The problem with a confident answer engine

I shipped Statpro's Ask feature a while back: you ask it a natural-language sports question, it figures out your intent, pulls the numbers, and renders an answer. The thing is, "figures out your intent" was doing a lot of quiet work. Ask "who has the most Gold Gloves?" and you'd get a clean leaderboard. Ask "who has the most All-Star selections?" and sometimes you'd get... the Gold Glove leaderboard. The answer looked right. The pipeline had no way to prove it actually answered the question you asked.

So yesterday I spent the whole day fixing that. Seven commits, one theme: the answer pipeline now has to prove the answer matches the question before it's allowed to render.

The fingerprint that makes Ask honest

The core idea is a typed discriminated union of every question kind the system actually knows how to answer. Not "a question about awards" as a vague string, but one of ~40 named kinds: award-winner, coach-win-leaders, team-record-vs-team, threshold-career, and so on. Each parsed query intent gets normalized, then a stable SHA-256 fingerprint is computed over it.

Why a hash of the normalized intent? Because two phrasings of the same question ("most Gold Gloves all-time" vs. "Gold Glove career leaders") should map to the same fingerprint. The fingerprint is correlation-only; it's never answer-certification evidence on its own. What certifies an answer is whether that fingerprint matches a known semantics kind.

Here's the technique, stripped down to the part that matters. The stableValue recursion sorts object keys at every depth before hashing, so key ordering in the parsed intent can't change the fingerprint. Without that, the same logical query would hash differently depending on which parser branch touched it last:

function stableValue(value: unknown): unknown {
  if (Array.isArray(value)) return value.map(stableValue);
  if (value && typeof value === "object") {
    return Object.fromEntries(
      Object.entries(value)
        .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))
        .map(([k, v]) => [k, stableValue(v)]),
    );
  }
  return value;
}

// Same question, different phrasing -> same hash.
function intentFingerprint(intent: ParsedIntent): string {
  return createHash("sha256")
    .update(JSON.stringify(stableValue(normalize(intent))))
    .digest("hex");
}

The branch on the right is the whole point. Before this, Ask would render something for every question. Now, if the fingerprint doesn't match a kind the system has verified answers for, the answer is suppressed. A wrong answer that looks right is worse than no answer, because you can't catch it by reading.

The long tail of the same beat

Once the fidelity gate existed, the rest of the day was making sure every question kind actually passed it:

• Coach and award asks: the award-contracts and coverage layers got their own fidelity paths so "who won the 2019 Cy Young" and "which manager has the most wins" stop borrowing each other's answer shapes.
• Record breakdowns: the publication helper was undercounting team-record splits, which made the verified cohort look smaller than it was. Fixed the counter, and the baseline snapshot that tracks duplicate code blocks.
• Verified cohort preview: shipped an owner-only preview of the full verified Ask catalog (about 100 pages) so I can eyeball answers in production before they go public. Team-colored hero banners per league, because a leaderboard without team colors reads like a spreadsheet.
• Suggestions rendered once: the team-leaderboard view was rendering both a related-questions block and an example-questions block. One assertion later, it renders one. Small bug, but it's the kind of double-render that makes a page feel broken even when the answer is correct.

What I'd do differently

The fidelity union should have existed before the first answer shipped. I built Ask top-down (parse, answer, render) and added the proof layer after. In hindsight the proof layer is the product: nobody trusts a sports reference site that's wrong sometimes. If I were starting over I'd write the typed semantics first, answer nothing until each kind had a verified test, and treat the render layer as a view over already-certified data. The gate is cheap to add late, but the answers you shipped before it are out there.