Building a sports Q&A engine without an LLM

I wanted StatPro to answer questions like "Who leads the MLB in home runs?" and "What's Aaron Judge's batting average this season?" The obvious approach is to pipe the question into an LLM, let it generate SQL or a function call, and return the result. I didn't do that. I built a deterministic parser instead, and it parses a question in 0.05 milliseconds.

Why not just use an LLM?

Three reasons. First, the question space is closed. Sports stat questions follow a small number of structural patterns: who leads X, what are Y's stats, compare A vs B, what's Z's record, when does team W play next, who won the championship. There are maybe a dozen intent shapes and you can enumerate them. Second, an LLM answer can be wrong in ways you can't debug. If it misremembers a stat or hallucinates a player, you have no audit trail. A deterministic parser either resolves to a typed intent or refuses. Third, cost and latency. Every LLM call is 200-500ms and a fraction of a cent. My parser does 0.05ms and costs nothing. For a Q&A surface that needs to be SEO-indexable and fast, that matters.

I studied a competitor we admire (they do this well) and the killed ask stack from an earlier plan (which did it wrong). The anti-pattern post-mortem from the earlier attempt was clear: the old stack tried to be too flexible, accepted free-form input, and produced wrong answers it couldn't trace. The new engine is a closed grammar. Every phrase the parser recognizes is an explicit lexicon entry. If it doesn't understand your question, it tells you so honestly.

The architecture: normalize, tag, slot-fill

The parser is a pipeline of four stages, each pure and testable:

1. Normalize: lowercase, strip punctuation, collapse whitespace. The input "Who leads the MLB in HR's?" becomes "who leads the mlb in hrs".
2. Gazetteer: longest-match scanning of player and team names over the word stream. Dictionaries are built once from the canonical DB. Last-name-only matches are allowed only when unambiguous within the league index, and reserved words (metric names, league words, stopwords) can never be swallowed by a bare-last-name match.
3. Lexicon: per-league metric synonyms, timeframe grammar, and superlative senses. "Homers," "dingers," "long balls," and "hrs" all map to the MLB home runs metric. Each metric carries its NLG fragments: the verb "has hit" and the noun "home runs" so the answer sentence assembles itself.
4. Intent matcher: tag the normalized stream (entities, metrics, timeframes, keywords), then slot-fill the first matching rule. Every path ends in a typed intent or a typed refusal.

The core dispatch is a pipeline of matchers that each consume words and tag them, or decline:

// Each matcher consumes >=1 words at `index` and tags, or declines with 0.
// The parser walks left to right, tagging as it goes.
type Matcher = (tagged: TaggedQuestion, words: string[], index: number) => number;

// Then intent resolution tries rules in priority order:
function matchIntent(tagged, league): MuseParseOutcome | null {
  return (
    compareIntent(tagged, league) ??   // "Judge vs Ohtani"
    leadersIntent(tagged, league) ??   // "Who leads MLB in home runs?"
    playerIntent(tagged, league) ??    // "Judge stats" / "Judge" (lookup)
    teamIntent(tagged, league)         // "Yankees record" / "next game"
  );
}

The refusal is the feature

The part I'm proudest of is the preflight refusal gate. After tagging, if there are leftover content words the parser didn't consume, it refuses rather than half-answering. "Who leads the MLB in dingers and RBIs" has two metrics competing for the leaders slot. The parser doesn't pick one and hope. It refuses with a typed reason and logs the unmatched words so I can grow the lexicon from real misses.

// Strict correctness: leftover content words mean we didn't
// fully understand the question, so refuse rather than half-answer.
function preflightRefusal(tagged: TaggedQuestion): MuseRefusal | null {
  if (tagged.ambiguousEntity && !hasEntities(tagged)) {
    return refusal("ambiguous-entity", tagged.unmatched);
  }
  if (tagged.unmatched.length > 1) return refusal("no-parse", tagged.unmatched);
  return null;
}

Every refusal carries a typed reason: ambiguous-entity, no-parse, ambiguous-league. The ask experience shows honest refusal views rather than blank results. And every question (answered or refused) is logged with its parse outcome, so the lexicon grows from real traffic rather than me guessing what people might ask.

QA: the golden corpus

The parser ships with 57 golden-corpus tests asserting full parse outcomes, including refusal cases (ambiguous entities, unsupported directions, non-fact questions like "who is the best player?"). There's a bench gate at <10ms per parse; actual is ~0.05ms. The gazetteer and lexicon have their own unit suites, and the NLG layer (which emits entity-linked token streams for answer sentences) has snapshot tests.

What's live now

The ask bar replaces the old search bar on the hub. Example chips show corpus-tested queries. Answer pages render on team-color heroes with tokenized NLG sentences (entity tokens link to player/team pages, inferred timeframes render muted). All six intent families answer: leaders, player stat lines, player compares, team records, team schedules, and champions. One bug caught live: NBA/NFL championship years map to their ending calendar year ("2016 NBA Finals" is the 2015-16 Cavaliers), and the first version had that backwards.

The parser is a package called @statpro/muse. It has zero runtime dependencies and no LLM calls. It's 71 tests and runs in a fraction of a millisecond. The lexicon grows from logged misses rather than guesses. And when it doesn't know the answer, it says so.