How do you generate thousands of sports questions without an LLM in the loop?
Ask StatPro is the natural-language query layer on top of StatPro: you type a question like "Who led the NBA in assists in 2023?" and it parses, resolves, and answers. The parser already worked. The problem was coverage. We had maybe a few dozen hand-written questions, and every new league or stat type meant another manual entry. We needed to generate candidates at scale without an LLM inventing questions that produce nonsense answers.
The approach we shipped this weekend: a four-stage pipeline that generates deterministic question candidates from a versioned capability matrix, verifies each one against the live answer engine, and activates only the ones that come back clean. No LLM in the generation path. Every candidate has a deterministic ID and full provenance.
Stage 1: The capability matrix
Before generating anything, you need to know what you can actually ask. The capability matrix is a versioned contract (v1.0.0) that declares, for each question family, which leagues are supported, what answer shape it produces, and whether it is deterministic, model-dependent, or subjective. This is the difference between "we can answer leaderboards for NBA, NFL, and MLB" and "we think we can maybe answer some leaderboard-ish things."
Each entry has a generation state: production-ready, dry-run-candidate, or blocked. The generator respects these states. Nothing blocked ever produces a candidate. Nothing in dry-run ever reaches production.
Stage 2: Candidate generation from versioned seeds
The generator takes a seed pack (also versioned, currently statpro-historical-scenarios v1.0) containing real player, team, and season scenarios with closed-set variable substitutions. It walks every capability family, fills templates with seed values, and produces candidate question strings. Each candidate carries full provenance:
export interface QuestionCandidateProvenance {
readonly capabilityMatrixVersion: typeof QUESTION_CAPABILITY_MATRIX_VERSION;
readonly familyId: string;
readonly generationState: CapabilityGenerationState;
readonly league: MuseLeague;
readonly templateId: string;
readonly templateIndex: number;
readonly seedPackId: string;
readonly seedPackVersion: string;
readonly seedScenarioId: string;
readonly variableBindings: Readonly>;
} Why does this matter? When a question breaks in production, you can trace exactly which template, which seed scenario, and which variable binding produced it. You can also reproduce it deterministically. There is no model temperature to blame.
Before a candidate is even emitted, it goes through parser preflight. The question string is parsed by the same Muse parser that handles live user input. If the parser rejects it, refuses it, or classifies it as a different intent type than expected, the candidate is skipped with a documented reason. The skip reasons are an enum rather than a free-text field:
export type QuestionCandidateSkipReason =
| "blocked-generation-state"
| "preview-required"
| "league-not-generatable"
| "missing-seed-scenario"
| "parser-refusal"
| "parser-intent-mismatch"
| "duplicate-normalized-question"
| "identity-collision";The duplicate-normalized-question check is the one that saves you from yourself. Two different templates can produce the same normalized question string. Without dedup, you ship two entries that answer identically, and the autocomplete shows the same question twice.
Stage 3: Verification against live answers
This is where it gets interesting. A generated candidate that parses correctly is different from a candidate that produces a correct answer. The verification stage takes each candidate, runs it through the full Ask evaluation pipeline (the same one users hit), and classifies the result.
Every verification result lands in one of three statuses: verified, failed, or ineligible. Failures are categorized:
- refusal: the answer engine refused to answer (unsupported question type).
- unsupported: the question parsed but the answer shape is not implemented for this league or era.
- empty-result: the query ran but returned zero rows. This is where you learn your seed scenario references a player who did not play that season.
- broken-result: the answer came back but the structure is wrong (missing tokens, invalid cells, non-finite numbers).
- timeout: evaluation exceeded the timeout. The verification itself has a
VerificationTimeoutErrorwith the timeout in milliseconds. - execution-error: something threw.
Ineligible is different from failed. An entity-lookup redirect (where the parser decides this is not a question but a player search) is ineligible rather than failed. The question is fine; it just is not the kind of thing that goes in a question catalog.
Stage 4: Activation of verified candidates only
The final stage takes the verified candidates and writes them to the catalog. This is a separate commit, a separate tool, and a separate gate. The generator has no production write path. The verifier has no production write path. Only the activator writes to the catalog, and it only writes candidates that passed verification. The activation step also normalizes catalog entries so the autocomplete surface stays clean.
The result: the question catalog went from a handful of hand-written entries to a generated, verified, versioned catalog that grows with the capability matrix. Adding a new question family means adding it to the matrix, adding seed scenarios, and running the pipeline. The generator, verifier, and activator are all CLI tools you can run locally. No deployment required to preview what the next catalog will look like.
Why not just use an LLM?
We could have prompted a model to generate sports questions. It would have produced plausible-looking questions that parse fine but return empty results, wrong answers, or broken answer shapes. The verification stage would catch most of them, but you would be paying API calls to generate questions that immediately fail. With deterministic generation from templates and seeds, every candidate is structurally valid by construction. The verifier is there to catch data-level issues (a player who did not play that season, a stat that does not exist for that era).
The capability matrix is the real unlock here. It is a single source of truth for what the system can answer, what it plans to answer, and what it has decided is out of scope. When someone asks "can StatPro handle X?", the answer is a row in the matrix.