Truth-blind calibration: building a timed review exercise for AI-resolved card identities

The problem with trusting your own resolver

I spent most of Monday on a question that doesn't have a clean answer: how do you know your automated entity resolver is actually right? Binderdex matches incoming trading card news stories to catalog card and set identities. The resolver had been shipping matches for a while, but "it works on the cases I checked" isn't the same as "it works." I needed a number.

So I built a calibration exercise. Not a test suite (those already exist, 599 passing) but a truth-blind timed exercise where a fresh operator sits down, looks at 50 resolved candidates with their evidence but without the ground truth, and decides: approve or reject each one. The clock runs. The receipt is strict. If you get all 50 right in under 10 minutes, the resolver passes. If you don't, it fails.

How the pack gets built

The exercise runs entirely offline. No database connection, no production reads, no network in the worksheet itself. Here's the shape of it:

1. Corpus: 100 owner-attested news stories with known correct card/set identities, frozen and hashed.
2. Catalog snapshot: a validated Gate B catalog export, hashed and bound to the corpus.
3. Pack generator: picks 50 candidates using a coverage-first selection policy that maximizes game, matcher, and relation-type diversity.
4. Offline worksheet: a single HTML file with a CSP that blocks network, storage, cookies, and external assets. The operator sees candidate metadata and resolver evidence but never the truth.
5. Receipt: the operator's 50 decisions, timing data, and identity bindings, downloaded as strict JSON.
6. Verifier: an independent command that reconstructs hidden truth from the corpus and checks every decision, the ordering, and the timing integrity.

The worksheet is the part I'm proudest of. It's a self-contained HTML file. You open it, press Start, and you get 50 candidates one at a time. Each shows you the card number, the resolved parent set, the matcher that produced the match, and the evidence string. You approve or reject. When you finish, you download a receipt. That's it. No resume, no source navigation, no way to peek at the truth.

The honest result

The first operator run scored 46/50 in 42 seconds. Too fast, and too many wrong. After I improved the visible identity context on the worksheet (showing card numbers and parent set names), a fresh truth-blind run scored 44/50 in 9 minutes 35 seconds. The verifier exited with code 2. The workload failed.

That's the real number. Four candidates were wrong, and the pack is sealed. The rules say you can't retry a finalized pack once its truth is revealed. Any future exercise needs a fresh operator and a fresh pack. So 44/50 stands.

The failure is the useful part. Tests pass silently. A timed human exercise fails loudly, and the failure pattern tells you which matcher or relation type needs work.

What the resolver remediation found

Before the timed exercise, the resolver itself went through a remediation pass. The key change: it now uses canonical source URLs as bounded structured evidence. If a story's URL points to an official One Piece card game page, the resolver extracts the card number from the URL path itself, with strict validation:

// Simplified: extract card numbers from trusted official URLs
function extractCardNumbers(parsed: URL, sourceId: string): string[] {
  if (sourceId !== "onepiece-official") return [];
  // Notice pages: /topics/OP01-001.php → ["OP01-001"]
  const notice = parsed.pathname.match(noticePath);
  if (notice?.[1] && !parsed.search) {
    return [normalizeCardNumber(notice[1])];
  }
  // Card list search: /cardlist?freewords=OP01-001 OP01-002
  if (parsed.pathname === "/cardlist") {
    const freewords = parsed.searchParams.get("freewords") ?? "";
    if (isValidFreewords(freewords)) {
      return freewords.split(" ").map(normalizeCardNumber).filter(Boolean);
    }
  }
  return [];
}

The interesting part isn't the extraction. It's what gets rejected. The parser checks for dot-segment encoding bypasses (%2e, %2f, %5c), rejects URLs with credentials or ports, and validates that the raw URL path matches the parsed path exactly. A URL that looks right to a browser can still be wrong if it's been encoded in a way that bypasses the path check. The resolver treats only canonical-form URLs as evidence.

After the remediation, the resolver scored set precision/recall 1.0/1.0 and card precision/recall 1.0/0.75 on the offline Gate B evaluation. Four ambiguous card truths were left as deliberate false negatives rather than unsafe auto-links. Cluster recall stays at 0 because the acceptance contract defines no recall floor yet.

The delegation model

One more thing shipped today that's worth mentioning. I switched the Binderdex agent orchestration to a Sol/Terra split: one agent (Sol, at high reasoning) owns planning, architecture, adversarial review, and commit narrative. Another agent (Terra, at standard) owns bounded implementation. They operate on disjoint files in parallel. Same-thread retry first, then Terra escalates to high reasoning if needed.

This isn't theoretical. The resolver remediation was planned by Sol, implemented by Terra, and independently reviewed by Sol. The review caught a stale resolver-version identity and a raw dot-segment URL normalization bypass before merge. The pattern is simple enough to copy: never let the agent that wrote the code be the only one that reviewed it.

What's next

The 50-candidate workload is sealed at 44/50. The resolver is measurably better than before but not yet at 100%. The four wrong candidates tell me where to look next. And the timed exercise itself is reusable: fresh operator, fresh pack, same setup. That's the point of building it right once.