Book 1 is in its hard-edit pass, and the plan was to let a panel of AI reviewers help decide what gets cut. Four reviewer lenses, every chapter, flags and scores. Before handing them that much say over 146,000 words, I asked a question that should have been day one: how good are these reviewers, actually?
An AI reviewer that says "this chapter works" tells you nothing if it says that about every chapter. One that invents problems is worse, because you'll cut good scenes to please it. So the panel got a pop quiz with an answer key.
Twelve planted flaws, six chapters, blind controls
The setup is deliberately mean. Six chapters of Book 1 carry twelve planted flaws, and every flaw comes with an answer key: what it is, where it sits, and what a correct review should say about it. Some chapters got two broken variants, some stayed clean as controls. Then one more layer: six chapters from a published novel I admire, character names swapped, dropped in unmarked. If a reviewer flags flaws in professionally edited prose, the bench is measuring noise.
This week the bench finished its first full run: 288 reviews, all scored against the key. The GPT family went through codex with two runs per item (192 reviews) so we could check consistency, and the Claude subagent lane contributed 96 more. bench.py builds the items, farms out the reviews, and grades the catches.
The scoring rule that nearly ruined it
The first scorer was too generous. A catch counted when the reviewer quoted a line near the flaw's location. That broke fast: anchor lines also appear in the clean chapters, so a reviewer could score a "catch" in a spotless chapter by quoting the same sentence. The fix sounds small and it changed every number: a catch now also has to name the problem, matched per flaw by keyword.
# the rule, simplified from bench.py
def counts_as_catch(review, flaw):
if not near_anchor(review.quote, flaw.anchor_lines):
return False
return bool(flaw.keywords & tokenize(review.explanation))Pointing near the spot is necessary. Naming the problem is what makes it evidence.
One negative result I'm keeping
The most useful finding cost almost nothing. I wrote a "setup lens", a reviewer prompted to hunt specifically for broken setup and payoff, and tested it on the bench. It missed both deleted-link flaws, on both model families. It does not join the pipeline, and the bench README records why it failed. A reviewer you haven't benched is a reviewer you're imagining.
The honest caveats: six chapters and twelve flaws make this a lab test, more than a census, and one model ran its judge a single time so a re-run could wobble. But it already changed how I read the panel's output on the real edit. When a reviewer flags something now, the question is whether it caught the planted version of that flaw class. And when it praises a chapter, I remember it praises clean chapters at the same rate.
If you use models to review anything of yours (code, contracts, prose), the pattern transfers cheaply. Plant a dozen flaws you already know about. Keep clean controls. Score against the key before you trust a single review of the real thing. Grade the grader before it grades you.