Our CI lied about which jobs would run. Now it can't.

Our CI lied about which jobs would run. Now it can't.

A monorepo CI pipeline drifts if you let it. Ours had twelve commits of drift baked in: jobs that ran when nothing relevant changed, a lint task that scheduled itself recursively, and coverage processed twice by two lanes that didn't know about each other. Nobody noticed because the pipeline stayed green. Green was the problem.

The pipeline plan became testable data

The rework started with scope flags. One cheap job looks at the changed files and emits plain booleans: statpro, native, ask, web, browser. Every quality job then selects itself from those flags. Selecting is pure logic, so it lives in one tested function (the snippet below is the real selector, verbatim, because tooling like this is copy-paste reusable):

export function selectQualityChecks(scope, candidateChecks = checks) {\n  if (!QUALITY_SCOPES.has(scope)) {\n    throw new Error(`Invalid quality scope: ${scope}`);\n  }\n  if (scope === "all") return [...candidateChecks];\n  if (scope === "etl-runtime") {\n    return candidateChecks.filter(\n      (check) => check.scope === "statpro" && check.id !== "react-doctor-web",\n    );\n  }\n  return candidateChecks.filter((check) => check.scope === scope);\n}

Three lanes (database, web, native) run in parallel off those flags. The part I kept thinking about was the fan-in: what makes it true that the plan you wrote is the plan that runs? Answer: make the plan data and assert it. A small verifier reads every lane's JSON summary, checks each one selected only what its flags allow, cross-checks cache keys against the selected workspaces, and hard-fails the merge otherwise.

Same story, drawn once:

flowchart TD
  F[scope flags from changed files] --> DB[database lane]
  F --> WEB[web + browser lane]
  F --> NATIVE[native lane]
  DB --> V[fan-in verifier asserts the plan]
  WEB --> V
  NATIVE --> V

Make absence loud

The verifier's strictest rule is also its simplest: every boolean output must be exactly true or exactly false; missing counts as broken. An unset flag used to read as "off", silently skipping a lane. A lane that forgets to emit a flag used to read as off, which silently skipped that lane's checks. Now that failure breaks the fan-in job with a message naming the missing output. Same idea, one level deeper: a coverage report that scanned fewer files than exist on disk gets rejected instead of reported. Partial success is a quieter lie.

The last mile of latency was fixtures

Once the lanes were parallel, the slowest thing left was flaky time. A couple of tests asserted against the real clock, so they passed at 11:58 and failed at 12:01. The fix is the standard one and it's worth repeating: generate timestamps once, inject them, and freeze anything that looks like a clock in tests. Two teams bumped package lint config in the same night and the task runner tried to reconcile their recursive schedules; flattening package lint to plain per-package tasks made the conflict (and 73 lines of lockfile) disappear.

Slow tests moved out of the way

Two scale suites were adding minutes to every run, and they only mattered for a few workspaces. Moved each into its own isolated acceptance workflow (with its own fixtures file so the fast suite stops importing the heavy path), then added a dependency-cache workflow that only fires when a lockfile changes.

Overlap the waits

After that, contracts and browser routes were still two sequential blocks of waiting. They don't share state, so they run overlapped now. Last fix: package lint tasks were being scheduled recursively by the task runner. Flattened to plain package-level lint and deleted 73 lines of lockfile that existed only for that.

What I'd tell anyone with a drifting pipeline: green tells you jobs passed, nothing about which jobs ran or why. If your selection logic is YAML, it's not logic. Pull it into a function, test it, and make CI verify its own plan. This night shipped all twelve commits without a single manual re-run (the fan-in gate caught two of my own mistakes along the way, honestly, and that's the point). I don't think the pipeline is done drifting-proof; drift never really ends. But now the drift trips an assertion instead of quietly shrinking what gets tested.