The release that broke everything (quietly)
Last night a pricing import shipped on Binderdex. The dashboards stayed green. Prices were fresh, the service was up, everything looked fine.
Except a chunk of portfolios were showing zero. Not wrong numbers. Just zero. People who had been tracking collections for months opened the app and saw nothing valued at all.
Here is what happened, and why the monitoring did not catch it.
The join that got too strict
The portfolio valuation joins each holding to a current price. Holdings get stored with a variant label, and prices come in per variant. The release tightened the join to match variants exactly, which is correct for new data. But legacy holdings were historically stored as Normal even when their only available price lived under a different variant lane (Holofoil, Reverse, and so on). The exact match meant those holdings could not join any price at all. Their valuation fell to zero.
No holdings were deleted. No prices were lost. The data was all there. The join just could not reach it.
The tiered fallback
The fix was a three-tier match instead of an exact-only join:
- Exact match: holding variant equals price variant. Use it.
- Compatible fallback: if the holding is legacy
Normaland no exact match exists, fall back to any compatible price lane for the same card and condition, ordered by price then freshness. - Preserve: if no compatible lane exists at all, keep the historical row untouched rather than zeroing it.
Explicit non-Normal variants stay exact-only (you do not want a Holofoil holding silently matching a Normal price). Graded cards are untouched. Both current and historical portfolio paths share the same resolved lane.
Why the alarm stayed green
This is the part that got me. The existing production monitoring measured price freshness: are the prices recent enough? And yes, they were. The prices were perfectly fresh. They just could not be joined to the holdings that needed them.
The alarm was watching the supply side (are prices fresh?) but not the demand side (can holdings actually join a price?). A green freshness alarm does not mean your users see values. It means your prices exist.
The regression guard
So we added a second alarm that measures what actually matters: the fraction of active raw holdings that can successfully join a compatible price lane. Call it valuation coverage. If coverage drops below 95% sustained, it fires.
// What fraction of holdings can actually reach a price?
const coverage = valuedHoldings / totalActiveHoldings;
// Below 95% sustained? Sound the alarm.
if (coverage < 0.95) raiseAlarm();
Plus a deterministic test persona: a legacy Normal holding whose only compatible price lane is a non-Normal variant. It asserts the same non-zero valuation through the web UI, the web API, and the mobile API. If any of them show zero, the test fails before shipping.
The state machine fix (same night)
While we were in there, there was a second bug from the same release. The Start another import button was clearing the visible UI state but the server-side session was stale. So a user who had finished an import and clicked start again would get a dead session instead of a fresh one. The fix was a state guard: only active in-progress states reuse the session. Any terminal outcome starts fresh.
What I am taking away
Your health check measures a proxy. It is always a proxy. The question is whether the proxy is close enough to the thing users actually experience. Price freshness was a fine proxy right up until it was not, because the break happened between the price and the holding, in the join, where no alarm was looking.
Next time you ship a change to a join: ask not just whether the data is fresh, but whether your records can actually reach it.