We turned tracing up to 100%, and it paid for itself within a day
Two nights ago the portfolio screen on Binderdex felt slow for one collector in particular, and nobody could say why. Last night we found out: a single query was eating 8.3 of the 9 seconds that request took. We only know that because earlier in the week we turned tracing up to 100% on every API route and let the data name the culprit.
This post is the whole arc, because the arc is the lesson. Instrument first. The thing the traces point at is never the thing you would have guessed.
Sampling at 100% without blowing the budget
The web server already traced at 10%, and we had zero database spans anywhere. That is not enough to answer which endpoint is slow. The trick that makes full sampling affordable is a sampler that keeps noise out entirely:
// Simplified from our sampler; the decision shape is the point.
function sampleRate(path: string, parentDecision: number): number {
if (path === "/api/health") return 0; // probes are not insights
if (path.startsWith("/api/")) return 1.0; // full fidelity where it counts
if (path.startsWith("/_next/")) return 0; // static assets teach nothing
return parentDecision || 0.1; // pages follow their parent
}One catch worth passing on: the tracing integrations cannot patch the Postgres driver unless you list pg in serverExternalPackages, so the driver loads from node_modules where the hook can wrap it. Skip that and you get request traces with no database children and no error explaining why.
The mobile clock that lost 6.8 days
Turning on mobile tracing produced the best bug of the week. Transactions arrived with durations measured in days. Not milliseconds. Days.
The cause: on React Native 0.86, performance.timeOrigin drifts behind Date.now() by however long the device has slept. One test device had accumulated 6.8 days. The Sentry SDK reads that clock once, lazily, on first use, so every span was anchored to a timestamp from last week and every transaction got rejected as invalid.
The fix is to correct the clock before the SDK ever reads it:
// Before Sentry.init(): pin the perf clock to wall time.
// On this runtime timeOrigin drifts by accumulated device sleep,
// which made every span duration wrong by days.
const drift = Date.now() - performance.timeOrigin;
if (Math.abs(drift) > 1000) {
performance.timeOrigin = Date.now();
}(That is an illustrative rewrite of our guard, but that is the whole idea: measure the drift, pin the clock if it is off by more than a second, and log what you did.) Newer versions of the mobile SDK ship this alignment themselves; we are pinned to an older major until a native build bump, so we carried it ourselves.
Then production shipped no spans at all
The web deploy went out and the dashboard stayed empty. Locally, everything worked. The difference: production runs the Next standalone output, and its file tracer quietly leaves instrumentation.js plus the chunk holding the server config out of the standalone folder. The Postgres instrumentation is loaded dynamically, which file tracers never follow. The Dockerfile now copies those files in explicitly and the build config pulls the pg instrumentation along. If your standalone container loses tracing, this is the class of bug to check first.
The noise tax
Once everything reported, we learned a second lesson fast: 65% of our error volume was development builds, and one device had emitted 69,000 network-error events in 28 days. Tracing everything means silencing everything that is not signal. Events now fire from production builds only, mobile events are rate-limited per key, and a handful of benign browser errors are ignored outright.
What it caught: 8.3 seconds of join
With clean traces flowing, the portfolio screen named its culprit within a day. For a collector with about 1,800 holdings, one query took 8.3 seconds of a 9 second request, joining a 30-million-row prices table once per holding.
The observation that fixed it: two copies of the same card in the same condition share an identical price series. So the query now groups holdings by price key, joins once per unique key, and multiplies by the count on the way out:
// Simplified from the real query path. Join once per unique
// price series, then fan the result back out per holding.
const byKey = groupBy(holdings, (h) => priceKey(h));
const rows = await joinPrices(Object.keys(byKey)); // "900 joins, not "1800
for (const row of rows) {
const count = byKey[row.key].length;
series[i] += row.prices[i] * count;
}Rows returned dropped by about half and warm timings roughly halved with them: the 30-day series went 326ms to 170ms, the 1-year series 2.6s to 1.3s. On top of that, long-period series now cache for 24 hours per user, period, and collection revision. Points before today are immutable once the overnight pipeline writes them, so only the newest point gets overwritten with the live total, and the chart cannot disagree with the headline number. The 1-day and 7-day series stay live.
The takeaway we keep re-learning: you cannot tune what you cannot see, and intuition is a terrible profiler. We would have sworn the slowness was the phone. It was one query joining the same price series 1,800 times.