How we cut crash → dashboard latency to under 40ms
When a player crashes mid-raid, the clock starts. Every minute that crash sits unseen is a minute closer to a one-star review. Our goal was simple to state and brutal to build: from the moment the SDK fires a fatal, to the moment it lights up your dashboard, in under 40 milliseconds at p50.
The naive pipeline
The first version did the obvious thing — POST the crash, write it to the store, let the dashboard poll. It worked at 100 crashes a minute. It fell over at 100,000. Symbolication blocked ingest, ingest blocked the API, and the whole thing queued into the void.
The trick was to stop treating a crash as one job, and start treating it as several independent ones.
Fingerprint first, symbolicate later
We split the pipeline: raw ingest acknowledges in single-digit milliseconds, fingerprinting groups the crash into a signature synchronously, and symbolication runs async against your uploaded symbols. The dashboard sees the grouped grave instantly; the readable stack trace fills in moments later.
// simplified ingest hot-pathconst signature = fingerprint(crash); // sync, ~2msawait queue.publish('symbolicate', crash); // asyncreturn { signature, ingest_ms: 38 };
The result: studios watching a Friday-night launch see the spike form in real time, not in Monday's standup.