Roll out safely: catch a bad build before everyone gets it

You push a new build to a slice of players. Most are still on the previous one, so every game-wide number mostly describes the old build. If the new one crashes three times as often, those numbers may barely move, and you find out when it reaches everyone. Release health answers one question while the slice is still small: did this build hurt anyone?

Every build is judged against the others

On the Releases screen, each build's crash-free sessions rate is compared with the session-weighted rate of every other build that has at least 30 sessions. The build under test is left out of its own baseline, so a canary is measured against the builds around it rather than diluted by them. A failing build shows red at any adoption level and sorts to the top, and the banner leads with the worst result. Each row carries one of four badges:

• BURNING (red): at least 2 points below your fleet's crash-free rate and measurably so, or under 90% crash-free sessions on its own.
• BELOW FLEET (amber): at least half a point below and measurably so, or under 95% crash-free sessions on its own.
• LEVEL (green): measured, and not measurably below the fleet.
• NOT MEASURABLE (grey): under 30 sessions, or no other build with enough sessions to compare against. The row says no verdict is drawn, and it never turns green by default.

“Measurably” is a real test. At 30 sessions one crash moves the rate by more than three points, so a raw gap would flag healthy builds constantly. Tombstack takes a 95% confidence interval (Wilson score) around the build's rate and raises the alarm only when the top of that interval is still below the fleet. A build past the line on too few sessions is labelled not measurably below your fleet, with how high its true rate could be.

Each row also shows crash-free players and the build's share of sessions over the last 24 hours, so you can see how far the rollout has gone. Crash-free players does not drive the verdict, but it catches the bug that hits many players once each and barely dents crash-free sessions.

Alerts that reach someone

The Release risk rule is built into the Alerts screen and checked hourly. It runs the same grading over the last 30 days, across all environments, and sends the row's own sentence, so the alert and the page cannot disagree. It pages for BURNING builds only, and says how many BELOW FLEET builds it left on the page.

• Email when no channel takes it. Critical alerts (Crash spike, Signature regression, Player impact, Release risk) that no webhook accepts are emailed to the studio's owners and admins.
• Alerts delivered. The Alerts screen lists what actually went out, when, and on which channel, recorded at send time rather than rebuilt from today's data.
• Crash spike counts crashes. Hard crashes and unclean shutdowns count; a burst of handled exceptions no longer pages you as a spike.
• Cooldowns per rule. By default a rule waits 6 hours before alerting again about the same game, signature or build. Add a cooldown rule to set it anywhere from 1 hour to 7 days. A withheld alert is shown as withheld.

Into Discord, Slack or Teams

Webhook endpoints live on your studio's Integrations page, open to owners and admins and linked from the Alerts screen. A studio can have up to 25, and each one chooses:

• Format: Discord, Slack, Microsoft Teams or signed JSON. Auto picks from the URL.
• Events: every alert type, issue status changes, and Player bug report received. Reports made before you subscribed are not replayed.
• Scope: one game or all of them, a minimum severity, and optional environments.

JSON deliveries are signed with a per-endpoint secret you can reveal and rotate. A delivery that fails on a network error, a rate limit or a server error is retried after 1 min, 5 min, 30 min, 2 h and 6 h; after 20 failed deliveries in a row the endpoint is switched off until you re-enable it. Deliveries keeps every attempt for 30 days with its status and response time, and any logged event can be redelivered. Payloads and signature checks are in the webhooks docs.

Ask from CI or an AI assistant

Release health is in the read API and in the hosted MCP server as get_release_health. Both run the same grading and return each build's verdict, its session count and the page's own sentence. A CI step can hold a rollout while the new build is burning:

# Hold the rollout while this build is BURNING
VERDICT=$(curl -s "https://tombstack.com/api/v1/read/releases?gameId=$GAME_ID&days=30" \
  -H "Authorization: Bearer $TOMBSTACK_STUDIO_KEY" \
  | jq -r --arg b "$BUILD" '.data.releases[] | select(.buildVersion == $b) | .crashFree.verdict')
echo "$BUILD: ${VERDICT:-not listed}"   # critical | watch | healthy | unmeasured
[ "$VERDICT" != "critical" ] || exit 1

Treat unmeasured as “not yet”, never as a pass. The read API takes a studio key or a game token granted the read scope (authentication). Through MCP, an assistant asked “is the new build safe to widen?” reads the same verdict.

How to try it

1. Keep session heartbeats on. The SDK sends them automatically, and they are the denominator behind every crash-free rate. Give each build its own version and label its environment.
2. Open Releases during your next rollout and read the banner first.
3. Connect a channel. On Integrations, choose Add endpoint, paste a Discord, Slack or Teams webhook URL, keep Release risk and Crash spike ticked, then Send test and check Deliveries.
4. Tune cooldowns on the Alerts screen if 6 hours does not suit your team.
5. Gate the next stage with the read API in CI, or ask your assistant, before you widen the rollout.

Once an alert fires, the launch-day runbook covers scoping the incident and deciding your rollback trigger ahead of time.