Sequential Testing for Guarded Releases

We’ve replaced the statistical model used under the hood for Guarded Releases. We changed from using Bayesian statistics to a new Frequentist sequential testing approach. You’ll get the same release experience, but fewer false alarms when we check for regressions repeatedly throughout the rollout.

What’s new

• New statistical model for Guardian: moved from Bayesian to Frequentist sequential testing plus multiple comparisons correction.
• Reduced false positive rate: Dramatic improvement in false positives, especially for percentile metrics.

Why it matters

As customers scale Guardian to run longer and guard more metrics, repeated checks can trigger false alarms, telling you there’s a regression even when things are actually fine.

With sequential testing and multiple comparisons correction, we intentionally make detection less sensitive so that false alarms become rare. The tradeoff in some cases is slower detection, but the outcome is consistency and much higher confidence of real regressions.

Learn more
Docs