This post is part of an ongoing series on how LaunchDarkly engineers are closing the loop of the AI SDLC—and what we're learning along the way.
Last summer, I shipped what I thought was a routine cleanup to production. It turned out to be a bug. But before the vast majority of users ever saw it, our AI software factory caught it and rolled back my change automatically.
That software factory is something we've been actively building at LaunchDarkly: an AI-powered development pipeline designed to automate how our own code moves from commit to customer. The LaunchDarkly platform is the runtime control layer, governing who sees a change, when traffic expands, and what happens when something goes wrong.
What happened
I did what I thought was a straightforward cleanup. We were migrating frontend callers of an old API to the new version of that API, and I was updating the last remaining caller. I couldn't think of any reason the change would be risky, because I’d already done this cleanup everywhere else. But it was touching code on the flag-targeting page, which is a surface customers use constantly, so I decided to feature flag it just in case.
After I merged and deployed the flagged code to production, our factory automatically started a guarded release. Guarded releases progressively increase traffic to a new variation while monitoring selected metrics for regressions. When one is detected, they can automatically roll back the release.
That’s exactly what happened here: LaunchDarkly users started experiencing more frontend errors only after they saw the “true” variation of my flag. When the guarded release decided it had seen enough evidence to roll things back, 13 of the 243 users exposed to the changed code had seen errors, but 0 of the 250 “control sample” users saw errors, making it a statistically significant result:
Debugging
Debugging was fast. I gave Claude a screenshot of the release dashboard—including the metric that had failed—and it queried Datadog to track down the errors in production. In one shot, it identified the issue: The newer backend API was rejecting requests and returning authorization errors where the old one wasn't.
The root cause was an entitlement check on the new endpoint that was incorrectly blocking requests for some folks. The old endpoint had never had this check, which is why the same UI call worked one way and failed the other.
Rolling out a fix
The fix was a straightforward backend change: removing the incorrect entitlement check from the read path in the new API endpoint.
I restarted the release from earlier. This time, it succeeded:
Takeaways
Guarded releases are powerful, and they can save you when you least expect them to be necessary. But it's important for guarding a change to be easy, so the cognitive cost doesn't discourage folks from making the safe choice.
This thinking has inspired some of the new tools we’ve built internally for our own software factory, which take the most annoying parts of the guarded release process off of the developer’s plate:
- Auto-flagging: Creating a new flag and gating new behavior behind it. In my example, I did this step on my own because we were still working on auto-flagging at the time.
- Auto-releasing: Starting a guarded release in each of our critical environments.
- Auto-cleanup: Cleaning up the flag from the code and archiving the flag.
When a software factory automates this scaffolding, the hard parts of shipping more safely become the default. We're building toward making that available to every engineering team.
