This post is part of an ongoing series on how LaunchDarkly engineers are closing the loop of the AI SDLC—and what we're learning along the way.
At LaunchDarkly, we’ve been thinking a lot about which parts of the SDLC can be safely automated, and which ones still need a human in the loop. My own answer to that question started as an internal hackathon project. It’s an autonomous QA agent that drives a real Chromium browser against a web app, finds bugs the way a frustrated customer would, and files Jira tickets with screenshots, video, console logs, and network traces. I call it “Thousand Cuts.”
What came next was inspired by an internal Slack channel where DarkLaunchers post the small bugs they trip over in our own product. They’re the kind of issues that most tests wouldn’t flag, but they still add up. The channel works, but it's manual, biased toward the few screens people happen to use personally, and it goes quiet whenever everyone is busy shipping. Noticing papercuts can't be anyone's full-time job, but it can easily be an agent's.
So I put Thousand Cuts on a schedule and set three more agents to work behind it. The result is an (almost) closed loop that looks a lot like a private software factory: Agents find what's broken in production, decide what's worth fixing, and build a fix. I decide what actually ships.
In the first week, that loop filed 118 tickets and opened 23 pull requests.
The factory line
The runtime underneath all of this is Cursor Automations, which makes it easy to run remote agents with access to our code and appropriate tooling. But what matters more than the runtime itself is the coordination mechanism, which is the Jira board. The agents talk to each other only through status transitions, so none of them knows the others exist, and adding a new station to the line means writing a new prompt. Every agent also narrates its work into an internal Slack channel, which tells me which tickets to read and where the agents got their own reasoning wrong.
Station one is the crawlers. Two automations run daily across different surfaces of our platform. Coverage comes from randomness. Each run picks a random start URL and a random mode: functional bugs, console errors, usability heuristics, or visual polish. The crawlers have no access to our code, so if the code says one thing and the screen says another, the screen wins. One model drives the command-line tooling; another drives the browser itself.
Station two is the classifier. It only reads the board and can't open a PR. On each run, it deduplicates the queue before promoting anything. It’s designed to not promote a ticket it's about to archive as a duplicate, and its standing instruction is to bias toward inaction. When it isn't confident, it leaves a ticket where it is rather than guessing.
Station three is the fixer. It picks up one ticket an hour (oldest first), reproduces the bug, writes the fix and tests, and opens the PR, tagging me for review. It can only move the ticket after the PR exists. One PR an hour is a throttle I chose, and even then, it’s been tough to keep up with.
Why is this different from code review tools?
Most AI-powered code review tools anchor to a change. They ask whether a new diff introduces a bug, or whether a change broke existing behavior. Thousand Cuts asks a different question: What's broken in production right now that nobody has noticed?
Most of what it finds aren't regressions. They’re likely defects customers may have hit for years. One ticket noticed that the browser tab on our Contexts page read "Users" for a couple of seconds on every load. That was the feature's name before we renamed it years ago. The web app had the right title all along, but the server that renders the page shell still had the old one. No merge-time tool catches that, because there's no merge to attach it to.
Every one of these bugs is individually rational to ignore. The fix is often one line, but the discovery needs someone opening a screen they had no reason to open, at a width they had no reason to try. Tests don't help, because you have to imagine the assertion before you can write it. So each one is correctly deprioritized, forever, and the sum is what a customer experiences.
Early results
In the first sixteen days, the crawlers filed hundreds of tickets. The classifier flagged almost half as duplicates and archived them, which is the cost of a crawler with no memory of our architecture.
Of the distinct bugs that remained, I rejected about 10% as not worth fixing. The fixer has turned almost 40% into PRs, and over a quarter of those are live in production, each reviewed and approved by me and a second engineer.
That leaves about half of the remaining tickets waiting on a decision, not on code, and a dozen open PRs waiting on my review, not on the fixer.
Anatomy of a machine-filed ticket
People are skeptical of these tickets until they read one. Here's the real body of a bug filed by the crawler, promoted by the classifier, fixed in one PR, and merged by me:
Title: Role description text breaks mid-word on the Roles page ("applicatio / n")
Severity: Low
Category: Visual
Repro steps:
1. Navigate to https://app.launchdarkly.com/settings/roles
2. Locate the "Developer - The main web application" role row
3. Observe the description breaks the word "application" with a lonely "n" on its own line
Expected: Role descriptions should wrap on whitespace, not break in the middle of words.
Actual: Shows “Grants developer permissions to our main web applicatio” then a broken line starting with just “n”. Likely `overflow-wrap: anywhere` or `word-break: break-all` on the description text.
Environment:
URL: https://app.launchdarkly.com/settings/roles
Viewport: 1280x800
UA: HeadlessChrome/148.0.7778.96
Run: run-20260722230038-1774d1Attached automatically were a full-page screenshot, a 15-second video clip trimmed around the finding, the network log for the preceding two minutes, and 40 lines of console output. That console output independently surfaced another bug in the same run: a storm of failed avatar image requests, one for every member without a profile picture, on every page load.
The fix was one line of CSS. It merged. All of this took me about two minutes of reading.
What went right
A few moments from that first week made me trust the system more.
When faced with a ticket it couldn't close, the fixer handled it correctly. One ticket asked for four environment tabs to fit inside a narrow viewport. The fixer reproduced it, measured it, and declined: At that width, with the sidebar open, the visible area is over 400 pixels narrower than the row needs, so it overflows by design, and no scroll offset shows all four tabs. Its note back to me was direct: "It's a product or design call, not a patch. Staying in Ready to build, blocked on that decision."
That refusal was written into the prompt, which tells it, "If the issue is ambiguous or missing critical detail, do not guess." Agents don't arrive with taste, but writing the refusal path down explicitly was the most useful paragraph in the whole system.
When the fixer ran out of work, it found something to do. With that ticket unbuildable and nothing else promoted, the fixer went 27 consecutive hours with nothing to pick up. Rather than sit idle, it swept a defect class per hour across our codebase. In its best hour, it fixed 114 tests for a 27-line diff. Every one of those runs ended the same way: no PR, because there was no ticket to attach it to. The bottleneck was my process. The agents had plenty of capacity.
I wasn't the only one building this. Ari Salem, an engineer on our Foundation team, has a companion post in this series, “Building a self-driving ops triage loop,” that he was writing the same week I was building this. His system watches for ops alerts instead of UI papercuts: A triage agent investigates and files a ticket with a plan, a validator agent approves, rewrites, or rejects that plan, and on approval, an implementation agent opens the PR. We built these independently, without knowing the other existed.
His design beats mine in one way: The agent that diagnoses a bug isn't the same agent that blesses the fix. I'm stealing that, along with his deterministic dedupe key, which would fix my duplicates. His loop is three prompts and two Jira rules; mine is four prompts and a schedule, and the pattern was cheap enough to appear spontaneously inside one company in a single week.
Both of us stop at the open PR, because a release layer isn't something one person builds in an afternoon.
The other half of the factory
Everything up to the merge was easy: four prompts and a scheduler. Everything after the merge is coordination, and that's where the work is. Any engineering director could reproduce the left-hand column below this week. The right-hand column is a different problem.
From idea to merged PR: Four prompts and a scheduler | From merged PR to safely released: Needs a platform |
|---|---|
Finding the bug from a customer's viewpoint, with video. | Automatically flag changes when appropriate. This is decided per change, and it has to be right. |
Filing it so nobody reinvestigates. | What is the release plan? Involves stages, percentages, and guardrails defined by a human. |
Triaging it, including declining when a product decision is buried inside. | Automatically release it everywhere: staging, production, EU, Federal. |
Fixing it with tests that fail against main first. | Where is it live? “Merged” does not mean “deployed,” “deployed” does not mean “exposed,” and “exposed in production” does not mean “exposed in Federal.” |
Refusing when the acceptance criteria are impossible. | Automatically clean up. The flag dies when the change becomes permanent. |
Everything in the right column is about coordinating a change after it exists. The one that resonates most with me is automatic release. A flag that never got turned on in Federal is indistinguishable from a feature we never shipped, and today that happens because promotion is a human walking the same clicks through four environments from memory. Agents now produce changes faster than anyone can shepherd them, and none of it is easy to automate.
Where LaunchDarkly comes in
The team at LaunchDarkly has been building the part of the software factory that no one can build alone. It watches PRs, classifies each change, proposes a flag and a release plan, and on human approval, creates the flag and wires it into the branch under an accountable human owner. The human stays in control the whole time through PR comments: Approve the plan, argue with it, or claim ownership of a bot-authored PR. That last one is exactly what my fixer needs.
My half of the software factory makes small, flaggable changes. The other half ships them safely, unattended.
Looking ahead
This is one week in, and there's plenty I'd still change. For instance, I'd like to:
- Tune deduplication, since having about a third of all total tickets be duplicates means the crawler still has no architectural context of its own.
- Let the fixer create its own tickets, so idle branches like the 27-hour stretch above don't go to waste.
- Crawl more of the app, prioritized by customer traffic rather than the screens I happen to know best.
- Partner with UX on what "good" looks like, since our internal design guidance was written for agents that can read source code, and the crawler only has a browser to work with.
- Wire the fixer into the software factory itself, so it can create the flag and release plan in the same run as the code.
The autonomous half was easy, and it gets easier every month. The coordination half is hard and holds all the remaining risk. But it’s also the part LaunchDarkly was built to solve from Day 1.
