This post is part of an ongoing series on how LaunchDarkly engineers are closing the loop of the AI SDLC—and what we're learning along the way.
I’m an engineer on the Foundation team at LaunchDarkly, and we’re responsible for keeping the platform running. Our entire engineering org has been working hard to close the loop of the AI SDLC, and for my team, that’s involved a careful look at ops triage. We’ve already built a self-reporting feedback loop into our MCP server, so I set out to do something similar for incident response.
We ended up with three Cursor agents that take an ops alert all the way to an open pull request without routine human intervention. An alert lands, it gets investigated, a plan gets written, another agent reviews that plan, and if it holds up, a scoped fix shows up as a PR with the on-call already tagged.
In this post, I'll walk through how it works, but also what didn't: the approaches we threw out, the snags we hit, and what I'd warn you about if you tried to build the same thing.
The problem
Our team gets a steady drip of Datadog monitor alerts and Spinnaker pipeline failures. Before we started this project, most of them played out the same way: Someone would read the alert, click into the logs or the failed execution, decide whether it was real, work out what broke, and either fix it or hand it off. It was high volume, it interrupted whatever you were doing, and in some cases, it also triggered a page from incident.io.
That last part is what made the workflow a good candidate for agents. The trick was keeping them from confidently doing the wrong thing.
The loop
We now have three separate Cursor agents, each with a narrow job. They talk to each other through Jira.
The triage agent watches for incoming alerts. When a Datadog or Spinnaker alert comes in, it digs into the monitor definitions, logs, execution output, and delivery state, then posts a triage summary in the thread. If it decides the alert is a real, actionable incident at medium or high confidence, it writes a remediation plan and opens a Jira ticket in our project.
The validator agent is the gate. It rechecks the evidence and the proposed plan against the original alert, then either approves it or rejects it and kicks it to a human. It does not rubber-stamp anything. It can rewrite a plan or throw it out entirely. If it approves, the ticket moves to the Ready For Development column with an implementation payload attached.
The implementation agent reads the approved plan, makes the scoped change, and opens a PR that links back to the ticket. It grabs the current primary on-call from incident.io to request review, and our existing GitHub automation moves the ticket along after the PR merges.
Every agent also posts back in the original alert thread, so the whole conversation—triage, review, implementation—reads top to bottom in one place.
Why Jira sits in the middle
Getting three agents to reliably pass work to each other was much harder than getting any one of them to do its job well. That’s why the least obvious decision here is the one that matters most. Jira is the source of truth for every handoff, not Slack. The first versions didn't work that way, and that's a really important part of the story.
What I tried first
I started with Slack reactions as the trigger. The triage agent would post a machine-readable handoff block in the thread and then slap a specific emoji on the message to wake up the next agent. It looked great in a demo when I triggered the emoji manually, but in practice, the handoff from machine to machine never took off. The reaction-added trigger didn't fire reliably, and when it didn't fire, the whole chain stalled. There was no ticket, no audit trail, and nothing to retry against. Debugging a handoff that hinges on whether an emoji registered is not something you want to spend your afternoon on.
I also looked at splitting the work across different tools for some of the steps instead of keeping everything in one place. The individual pieces were fine; the seams were the problem. Each tool has its own notion of how it gets triggered and what it hands off, and gluing them together just multiplied the number of fragile trigger points. Wherever one agent came up short, another filled the gap—but those same agents were missing capabilities that the loop actually needed. Neither side was a superset of the other, so no matter how I divided the work, some step ended up on a tool that couldn't do it.
Where I landed
The Jira ticket became the handoff. Triage creates a ticket, and a Jira automation POSTs to the validator. The validator then moves the ticket to Ready For Development, and a second automation POSTs to the implementation agent. State lives in the ticket status and description, which means that Jira provides a durable and auditable record for each handoff; nothing rides on a Slack reaction firing, and if a step fails, the ticket is still there in a known state, ready to retry.
The trade-off is that the trigger logic lives in Jira automation config, not in the agents, so the wiring is spread across two systems. That's a genuine cost. But it's a cost you can see and poke at, which is a lot more than the reaction approach ever gave us.
The infrastructure gotcha: MCPs in a cloud automation
Beyond the trigger mechanism, the other big challenge was giving the cloud automations the tools they need to do their jobs. In a local environment, giving an agent an MCP to run with is pretty straightforward. In cloud environments, it's trickier than it looks.
Getting those connections working meant standing up custom MCP connections for both Datadog and Courier, rather than leaning on a local or default setup. This is easy to underestimate. An agent that behaves perfectly when you run it by hand can be completely inert as a cloud automation just because it can't reach its tools. It’s important to give yourself real time for the connection and auth plumbing, and confirm each connection is actually reachable from the automation before you test any of the agent logic sitting on top of it.
One notable observation: The GitHub connection had to be authorized by a real person, which is why the generated PRs show up under whoever authed the connection, rather than a bot.
Guarding against repeat work
After the happy path worked, the next risk was obvious. If the same error fired five times, the triage agent would cheerfully write five near-identical plans and the implementation agent would open five near-identical PRs. That was wasted review time and burned tokens.
The fix is a deterministic dedupe key built from the stable parts of a failure: source, service, environment, monitor or pipeline name, and a normalized primary error with all the volatile bits stripped out. This approach is designed to assign the same key to two alerts about the same underlying failure. The automations check that key at three points: Triage searches for an open ticket with the same key before filing a new one; the validator does a second pass to catch the race where two alerts both clear triage before either ticket exists; and implementation checks for a PR with the same ticket-key prefix before opening one.
The subtle part is what "done" even means. A closed ticket isn't one thing—it might have been rejected as not actionable, closed as a duplicate, or actually fixed. Lump those together, and you either suppress real recurrences or rerun work a human already turned down. So I split the terminal states. Deliberate rejections go to a Won't Fix column, real fixes land in Done with a merged PR, and duplicates land in Done with a duplicate link. The dedupe check can then branch the right way: Suppress work that's already in flight, escalate a fix that shipped but came back, and never reopen something a person already said no to.
There's a difference between "a human said no" and "a human hasn't looked yet." When an agent can't safely finish, that's the second case, not the first, so it gets its own Waiting column that sits outside the Done states. Keeping them apart matters for dedupe: Lump an escalation into Won't Fix, and the next recurrence gets suppressed as "already declined" when it was really just waiting on a person.
The other half of that is making the ticket legible on its own. Every terminal or escalation move leaves a comment explaining why, not just a status change. A rejection says what failed the review. A duplicate close links the canonical ticket. A Waiting escalation links back to the original alert and spells out what the human should verify and do next. The whole point of Jira as the source of truth falls apart if you have to go hunting through Slack to learn why a ticket is where it is.
Picking the right model for each job
The three agents don't all run on the same model, and that's intentional. Triage and validation both run on a heavier reasoning model, while implementation runs on a cheaper, faster one. The logic follows where the hard thinking actually lives.
Triage has to look at a raw alert and decide whether it's real, what broke, and whether it's worth acting on. Validation has to independently pull that conclusion apart and catch an overconfident or wrong plan before it becomes code. Both are open-ended judgment calls where being wrong is expensive, so they get the model that thinks harder.
Implementation is a different kind of work. By the time a ticket reaches it, the plan is already written, reviewed, and scoped to specific files and repos. The agent isn't deciding what to do—it's carrying out instructions that a stronger model already validated. That plays to exactly what cheaper models are good at: Give a lower-cost model a clear, high-level plan and it can execute reliably without needing the reasoning budget of a frontier model.
This is a small version of a broader token-optimization pattern: Put the expensive reasoning where the ambiguity is, and after the ambiguity is resolved into a concrete plan, hand it down to a cheaper model to carry out. Splitting the work into separate agents is what makes that possible.
Lessons learned
The hard part was the handoffs, not the agents. The reasoning inside each agent was rarely what held us up—getting work reliably passed from one step to the next was. If you're building a multi-agent flow, put your design energy into how work gets handed off and where state lives, not into clever prompts.
That starts with picking a durable source of truth early. Slack reactions felt lightweight and turned out to be fragile and impossible to audit. A boring ticket with a status is a much better foundation for orchestration than an ephemeral signal, exactly because you can inspect it and retry from it.
When the handoffs are solid, the validation gate earns its extra hop. Splitting triage from review means the thing that finds the problem isn't the thing that blesses the fix. The validator catches overconfident triage plans, and since it can rewrite or reject instead of only approving, it's doing real work rather than acting as a checkbox.
Don't overlook cloud tool access—it's its own project. An agent is only as capable as the tools it can actually reach from wherever it runs. Custom connections and auth were prerequisites that stayed invisible right up until the automations couldn't do anything without them.
Design for duplicates from Day 1. The moment something is automated, it runs at machine frequency, and duplicate suppression stops being a nice-to-have. Deciding what makes two failures "the same," and what each terminal state means, is a design question, not an implementation detail you can bolt on later.
On that note, match the model to the work, not to the whole pipeline. The stages where being wrong is expensive get the heavier reasoning model; the stage that just executes an already-validated plan runs on a cheaper, faster one. Splitting the work into separate agents is what makes that possible.
Finally, give your states honest meanings and never overload one. The temptation to reuse Won't Fix for "an agent gave up and needs a human" was real, and it would have silently broken the dedupe logic. Keeping "declined" and "waiting on a person" as separate columns cost almost nothing and kept the board truthful. And whenever an agent moves a ticket to a terminal or waiting state, have it leave a comment saying why—a status change tells you where a ticket is; a comment tells the next human what to do about it.
What's still open
This is early, and I'm keeping a close eye on a few rough edges:
- Scheduled E2E and Playwright failures don't carry much detail in the alert itself; the failing test, the trace, and the screenshots all sit behind the CI run. Until the agents can reach those artifacts, these correctly dead-end at "insufficient evidence." Wiring that up is the next tooling step, and it comes with its own judgment call: Scheduled UI tests are often flaky, and the agent needs to tell a real defect from a transient timeout before it files anything.
- Repo scope is a hard boundary. The implementation agent can only open PRs against repos in its config. Plans that target anything outside that scope stall by design instead of guessing.
- Ambiguous, unsafe, or recurring fixes land in a Waiting column with a comment explaining what needs checking, and the on-call gets pinged. That's on purpose; the goal is to take away the mechanical work, not the judgment.
Right now, the loop starts after something lands in the alert channel, but the bigger goal is to move triage upstream entirely. Picture a preincident gate that watches a spike in errors and decides whether it actually warrants paging on-call, instead of paging first and sorting it out after. This involves the same judgment, applied earlier.
Join the waitlist for early access to LaunchDarkly tools for the AI software factory.
