Key takeaways
- Self-healing AI agents run a recovery plan your team defines in advance. They don't retrain themselves or decide what counts as fixed.
- The self-healing loop has four steps: detect, diagnose, recover (roll back, swap models, or reroute), and fix.
- Error rates catch infrastructure failures like timeouts, but reasoning failures (the call succeeds, the answer is wrong) need evals.
- Ship each corrected prompt or model as a guarded rollout that can roll back automatically if a monitored metric regresses.
Your support agent might have passed every eval in staging, but in production it’s drafting refunds for the wrong amount. And, there’s no log of any failures. Each model call returned a response, and every tool call completed—you only found out about the error because of a support ticket from an upset customer.
Self-healing lets you find out from a metric instead, so you can start recovering when the numbers move, instead of when a customer complains.
What are self-healing AI agents?
A self-healing AI agent is an agent under runtime control: it's wired into a recovery loop your team defines before anything breaks. Prompts and model choices live in config outside the code. Metrics or eval scores detect when quality degrades, and if they do, a predefined action runs:
- Roll back to the last good config
- Swap to a fallback model
- Reroute the request
That definition is deliberately narrow: the agent acts only within the room your team gives it.
Researchers Jeong and Shin describe a self-healing framework that integrates failure detection, reliability assessment, and automated recovery through adaptive replanning and corrective prompting. The production layer adds a few extra variables:
- Who is allowed to change the agent
- How to ship the fix
- How you undo it
Application code already uses this idea. Self-healing software with release observability applies it to releases that watch their own metrics and roll back to the last good state. Agents need it more, though, because the same inputs can produce different outputs, and quality can drop without anyone changing a line of code. The runtime controls cheatsheet walks through setting up self-healing for application code and for AI agents.
Note: You'll also see "self-healing" used for test automation that repairs broken selectors and CI pipelines that retry failed jobs. Those are different problems; this piece covers agents in production.
Common AI agent failures in production
Agents fail in two ways, and each needs its own detector. Infrastructure failures tend to be loud. The model provider times out, returns a 429, or throws a 5xx, and your error rate usually shows it right away. You know about these ones.
Reasoning failures, though, often go unnoticed. The call succeeds but the answer is wrong. The VIGIL paper calls these soft failures: the logs report success while the behavior is off. Error rates won't catch them, but evals can.
Reasoning failures (the call succeeds, the answer is wrong) | Infrastructure failures (the call fails) |
|---|---|
Hallucinated facts, prices, or policy details | Model provider timeouts |
Tool called with bad parameters, like the wrong account ID or date range | Rate limits (429s) |
Output drifts off-tone or out of format and breaks a downstream parser | 5xx errors from the provider |
Agent reports a task as done when it isn't | Out-of-memory errors on self-hosted model servers |
One agent's bad output becomes the next agent's input | Runaway token usage from retry loops |
Caught by: Evals and judge scores | Caught by: Error rates, latency, status codes |
Many of these failures happen without anyone on your team changing anything. It’s a provider shipping a new model version or a tool returning slightly different data. However, sometimes your team does change something, and a prompt edit can change the agent’s behavior as much as a code release, but it rarely goes through the same tests and review. That’s one reason why AI deployments break standard CI/CD. And since model output isn't deterministic, a change can pass staging and still fail on real traffic.
The usual response, find it, fix it, redeploy, was built for software that holds still. An agent doesn't. It keeps running on real traffic while the fix waits on a deploy, and customers feel the problem for as long as that takes. Self-healing moves the response to where the agent runs.
Learn how to catch and revert AI failures in production with our example walkthrough.
How do self-healing agents work?
Self-healing agents work in four steps: detect, diagnose, recover, and fix.
- Detect. Traces and eval scores show that errors are rising or output quality is dropping.
- Diagnose. You narrow the problem to the prompt, model, tool, or variation responsible.
- Recover. A predefined response runs: roll back, swap models, or reroute. Then it alerts someone.
- Fix. Your team writes a corrected prompt or config, and it goes out behind a guarded rollout that can roll back automatically if a monitored metric regresses.
The first three steps are all about handling the incident. They focus on speed, control, and damage mitigation. The fourth (and last) step is about the actual fix, and it's the easiest one to skip.
1. Detect: Catch failures with traces and evals
You catch failures by monitoring AI agents in production. Trace every model call. With LLM observability, each span records:
- Model name
- Prompt and response
- Token usage
- Request duration
- Provider
That's enough details to catch most infrastructure failures and spot cost or latency drifting. That covers uptime and performance, but not correctness. To know whether the answer was right, you need evals.
How to evaluate AI agents in production
Score a sample of live responses with an LLM judge. Online evaluations give each evaluated response a score from 0.0 to 1.0 (where higher is better). There are built-in judges for accuracy, relevance, and toxicity, and you can write custom judges for what matters to your agent, like whether a refund amount matches the order. Judges can run asynchronously, so they don't have to slow the response. Each evaluated response is an extra call to your model provider, so your sampling rate is also a cost decision. At a 10% sample, 10,000 requests a day add about 1,000 judge calls. Sample too little, though, and judge-score regressions take longer to reach statistical significance, so a guarded rollout catches them later.
In Python, a config with an attached judge returns its evaluations alongside the response, as this judge evaluation example shows:
With LLM-as-a-judge scoring on completion-mode configs, each score is stored as a metric, so a quality drop can trigger the same guardrails as an error spike.
You can also turn signals already in your traces into metrics. Trace metrics build a metric from a query over OpenTelemetry spans, so you can watch something like failed tool calls without re-instrumenting them as metric events.
Pick these metrics before anything breaks. Self-healing needs a target you can measure and a signal that shows when something breaks. Set an alert on each one so your team hears about drift before a threshold fires.
2. Diagnose: Trace the failure to its root cause
Diagnosis finds which prompt, model, tool, or variation caused the failure, so you know what to roll back.
For example, detection might tell you about a drop in refund accuracy, but there could be a number of underlying causes:
- Prompt edit
- Model provider update
- Tool that changed its response format
- One variation being served to 10% of users
Each one has a different fix, so treat root cause analysis as its own step. Skip this step and you risk rolling back the wrong thing.
One of the quickest ways to narrow it down is to tie every trace to the config that produced it. With LLM observability in the docs, you can filter traces by config key or variation, as long as spans are exported through the observability SDK. If you use your own tracing pipeline, attach the config key and the evaluated variation to the root span yourself. You can also query LLM span attributes directly.
The failure pattern will usually lead you to the cause:
What you see | Likely cause | Where to look |
|---|---|---|
Judge scores drop on one variation only | The prompt or model in that variation | Compare variations side by side, or run an experiment on them |
Scores drop on every variation at once | A provider model update or a change in tool data | Traces grouped by model and provider |
Latency rises on one model | Provider slowdown | Request duration by model |
Tool call errors rise | Changed tool schema or bad parameters | Tool spans and their inputs |
Vega, the LaunchDarkly AI debugging assistant, can do much of this for you. It reads logs, traces, and errors, identifies likely root causes, and often ties them to a recent flag or code change. When a guarded rollout rolls back automatically, Vega starts investigating right away.
A robust AI observability platform can tie traces to the config behind them when spans go through the observability SDK, so finding what changed is a filter instead of a search.
3. Recover: Rollback, fallback models, and rerouting
Recovery happens at two layers:
- Request time
- Config level
Keep them separate, because they're owned by different things.
Request time: Your application code
Retries, timeouts, failing over to a backup model, replanning, and checkpointing all run per call, inside your app. They handle infrastructure failures as they happen. This is application code you write and test.
Config level: Every request after this one.
When the problem is the prompt, model, or path, you change what the config serves, and the next requests pick it up without a redeploy. That can mean rolling back to the last good variation, swapping to a fallback model, or using targeting rules to send some (or all) traffic down a different agent path.
The runtime controls cheatsheet has a step for this called "Configure automated rerouting." You decide in advance what happens when eval scores degrade:
- Swap to a fallback model
- Roll back to a good prompt configuration
- Reroute the request to a different agent path
Adaptive triggers handle the swap: a trigger changes the config's variation when a set number of events lands in a time window, then alerts the people you name. For a path you want off entirely, a kill-switch flag sends users to a graceful fallback without a redeploy.
When a threshold is crossed, that response runs and notifies the on-call engineer, who reviews what happened instead of starting the fix from scratch.
Model swaps only work as a recovery step if model choice already lives in config. At Poka, it’s a flag. “We have a flag for which model we’re going to be using,” says Edmund Lam, Staff Software Developer. That’s how the team was able to start testing Claude 3.5 Sonnet just one day after it was made available.
4. Fix: Ship it as a guarded release
A rollback stops the damage, but it doesn't permanently fix anything. Someone still has to write a corrected prompt or pick a better model. And that change can be risky: it ships under pressure, to an already misbehaving agent, and it can introduce a brand-new failure despite fixing the old one.
So ship the fix like any other production change.
In LaunchDarkly, AgentControl, the control plane for agents in production, keeps prompts, models, and parameters in configs outside your code, so changes reach production without a redeploy. Each version of the agent's behavior is a variation. A corrective prompt (or one you tuned from what the traces showed) is a new variation your team writes. Try the corrected prompt in the playground before you ship it.
LaunchDarkly doesn't write it for you. It controls who gets it, and how fast. After each incident, review what the traces showed and fold it into the next variation and your thresholds. That's how the loop improves without the agent retraining itself.
Release that variation with a guarded rollout, and the traffic moves to the new variation in steps while LaunchDarkly compares it with the original on the metrics you chose:
- Error rate
- Latency
- Judge scores (for completion-mode configs)
Regression detection flags a regression when one of those metrics shows a statistically significant negative impact compared with the original. When it finds one, you can configure LaunchDarkly to (a) "Notify you and the flag's maintainer" or (b) "Automatically roll back the release." When creating guarded rollouts, that's the Automatic rollback box next to each metric.
How to roll back a bad prompt or model change
Serve the previous variation. If the change went out behind a guarded rollout with automatic rollback turned on, LaunchDarkly does that when a monitored metric regresses. If it didn't, point the targeting back at the last good variation. Either way, the next requests can pick up the old behavior without a redeploy.
Relay Network moved model selection and prompts into config and reduced prompt iteration time from full-day deployments to seconds.
Can AI agents fix themselves without human oversight?
No, and they shouldn't. Automation handles the response, but people handle the decisions.
Your team picks the metrics, thresholds, fallback variations, and who is allowed to change any of them. Automation acts inside those limits and reports what it did. In practice, that’s a short list of pre-approved moves: your team writes the list ahead of time, and the automation runs whichever one the numbers call for. Your team can also make these changes from an AI coding assistant through the LaunchDarkly MCP server.
Teams that let agents take on more have to decide in advance what happens when quality dips. When it does, the automation or rollout carries out that decision within the scope your team set, and the on-call engineer gets paged to review what it did.
Handled automatically | Escalated to a person |
|---|---|
Pausing or rolling back a guarded rollout when a monitored metric regresses | Deciding whether the change was bad or the metric was |
Serving a fallback model or path your team defined | Writing the corrected prompt or choosing a new model |
Retrying or failing over on timeouts and rate limits (in your code) | Changing thresholds, fallbacks, or who can edit configs |
Notifying the on-call engineer and the config maintainer | The same config triggering rollbacks again and again |
Recording what changed, when, and to which variation | Any failure with no predefined response |
Keep a record of every change, automated or not:
- Which config
- Which variation
- Who changed it
- When it changed
AI agent governance pairs that change history with role-based access and approval workflows. Page people when automation acts or fails to act, not every time a metric twitches. Otherwise, on-call learns to ignore the alerts. And test the loop before you fully trust it: in a non-production environment, inject errors or degrade eval scores, then confirm the threshold is detected, the remediation runs, and your app picks up the change without a redeploy.
Self-healing in multi-agent systems
Self-healing in a multi-agent system means isolating and rolling back the one agent that failed, without touching the rest. In a multi-agent system, one agent’s bad output becomes the next agent’s input. This gets passed on until the final response is wrong, but the cause could be three handoffs back.
Two things can help prevent this from happening in multi-agent systems.
- A supervisor agent orchestrates the workflow and can route around a specialist that’s failing.
- Giving each agent its own config means you can roll back one agent without touching the rest.
Agent graphs model this directly. Each node is an agent config, each edge is a handoff, and metrics show per node, so you know which agent to check first. For example, in our agent graphs tutorial, a security agent slows to five seconds per call. The fix is just a model swap in that agent’s config, with no code changes or deployment.
Govern the fixes and the failures
Self-healing is only as safe as the controls you put around the fix. Detection and rollback get you through the incident. Shipping the corrected prompt as a guarded, reversible change keeps the fix from starting a new one.
Start with one agent:
- Move its prompt and model selection into a config.
- Attach a judge and pick two metrics you'll act on, like accuracy and p95 latency.
- Define the fallback: the last good variation, a backup model, or a simpler path.
- Break it on purpose outside production and watch the loop run.
Our AI iteration loop tutorial walks through the build side with LangGraph, from kill-switch flags to targeted variations. For code changes, Release Autopilot runs the same loop, automating flagging, rollouts, rollbacks, and cleanup. Some Release Autopilot capabilities are currently in beta.
Learn more about how LaunchDarkly helps enable self-healing systems.
FAQs
Are retries and model fallbacks enough to make an AI agent self-healing?
No. Retries and model fallbacks only handle infrastructure failures, one call at a time. They run in your application code and respond to timeouts and rate limits, but they can't fix a bad prompt. Config-level recovery can: rolling back a variation, swapping models, or rerouting changes every following request without a redeploy.
What can you do when a model provider update breaks your AI agent?
Swap to a fallback model or reroute traffic, because rolling back your own config won't undo a change the provider shipped. The usual sign is judge scores dropping on every variation at once, not just one. The swap only works without a redeploy if model choice already lives in config and your team picked the backup ahead of time.
How do you stop a self-healing AI agent from rolling back a good change?
Compare the new variation against the original and act only on a statistically significant regression, not a single bad reading. Guarded rollouts with regression detection work this way, so ordinary metric noise is less likely to trigger a rollback. If one config keeps rolling back, have a person check whether the change was bad or the metric was.
Is self-healing for AI agents the same as self-healing test automation?
No. Self-healing test automation fixes broken tests, not agents in production. It repairs broken selectors, and self-healing CI pipelines retry failed jobs. Self-healing AI agents deal with output quality that drops in production, often without anyone changing a line of code.





