AI agent observability and monitoring: Identify, intervene, verify

AI agent observability and monitoring tracks both how an agent ran and whether its behavior was acceptable for the task and users involved.

Service health covers only part of the job. An agent can return a fluent answer even when a required tool never runs, while infrastructure metrics remain green.

That kind of failure is routine for agents in production, and it rarely starts with a code change. A model checkpoint updates on the provider's schedule, a tool dependency shifts, or a prompt edit changes behavior for one cohort. Your customers feel it first. And the standard response of find it, fix it, redeploy is too slow for a system that never stops answering.

Monitoring reveals that behavior changed. Traces and logs reconstruct the run, while evaluations determine whether the result met the task’s requirements. But a signal is only useful when it connects to a control path prepared in advance. That path needs a defined target, a known blast radius, and enough configuration and cohort context to fire safely.

This doesn’t mean that whatever spots the problem should also fix it. Detection and remediation can (and often should) live in different systems. The requirement is that they share the run, configuration, and population context, so an operator can act on the evidence without guessing who was affected or what a response would touch.

AI agent observability should support the full loop of identification, intervention, and verification, from isolating the affected behavior to confirming recovery for the same task and population. This is runtime control in practice: you observe a live agent and iterate on it without a redeploy. The evidence must give the team enough context to choose a bounded response and determine whether it worked. The loop doesn’t end at verification. The team uses what they learned from confirming recovery to sharpen the next detection.

This is where most observability stacks stop. A standalone tracing or eval tool can show you that a run failed and which step failed. It holds no variation to revert, no cohort to scope the revert to, and no way to change the agent’s behavior short of a redeploy. The signal arrives, and the response still waits on a person and a deploy cycle. Runtime control closes that gap by keeping the control path next to the evidence.

How to monitor AI agents in production

Start with the common question an on-call engineer will ask during an incident: “Which runs changed, for whom, and under which configuration?” The answer depends on the traces that connect each run to the specific configuration and variation received.

An AgentControl config stores the prompts, instructions, and model settings that control how an application uses an LLM outside application code. The trace context below uses the config key and evaluated variation, which helps you identify which configured behavior the request received.

Run and configuration context

Use the root agent span to represent the run. Add the conversation identifier, environment, AgentControl config key, evaluated variation, and its version, where available, to the trace context. Include only the context attributes needed to distinguish the affected population. For a multi-turn interaction, give every span you want grouped into the conversation the same gen_ai.conversation.id. That way, the conversation can remain connected when it crosses traces.

Then, under that root, create child spans for model calls, retrieval, tool execution, agent nodes, and handoffs. Where the agent records its plan or tool choices, capture them as span events so you can trace its decision path, not just its calls. Correlate application logs with the same trace or run identifier. The shared identifiers let you move from a metric change to the execution sequence that produced it.

OpenTelemetry’s generative AI conventions provide a common shape for this instrumentation, including an invoke_agent span with child chat and execute_tool spans. The conventions remain under active development, so treat them as an evolving contract. Make sure to preserve the stable operational context your team needs, even if the individual attribute names change. Captured content from prompts or responses, including tool results, may also help with debugging.

Useful correlation does not require full content capture. When you do capture prompts or responses, redact personal data before it leaves your application. Your privacy and security requirements should determine what you record and how long you retain it. IDs, configuration context, tool outcomes, and domain-specific measures can often narrow an incident without retaining every message.

Operational and outcome measures

Operational metrics show how the system ran, including:

• Latency
• Errors and retries
• Token use and cost
• Tool success

A task-specific quality evaluation can show whether the result was acceptable. Support agents can be measured on task completion. Research agents can be gauged against a grounded-answer score plus a deterministic citation check, and coding agents on tests or policy checks. Choose a measure that shows exactly what became unacceptable.

To monitor hallucination or drift, define the failure you want to detect. Pair a judge or deterministic measure with a baseline and threshold. Keep the task and population comparable for drift checks so a fluctuation in traffic does not look like a change in agent behavior.

Within LaunchDarkly, metrics, traces, and evaluations stem from separate capabilities with differing instrumentation requirements. Understanding those distinctions helps each signal carry the right context during an incident.

AgentControl’s Monitoring tab displays metrics recorded by your application through a LaunchDarkly AI SDK. Those can include latency and token use, along with cost and evaluation scores. The Monitoring tab shows results by variation for the selected config. Your application still has to record them. Monitoring does not discover uninstrumented activity. For a cross-config view of cost, latency, and quality trends, AI Insights summarizes the same recorded metrics.

LaunchDarkly LLM observability handles traces separately from monitoring metrics and uses OpenTelemetry spans. LaunchDarkly can associate the trace with the AgentControl configuration, provided the LaunchDarkly Observability plugin, a valid LaunchDarkly context, and the model call are in the same request flow. If those conditions are not met, the trace may still reach the global traces view, but without the configuration association you need during an incident.

Meanwhile, online evaluations run asynchronously at the sampling rate you configure. Attached judges run automatically for sampled requests in completion-mode configurations. Other agent workflows may need direct invocation through an AI SDK. Check both sampling and invocation before treating a judge attachment as evidence that every response was evaluated.

How to identify an agent failure

To identify an agent failure, detect the quality regression, isolate the affected variation and cohort, reconstruct a failed run, judge the outcome, and compare against the baseline before acting. As an example, consider a multi-agent knowledge assistant undergoing a gradual rollout. A coordinator receives an employee’s question, a retrieval agent searches an approved internal corpus, and an answer agent writes the response. Your team is testing a candidate AgentControl variation with a specific employee cohort.

For this agent workflow, the application invokes a custom judge through a LaunchDarkly AI SDK on the responses selected for evaluation. The judge then checks whether the answer uses the required internal source, and the application records the resulting grounded_answer score against the config.

Dashboards and alerts as starting points

Once the application records the quality signals, dashboards and alerts must place them within the greater context of the agent’s operational health. To do this, group both types of signals by the relevant population and config variation so an alert highlights where the behavior changed rather than flattening the incident into “the agent got worse.”

Within LaunchDarkly, observability alerts can watch thresholds or anomalous changes in telemetry. You can then use that signal to open the matching time range in LLM Agent dashboards, which use generative-AI trace attributes to show agent activity. Vega can then investigate the observability data and summarize its findings. Plus, with GitHub connected in agent mode, Vega can propose or open a pull request for your team to review.

Connect a quality score to an automatic response only when the sampling design, threshold, evaluation window, and delay provide timely evidence for that task. Otherwise, use the alert to start the investigation and keep the response manual.

Whether the intervention is manual or automatic, you still need to know what changed and where before figuring out how to respond. Within this example of a knowledge assistant, that investigation would follow these steps:

Five steps to diagnose the regression

1. Detect the regression. An alert initially fires because the grounded_answer score has declined. Provider errors and latency look normal, but the assistant is still returning polished responses, so a conventional availability dashboard may not reveal the failure.
2. Isolate the affected variation. The monitoring signal identifies the environment, configuration, candidate variation, task family, and cohort. You can now determine whether the change is isolated to the candidate path or shared by the baseline.
3. Reconstruct a failed run. Open a sampled run from the affected population. Its root span carries the run and conversation IDs plus the configuration and evaluated variation. Child spans show the coordinator’s model call, the handoff to retrieval, the retrieval tool, and the answer agent. Logs use the same trace ID.
4. Judge the outcome. The candidate path hands work to the answer agent without a successful retrieval-tool span, and a related log records the skipped retrieval outcome. The quality evaluation explains the failure. The final response made an unsupported claim instead of using the required internal source.
5. Compare before acting. The trace, its related logs, and the evaluation will narrow the diagnosis without proving causality from a single dashboard view. To dig deeper, inspect more failed runs from the same task and cohort, compare them with the baseline variation, and check whether the missing retrieval step is consistent. To compare the two variations with statistical rigor rather than spot checks, run them as an experiment.

If candidate runs repeatedly miss retrieval while the baseline runs do not, the evidence may justify a bounded mitigation while you continue the investigation. It still does not prove that the variation caused the failure.

If baseline runs show the same pattern, don’t treat a variation-specific fallback as the primary response. Instead, investigate shared data, tool, or runtime dependencies.

Node-level diagnosis

After figuring out whether the problem is limited to one issue or more widespread, the next step is to determine where the problem occurs within the broader workflow. For a multi-agent workflow, a flat list of model calls makes ownership difficult to see, since it obscures which agent or handoff owns each action. Agent graphs expose the graph’s structure and node-level measures such as invocations and latency, along with errors and tool activity. Those measures appear when the application records them through an AI SDK tracker with the graph key. You can then compare retrieval-node tool activity with the coordinator and answer-node metrics to see whether the change is isolated.

Agent graphs do not execute the workflow. LaunchDarkly does not call the providers, manage graph state, choose the next edge, or enforce termination. Your application still owns traversal and execution. The graph supplies operational context; the trace and application evidence show what happened in a particular run.

How to intervene through prepared control paths

To intervene, use a control path you configured before the alert, such as a guarded rollout or an Adaptive Trigger, so the response has a known target and impact area. Let’s say the evidence shows that the failure pattern is concentrated among runs receiving the candidate variation. With the affected traffic identified, you can now select a bounded response.

A response defined before the alert gives you a known target and blast radius. For this incident, two configured control paths are relevant.

In the knowledge-assistant incident, you could roll back the guarded rollout before it expands the candidate variation to include more traffic. If you had configured an Adaptive Trigger around the appropriate accuracy measure, the trigger could serve the predefined stable variation.

An automatic response depends on a configuration set before the alert fires. If the evidence does not map to the configured metric, fallback, or target population, keep the response manual.

Adaptive Triggers evaluate their metric source across the whole environment. Their scope includes contexts that do not receive a specific configuration. Before connecting a broad signal to a fallback, confirm that the metric and target population describe the same failure. A global quality average, for example, can hide a severe regression in one cohort, while a traffic-mix change can move the average without any configuration regression.

Connect observability evidence only to a configured control with a known blast radius. Require approvals for changes to the control path, and use the audit log to see who changed what and when. If neither prepared path fits the evidence, it’s best to stick with a manual response.

How to verify recovery for the same task and population

After the response reduces exposure to the candidate variation or switches to a predefined fallback, new comparable runs will test whether it had the intended effect.

So return to the cohort and task family that triggered the response, then confirm that new runs resolve to the expected variation or traffic allocation. Check that retrieval spans reappear, the tool completes successfully, correlated logs no longer show the skipped-retrieval path, and the operational and quality measures recover over a sufficient window.

Use the same grounded-answer definition and evaluation version. Account for evaluation sampling and asynchronous results. If the failing population was one employee cohort, do not claim recovery from a global average that mixes it with unaffected traffic.

In particular, you’ll be looking to verify four things:

1. Did the intended population receive the intended configuration?
2. Did the execution path return to the expected sequence?
3. Did task quality recover without creating a new latency, error, token, or cost regression?
4. Did the result persist across enough comparable runs to reduce the chance that a transient change looks like recovery?

Each answer revises the system that will detect the next regression. The baseline now includes a documented bad period to compare against. The threshold that fired either proved timely or proved too slow to prevent exposure. If it proved too slow, retune it rather than leave it alone.

Any question you could not answer names a specific gap: a missing span attribute, a sampling rate too low to produce comparable runs quickly, or an evaluation window longer than the task can tolerate. Feed those revisions back before the next rollout. Then run the candidate variation against a saved dataset of past failures with offline evaluations in AgentControl Configs, so a known regression is caught before users see it.

A configuration update may propagate quickly, but telemetry ingestion and sampled evaluation have their own timelines, as do diagnosis and recovery verification. Measure the complete loop instead of attaching the configuration-propagation latency to the entire incident.

How to test the full response loop

You can close an incident only when the stack produces three things: the affected run and variation, a bounded response, and comparable post-change evidence. Use those requirements to assess an existing stack or a new implementation.

A stack that fails any row fails the whole test. Telemetry that identifies a regression you have no prepared way to act on often produces a well-documented outage. A control path with no cohort context can produce a response whose blast radius nobody can predict.

Before resuming or expanding the rollout, confirm that failed and recovered runs share the same task, cohort, configuration, and evaluation context. Otherwise, any change in traffic might be confused with recovery.

Then, resume the loop rather than closing it. The verified incident sets the baseline, the evaluation definition, and the threshold the next candidate will be judged against. The control path you used (or discovered you did not have) is the one to prepare before that candidate ships. Runtime control for the agents you run in production is a loop you keep tight: identify the affected behavior, apply a bounded intervention, verify the result against the same task and population, and apply what you learned to the next detection.

That loop is what runtime control looks like for the AI you run in production. The same evidence that tells you an agent changed also hands you a bounded way to change it back, without a redeploy, and a way to prove the fix held for the people it affected. In LaunchDarkly, that is AgentControl.

If you want to see the identify, intervene, verify loop on your own agents, book a demo. Move at AI speed. Stay in control.

FAQs

What is the difference between AI agent observability and monitoring?

Observability adds traces, logs, and evaluations that reconstruct a specific run, so a team can see which step failed and whether the result met the task's requirements. Monitoring tells you that an agent's behavior changed by watching metrics such as latency, errors, token cost, and quality scores, while observability helps you explain why.

What is a config variation for an AI agent?

A config variation is one version of an agent's configuration, meaning its prompts, instructions, and model settings, that a given request receives. Recording the evaluated variation on each trace lets a team compare a candidate version against the baseline and tell whether a regression is isolated to one version or shared by both.

What does blast radius mean for an AI agent incident?

Blast radius is the set of users, requests, and configurations that a response will touch. A bounded response, such as reverting one variation for one cohort, has a known blast radius. Without cohort and configuration context, an operator cannot predict who a rollback or fallback will affect, so the response risks reaching unaffected traffic.

How do you monitor a multi-agent AI system?

Monitor a multi-agent system at the node level, where each agent and handoff stays visible. Give each run a root span, with child spans for every model call, retrieval, tool execution, and handoff, then record per-node invocations, latency, errors, and tool activity so you can see which agent or handoff owns a failure.

How long does it take to recover from an AI agent regression?

Recovery from an AI agent regression takes longer than the configuration change itself. A config update may propagate quickly, but telemetry ingestion, sampled evaluation, diagnosis, and recovery verification each run on their own timelines. Measure the complete loop rather than reporting configuration-propagation latency as the incident's duration.