Observability has been foundational to software development for the better part of two decades. As distributed, cloud-based systems became the norm, engineering teams needed a common framework for understanding what was happening within them. Logs, metrics, and traces emerged as the lingua franca for monitoring and diagnosing issues at scale, powering the dashboards and alerts that engineering teams have come to rely on.
But traditional observability tools can only tell you what happened. They don’t tell you which change caused the problem, and they don’t proactively act on what they see. This creates a gap between the moment you know something is wrong and the moment you’re able to fix it. An alert fires, someone gets paged, and the manual investigation begins. This is a reality that teams have largely learned to live with, but in the AI era, it’s become a liability that shouldn’t be ignored.
There are two reasons for this shift. First, it’s now standard practice for most engineering teams to use AI to write code. Second, many of these teams are also building AI agents into their products, which are enormously powerful but inherently unpredictable. These are distinct yet interconnected forces that converge on a single imperative: control that lives in production, acts automatically, and operates at the change level—all at runtime, in real time.
AI has fundamentally changed how software is built
It’s no secret that teams are using AI to write code faster than ever, but that velocity comes with a corresponding increase in production incidents. According to the LaunchDarkly AI Control Gap Report, 94% of survey respondents confirm that AI has accelerated their team’s output, but nearly as many (91%) say they're more cautious about pushing AI-written code live. For every two steps forward, there's one all-too-frequent step back.
The impact of this problem isn't abstract. It can be seen from within an organization when middle-of-the-night firefights become the norm and engineers resign. And it can be seen from the outside when users lose trust in their favorite products and decide to try a competitor.
Simply put, it’s no longer feasible for human engineers on most teams to fix user-facing issues at the rate at which they're introduced. This problem is also reflected in survey data: 24% of respondents report that their team has to roll back or hotfix production issues daily, and 14% of teams get caught in this cycle multiple times a day. And finding a real solution—not just a band-aid—takes meaningful time and effort. That’s because traditional observability solutions can tell you something is broken, but they can’t identify which of the 47 changes that were deployed in the past 24 hours caused it.
Teams are therefore faced with an impossible choice: either slow down and risk losing competitive ground, or move ahead as quickly as possible while putting the user experience—and the business’s reputation—at risk.
AI agents are nondeterministic by design
The challenges of managing code that was written by AI are real, but they’re only part of the story. The most ambitious teams are building AI agents directly into their applications, pushing the boundaries of what software can do and redefining what users expect from it.
These agentic systems are defined by contingency and variability at every level. Nondeterminism isn’t a flaw; it’s the whole point. AI agents reason and adapt dynamically, which means their behavior can’t be reliably predicted—even by the teams that built them. Additionally, the models that power these agents are constantly and quietly being updated by providers, and the users interacting with them are endlessly variable in how they ask questions, what context they bring, and what they expect.
This unpredictability makes the limitations of preproduction testing painfully apparent, with users often sounding the first alarm that something is wrong. And even once teams know there’s a problem, the path to remediation is almost never straightforward. The definitions shaping agent behavior are scattered across repos and frameworks, and when an issue crops up, the toolchain offers little relief. Evals live in one tool, behavior control is elsewhere, and implementing a tested fix still requires a redeploy.
This delay between detection and remediation is a critical problem because a misbehaving agent doesn’t stop running while teams figure out how to handle it. Customers may continue to be exposed to bad responses for as long as the deployment cycle takes.
Runtime control bridges the gap between knowledge and action
In this landscape, teams have a clear and urgent need to move beyond reactive monitoring and toward proactive remediation. This evolution requires a new operating model: runtime control.
Runtime control doesn’t replace observability; it extends it. While observability tools provide visibility into what’s happening in production, they're not designed to intervene. Someone still has to investigate the problem—and then write and deploy a fix. Runtime control bridges that gap, giving teams the ability to automatically detect and respond to concerning, change-based signals live at runtime, before users feel the impact.
With this approach, the incident that used to take hours to diagnose and resolve can be handled in seconds. Whether the problem is a bug in AI-written code or a misbehaving agent, engineers wake up to “something happened, and it’s been handled,” instead of a 2 a.m. page.
Runtime control is the foundation for the AI software factory
As AI continues to transform the nature of software and how it gets built, the question teams should be asking isn’t whether their observability tooling is good, but whether it’s enough. Consider whether your team can:
- Release AI-generated changes progressively, limiting exposure while observing real-world impact.
- Control and govern AI agent behavior in production, not just monitor it.
- Halt or roll back within seconds when performance falls outside acceptable thresholds—without a redeploy.
- Trace an incident to the specific change that caused it, automatically.
- Automatically act on concerning health and performance signals before users feel the impact.
The teams that can do these things are able to ship faster with fewer incidents, and are best positioned to see stronger ROI from their AI investments. With runtime control in place, the loop of the software development lifecycle starts to close itself. Agents are able to build, release, observe, and iterate autonomously, with human judgment reserved for the moments that matter most. Engineers stop managing systems and start setting goals. That’s the AI software factory, and runtime control is the foundation it’s built on.
For a deeper look at how runtime control helps teams navigate the conflicting priorities of AI adoption and production reliability, check out the LaunchDarkly AI Control Gap Report.
