> ## Documentation Index > Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Finish your software factory: take a bad change back before anyone notices **Developer** **Federal docs** **EU docs** *Published September 4, 2026* Portrait of Scarlett Attensil. by Scarlett Attensil Most agentic factory stacks stop at the deploy. They generate, they validate, they ship, and then the instructions turn into a vague "monitor" step, which in practice means a person watching graphs. The capability that is actually missing is narrower than monitoring and harder to build: a change that moves the process gets taken back without a person. That is a throughput problem before it is a tooling problem. When a person wrote every change, review was the quality system, and a few changes a day could each be read before they shipped. A factory whose agents deploy continuously has no room for that, and the failures it produces are the kind review misses anyway. One of the two failures here passes every check the build side runs. The other one *is* the checks, all of them firing correctly while the pipeline quietly stops producing. So a factory without this is not slower than one with it. It is less certain. It ships improvements and degradations at the same rate, reports success both times, and leaves you one lever: a deploy-level rollback that takes back everything else along with the change you wanted gone. What you build here instead is a bounded blast radius while a change proves itself, a verdict that does not depend on who is paying attention, and a reversal that touches one change and leaves the rest of the band playing. The example repo builds an agentic band. Its loop generates, validates, and deploys about eleven times a minute. In this walkthrough, you start the music factory, listen to it, break it on purpose, and watch a [guarded rollout](https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts) put it back. Three agents make up your virtual band: a drummer, a bassist, and a keys player. They generate electronic music in the browser while a conductor keeps tempo and key. Every few bars each agent asks an AI model for the next version of its part. A parser accepts or rejects the result. If the new pattern is well-formed, the next cycle plays it. If it is not, the last good pattern keeps looping. [Shipping Floor](https://github.com/launchdarkly-labs/shipping-floor) contains no musical vocabulary of its own. Personas, gain limits, and groove recipes all come from LaunchDarkly [prompt snippets](https://launchdarkly.com/docs/home/agentcontrol/snippets.md) at runtime. If LaunchDarkly is unreachable, the band holds and then falls silent. There is no local fallback for this demo. Two things about the band can change without a deploy, and each uses a different LaunchDarkly object. What an agent *is*, its model, its parameters, and a system message assembled from pinned prompt snippets, is a variation in an [AgentControl config](https://launchdarkly.com/docs/home/agentcontrol). The code *around* the agent, including the limit its validator enforces, sits behind a boolean flag. Either one changes what the running band does on its next generation, and neither needs a restart. The parser only asks whether a pattern is well-formed, which is a narrower question than whether the music got better. An agent asked to make its part more interesting can comply by turning itself up. Past the point where the speakers reproduce the mix cleanly the sound stops improving, and every check still passes and reports another good version. That is why a variation carries a gain ceiling, and why a [metric](https://launchdarkly.com/docs/home/metrics) reads the peak gain of what actually shipped. This walkthrough ships that limit as a flag, `strict-mix-gate`, set tighter than the drummer can satisfy at all. Every pattern it generates is rejected. Hold-last-known-good replays the previous bar while the bassist and keys, whose limits tighten only slightly, keep publishing around it. One instrument is dead, the other two carry the mix, and nothing crashes. You will not hear it. The publish rate will. The correction is not to loosen the gate, since nothing yet tells you the gate is the problem. It is to count the attempt and the success separately, which means a rejected candidate contributes a zero instead of contributing nothing. Then guard the change with the metric that can detect its failure shape. That is the whole argument in one line: gate each artifact on an invariant you can compute, measure the process for drift, and let a change that moves the process be taken back without a person. Those are three different clocks, and the last one is a release control rather than a gate, which is why it can catch a change that every individual gate approved. A loud regression and a silent one are not visible to the same instrument. Each change needs its own choice of metric. A factory gates in three places, and they run on different clocks. Only the first is a runtime gate:
Layer Clock Decides On failure Here
Parser and `maxGain()` Every generation Is this artifact allowed Reject and retry Already in the repo
Hold-last-known-good Every generation, on rejection What plays instead Replay the previous bar Already in the repo
Guarded rollout One monitoring window Is this change allowed to continue Revert the change You configure it in [step 4](#step-4-break-it)
## Prerequisites * Permission to [create a LaunchDarkly project](https://launchdarkly.com/docs/home/account/project) and a server-side [SDK key](https://launchdarkly.com/docs/home/account/environment/keys.md), plus permission to create metrics and start a guarded rollout. Used from [step 2](#step-2-stand-up-the-project) onward. Do not reuse an existing Shipping Floor project. * An Anthropic API key, since the agents call Claude to generate their patterns. Used in [step 2](#step-2-stand-up-the-project). * A coding assistant you can connect to a Model Context Protocol (MCP) server. Used in [step 2](#step-2-stand-up-the-project) to create several dozen resources for you. * Node 22 or later. Used in [step 1](#step-1-clone-and-install-the-application). ## Step 1: Clone and install the application Run this from a directory that is not already a Shipping Floor checkout. Cloning inside an existing clone nests a second copy and fails. Here is the clone: ```bash theme={null} git clone https://github.com/launchdarkly-labs/shipping-floor.git cd shipping-floor npm install ``` If you already have the repo, skip the clone and run `npm install` in that directory. Do not start the app yet. Until LaunchDarkly is serving prompts, there is nothing for it to play. ## Step 2: Stand up the project The application already evaluates a boolean flag called `strict-mix-gate` and already emits the guardrail metrics. Your job is to create the LaunchDarkly resources the seed lists in a **new** project, not to edit application code, and not to write into an existing Shipping Floor project. Using the shared demo is not a new implementation. ### Create and connect a new project To create the project and connect your assistant to it: 1. Open [Projects](https://app.launchdarkly.com/settings/projects). 2. Click **Create project** and copy the new project key. 3. Connect [the hosted MCP server](https://launchdarkly.com/docs/home/getting-started/mcp-hosted.md) in the client you are using. 4. Add `https://mcp.launchdarkly.com/mcp/launchdarkly` under the MCP settings (or use the [install link](https://mcp.launchdarkly.com/mcp/launchdarkly/install)), then finish the browser **Allow** prompt. `claude mcp add` configures Claude Code only. 5. Ask the assistant to run `list-projects`. This real API call is the authentication check. If it returns `token_expired` even after an auth helper reported success, remove the MCP server, add it again, and finish the browser consent flow once more. Here is the equivalent command for Claude Code: ```bash theme={null} claude mcp add --transport http "launchdarkly" \ "https://mcp.launchdarkly.com/mcp/launchdarkly" ``` Cursor may hold production MCP writes behind an approval card. If targeting, metrics, or the rollout appears to stop, approve the card and let the assistant retry. These writes stand up the project. They are not a customer rollout. ### Point the app at the new project Connect the application to the new project before bootstrap, because otherwise the assistant defaults to an existing `shipping-floor` project. Copy the example environment file: ```bash theme={null} cp .env.example .env ``` Set these three values in `.env`: ```text theme={null} LAUNCHDARKLY_PROJECT_KEY= LAUNCHDARKLY_SDK_KEY= ANTHROPIC_API_KEY= ``` Print the resource specification: ```bash theme={null} npm run seed ``` Run `/factory-bootstrap` in your assistant (the command file is `.claude/commands/factory-bootstrap.md`) and give it the project key from `.env`. Before it pauses, the command: * Creates the snippets, configs, and targeting from `seed/` * Creates `strict-mix-gate`, turns it **ON**, and sets its default rule to `false` * Stops before creating metrics if `request` is not yet available for experiments Check the two project lines in the seed output. `seed default project: shipping-floor` names the checked-in example. `this run targets:` must show the new project key from `.env`. ### Register the request context The metrics randomize on the `request` context kind. That kind has to exist and be available for guarded rollouts before bootstrap creates the metrics. > [!WARNING] > > **Do not create the context kinds first** > > **Add kind** is restricted to admin users, and a new project does not grant > that permission. The [Contexts list](https://launchdarkly.com/docs/home/flags/contexts-list) shows > instances. It does not create kinds. Let SDK evaluations create the kinds > instead. When `/factory-bootstrap` pauses before metrics, complete these steps: 1. Run `npm run seed -- --verify`. The evaluations create the `musician`, `performance`, `listener`, and `request` [context kinds](https://launchdarkly.com/docs/home/flags/context-kinds-create.md#create-context-kinds-from-an-sdk). 2. Open **Features → Contexts** (or **Code → Contexts**) and click the **gear**. That page is the instance list. Kinds live behind the gear, not behind **+ Create**. 3. Edit `request` and select **Available for experiments and guarded rollouts**. The other three kinds do not need this setting for the walkthrough. 4. Return to the assistant and tell it to continue. It now creates `peak-gain`, `ceiling-breach-rate`, and `publish-success-rate` with `randomizationUnits: ["request"]`. The order matters. Creating a metric before `request` is available fails with `Randomization unit "request" not found`. Omitting `randomizationUnits` avoids that error but silently defaults the metric to `user`, and MCP has no `update-metric` tool to repair it. The [context kind API](https://launchdarkly.com/docs/api/contexts/put-context-kind.md) can set the experiments checkbox, but the hosted MCP server has no context-kind tool. That is why this walkthrough uses one UI edit after the SDK creates the kinds. ### Verify the finished setup Run the verification command again after bootstrap creates the metrics: ```bash theme={null} npm run seed -- --verify ``` This evaluates every config through the SDK and checks that `strict-mix-gate` is serving `false`. The command has to reach `stream.launchdarkly.com`. A run without network access, or an offline run, fails on DNS. Do not start the application until this exits 0. ## Step 3: Play and listen Start the application: ```bash theme={null} npm start ``` Open `http://localhost:3000` and press **Play**. Audio does not start until then. Three agents start trading patterns, each one regenerating every few bars. `npm start` runs a preflight check first and refuses to start if LaunchDarkly is not delivering usable configs. The heads-up display (HUD) has two kinds of activity on one screen: the model calls that keep the band playing, and the release work that changes how those calls are governed. Here is the running band after a few minutes of Play: ![The Shipping Floor HUD with drummer, bassist, and keys in the rack. The production line shows Generate at 74. Classify, Flag, Measure, Release, and Clean up are 0. The drummer panel serves the drummer config, variation four-on-the-floor v1 on claude-haiku-4-5-20251001, with strict-mix-gate false and three pinned v1 prompt snippets. Its guardrails show publish rate 44%, breach rate 0%, peak gain 1.10 against an enforced ceiling of 1.15, published 12, held 0, and breaches 0. The footer reads 59% publish rate, 74 generated, 43 shipped, 1 rejected, and 0 held.](https://mintcdn.com/launchdarkly/cQxUtI1zUzSBVto8/images/tutorials/agent-pipeline-quality-control/shipping-floor-hud.png?fit=max\&auto=format\&n=cQxUtI1zUzSBVto8\&q=85\&s=72fac5eae56bff5bd633bef77c8e2f4a) The Shipping Floor HUD while the band plays, with the drummer selected in the Control Tower. The guardrails lead with publish rate and the ceiling the validator is actually enforcing. The HUD has two clocks on one screen. The production line and the counters are the per-generation factory: generate, gate, deploy, hold last good. The Control Tower's **strict-mix-gate** row and the LaunchDarkly rollout are the per-change factory: a flag arm, a metric verdict, a revert. Only the first clock is a gate. Read the current layout in three passes. ### Read the production line **Generate** is the count of model calls since the server started. Retries count as additional calls. The number is not a song count or a count of completed bars. The dot turns green when a candidate ships and flashes an alarm color when one is rejected. The labels after the seam describe the release side of the factory. They are not stages that every candidate passes through. This demo updates two totals in the line: **Generate** counts model calls, and **Measure** counts ceiling breaches. **Classify**, **Flag**, **Release**, and **Clean up** remain at zero because the browser does not receive activity from those assistant-driven workflows. A **published** increment in the ledger is one musician shipping one bar, not one pass through that six-node line. ### Read the selected musician **Serving** names the config, variation, model, evaluation contexts, and the `strict-mix-gate` arm this musician last ran under (`false`, or `true · strict`). An empty variation key is the alarm: LaunchDarkly served the SDK default and is not driving this musician. **Prompt composition** names the three pinned snippet versions. The HUD never renders the assembled prompt text. **Guardrails** lead with two rates, in the same direction LaunchDarkly reads them: **publish rate** (higher is better) and **breach rate** (lower is better). Under those, **peak gain / enforced ceiling** is the loudest `.gain()` in the current accepted pattern against the ceiling the validator is actually using. When the gate is off, that is the variation's `gain_ceiling`. When it is on, it is `strict_gain_ceiling`. Showing the nominal ceiling here used to make a legitimate rejection look like a HUD bug. **Published · held** is how many candidates shipped versus how many turns kept the last good bar. **Breaches** is how many candidates exceeded the enforced ceiling since this process started. ### Read the totals and serving details The footer is band-wide and leads with **publish rate**, then generated, shipped, rejected, and held. The tower covers one musician. This line covers all three. A live capture can be one call ahead of its results while a request is in flight. **Served split** summarizes the AgentControl variations behind recent published patterns. It is observed traffic, not configured rollout progress, and it does not show the `strict-mix-gate` flag arms. Let the band run until the healthy pattern is familiar. Generated and shipped keep climbing, the code in the rack changes, peak gain stays below the **enforced** ceiling, and held moves rarely. In [step 4](#step-4-break-it), requests assigned to the stricter gate fail more often: publish rate falls, held rises, and the drummer repeats its last good bar more often. The music may hide an occasional repeat. The counters do not. ## Step 4: Break it The flag is already in the code and already in your project. You are going to roll `true` out against `false`. When `true` is served, the validator uses `strict_gain_ceiling` (0.60 on the drummer) instead of `gain_ceiling` (1.15). Any drum candidate above 0.60 now fails. When every retry for a regeneration fails, hold-last-good takes over and the drummer repeats the previous bar. Nothing throws. Start the [guarded rollout](https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts) first, then generate traffic. A burn-in that finishes before the rollout is live does not feed it. If you are starting the rollout through MCP, `start-guarded-rollout` requires `regressionThreshold` in its schema, and the API rejects a real threshold. Send `regressionThreshold: 0`. From the `strict-mix-gate` flag's **Targeting** tab, on the rule already serving `false`:
SettingValueWhy
Randomization unit`request`One coin flip per generation, which means the rollout is audible within a single session
Guardrail metric`publish-success-rate`The failure you are watching for is output that stops arriving
On regressionRoll back automaticallyNo human in the loop
First stage5%A real failure reaches little traffic before it reverts
Then start the load generator: ```bash theme={null} npm run burn-in -- --state section=lift,energy=high,isBoundary=false --generations 500 ``` Keep the browser tab open. Requests assigned `true` use the tighter limit. The drummer's strict ceiling is 0.6 against a kick range its own prompt puts at 1.0 to 1.15. The drummer cannot comply and stops publishing entirely. The bassist and keys tighten only slightly and keep regenerating. Nothing fails loudly. The drummer's held count climbs, the band-level publish rate falls partway because two thirds of the work still lands, and if you had walked in partway through you might not hear that anything was wrong. ## Step 5: Watch it revert The failure is not silent in LaunchDarkly. The **Targeting** tab shows the rollout status, and the **Monitoring** tab next to it has a tile for `publish-success-rate`. When LaunchDarkly ends the rollout, the banner reads **Default rule rolled back automatically after detecting a regression for Publish success rate**. ![The Targeting tab for the Strict mix gate flag in Production. A banner labeled Release phase 1 of 1 reads Default rule rolled back automatically after detecting a regression for Publish success rate, ended Oct 22 at 2:38 PM. Below it, the targeting configuration shows the flag On, serving variations based on rules, with 601 evaluations, and the default rule serving false.](https://mintcdn.com/launchdarkly/xb62UfS4wMVrvFXT/images/tutorials/agent-pipeline-quality-control/targeting-rolled-back.png?fit=max\&auto=format\&n=xb62UfS4wMVrvFXT\&q=85\&s=19892cda7307eb8890e13db2c381f589) The strict-mix-gate Targeting tab after the rollback. The flag is on and the default rule serves false. Expand the banner to read the summary that follows. ![LaunchDarkly guarded rollout summary. The banner reads Default rule rolled back automatically after detecting a regression for Publish success rate, ended Oct 22 at 2:38 PM. The true variation shows 39 unique request evaluations at 25% traffic. The Publish success rate chart is labeled minus 33.3 percentage points (pp) with an interval from minus 66.3 to minus 0.4 pp. The table below it shows true at an estimate of 49%, from 18% to 80%, with 19 conversions in a sample of 39, and false at 82%, from 58% to 100%, with 32 conversions in a sample of 39.](https://mintcdn.com/launchdarkly/cQxUtI1zUzSBVto8/images/tutorials/agent-pipeline-quality-control/guarded-rollout-rollback.png?fit=max\&auto=format\&n=cQxUtI1zUzSBVto8\&q=85\&s=b43bd2e0ddce161d1937ca1df6c48c46) The guarded rollout summary after automatic rollback. Publish success rate dropped 33.3 pp. The default rule is serving false again. Read this panel as a comparison between two policies, not as a second copy of the HUD. The `true` arm received 39 unique `request` contexts while the rollout was at 25% traffic. LaunchDarkly estimated that 49% of requests in that arm published a pattern, compared with 82% for the original `false` arm. Because publish success is a higher-is-better metric, the −33.3 pp difference is a regression: the stricter gate is stopping new work from reaching the player. The interval beside it, −66.3 pp to −0.4 pp, sits entirely below zero. **Rolled back automatically, now serving false** is the action LaunchDarkly took in response. The HUD and the rollout tile answer related questions at different levels. The HUD's publish rate asks, “Of this musician's generations since the process started, how often did a pattern ship?” The rollout metric asks, “For the request contexts assigned to each arm, how often did a generation publish at all?” That is why the tile can compare `true` with `false` even though the HUD shows one process-lifetime rate. Timing matters here. Generations that finish before the rollout is live never reach the comparison. Start the burn-in only after the rollout exists. Run that way, its summary covers both arms together and exposes the gate:
VariationPublishedCeiling breachesHeld
`breakbeat-chopping` (drummer)94.0%2010
`rolling-synth-bass` (bassist)100%130
`stab-chords` (keys)97.0%175
The drummer is the musician the tight ceiling hits. Held 10 is hold-last-good: those bars never reached the speakers, which is what dropped `publish-success-rate`. Without anyone touching the flag, the drummer starts varying again. The default rule is serving `false`. Open [change history](https://launchdarkly.com/docs/home/releases/change-history.md) for the timestamps and the actor: ``` 21:36:53 Scarlett Attensil Starting guarded release: true (5%) and false (5%) 21:37:08 (via API) member: null Guarded release advanced to the next stage: true (10%) and false (10%) 21:38:07 (via API) member: null Guarded release advanced to the next stage: true (25%) and false (25%) 21:38:39 (via API) member: null Reverted the guarded release on the default rule and is now serving `false` ``` `member: null` is the line worth keeping. No person advanced or reverted that rollout. In the run this tutorial draws on, it started at 5%, advanced to 10% and then 25%, and reverted at 2:38 PM, 105 seconds after it began. Detection under sequential testing varies by run. Do not plan around a duration. ## How the run side is wired You can stop here. What follows is what to copy if you are adapting your own pipeline. The three layers from the opening table, in the detail you need to rebuild them. The design gets clearer after you stop calling all three of them gates. **Per artifact, inline.** The parser and `maxGain()` in `src/validate.js` judge one pattern before it reaches the speakers. This is the only true runtime gate. It runs on every generation, it decides in microseconds, and it publishes or rejects. It can enforce any invariant you can compute from the artifact, and nothing beyond that. A gate that has to call a model to reach its verdict is not a gate, it is another generator. **Per artifact, on rejection.** Hold-last-known-good decides what plays when the gate says no. This is the layer that turns a hard failure into a quiet one. Because of that, it carries its own counters: attempts, holds, and the age of what you are still serving. **Per change, over a window.** The guarded rollout is not a runtime gate. It never inspects a pattern and it cannot stop one from playing. It decides how much traffic a *change* is allowed while a metric gathers enough evidence to judge it, then keeps the change or takes it back. Its clock is the monitoring window, not the generation. Conflating the third with the first two is the common mistake. A gate answers whether this artifact is allowed. A guarded rollout answers whether this change is allowed to keep going. No amount of tightening the first produces the second, which is why a factory needs both. Three rules follow. **Gate on invariants, measure for drift.** A per-artifact verdict cannot express a distribution, and a distribution cannot stop one bad artifact. The gain ceiling is the invariant. `peak-gain` and `publish-success-rate` are the drift. **Count every fallback.** A hold that increments no counter is a silent failure by construction. Absence has to be recordable, which is why `publish-success-rate` fires on the publish rather than on the attempt: a generation that never publishes contributes a zero instead of contributing nothing. **Put every gate behind a flag.** `strict-mix-gate` is a flag, not a constant. A limit that turns out to be wrong is then a targeting change rather than a release. A gate nobody can loosen in seconds is a gate somebody eventually disables for good. Three implementation notes carry the rest. **The prompt is pinned snippets.** A variation's system message is three [prompt snippets](https://launchdarkly.com/docs/home/agentcontrol/snippets.md) by key and version. The limits snippet sits above the vocabulary snippet so the model reads the invariant first. Open `src/config/launchdarkly.js` to find the resolve and the length check that catches a snippet reference that rendered to nothing. **The unit is one generation.** The application builds a [multi-context](https://launchdarkly.com/docs/home/flags/multi-contexts.md) per generation. The `request` kind is the randomization unit. A single static context gives a rollout a sample size of one forever, and LaunchDarkly reverts it for failing the minimum-context requirement. A flag and a config variation are the same shape at this layer, one evaluation per generation, which is why the same rollout mechanism reverses either one. **The metric comes from the artifact.** `src/validate.js` reads peak gain from the pattern string. It does not call a model or inspect audio, which is why the headless load generator and the live application agree. `publish-success-rate` fires when a regeneration publishes a pattern. A held result emits no success event, which means the rate falls when the gate blocks new work. Swap the artifact and the rule, and the same shape sits in front of generated SQL, generated JSON, or generated code. A moderation classifier whose auto-approve threshold rises is the same failure: nothing errors, every item routes to human review, and only the publish rate shows it. ## What the run side actually gave the factory A finished run side has five requirements. This walkthrough proved two of them and started a third. The remaining two are named so you know what you did not do. Here is how this walkthrough covers each requirement:
Requirement In this walkthrough
Know what shipped Proved. The identity of the failure is the `strict-mix-gate` flag serving `true`, not a deploy job. The Control Tower names the config, variation, pinned snippets, model, gate arm, and enforced ceiling on every bar, including rejections.
Keep a record that outlasts the change Started. [Change history](https://launchdarkly.com/docs/home/releases/change-history.md) records the rollout and the revert, including that no person triggered it. Long-term export is out of scope.
Decide exposure by policy, not by a person Skipped. You filled in the rollout dialog yourself. A [release policy](https://launchdarkly.com/docs/home/releases/release-policies.md) is how that default attaches to every release later.
Reverse one change, not the deploy Proved. `strict-mix-gate` went back. The rest of the band kept playing.
Clean up what the change left behind Skipped. The flag and snippets stay. Read [flag cleanup](https://launchdarkly.com/docs/home/flags/manage/flag-cleanup-vega.md) before you let agents create the next dozen flags.
This walkthrough only shows a change being taken back, because that is the part that leaves evidence. You do not hear the freeze. The metric tile falls and the default rule goes back on its own. The rails do not know the difference. A candidate that stays inside its ceiling advances through the same stages, completes its last one, and becomes the new baseline. That symmetry is the point. A run side trusted only to block changes is a run side people route around.