> ## Documentation Index
> Fetch the complete documentation index at: https://launchdarkly.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt an A/B test using Cursor and LaunchDarkly

> Prompt a coding agent like Cursor to build an app, create a flag, and set up an A/B test in LaunchDarkly, with your approval at every step.

**Developer**

**Federal docs**

**EU docs**

*Published November 13, 2026*

Headshot of Carmen Romero.

by Carmen Romero

Product decisions often come down to opinion. "Which button text should we use?" usually
gets settled by whoever has the strongest argument, not by which option performs best.

The data-driven answer to these questions is to run an experiment. But designing, running,
and analyzing an experiment can feel like a big project, especially if you're not a data scientist.
That friction is often enough to skip testing altogether. When teams ship changes based
on guesses, they ship without a way to show the impact of their work.

A coding agent can set up an experiment for you, but handing it everything is risky.
An agent doesn't know what's worth testing or what success looks like. Left on its own,
it can act haphazardly, starting an experiment or turning on a flag before you're ready,
which changes what your users see.

In this tutorial, you use a coding agent and the [`launchdarkly-experiment-setup`](https://github.com/launchdarkly/ai-tooling/tree/main/skills/experiments/launchdarkly-experiment-setup) agent skill to run
your first A/B test. Your agent handles the busywork: it builds a small app, creates the flag,
and sets up the metric and experiment. You make the important decisions: you write the experiment design,
approve it, turn on the flag, and decide when the experiment starts.

When you're done, LaunchDarkly results show which version performed better and how confident
you can be. That way, the next debate about button text ends with data instead of opinion.

## Prerequisites

**Accounts and information**

* A LaunchDarkly account with Experimentation, and permission to create flags, metrics,
  and experiments.
* A LaunchDarkly [project](https://launchdarkly.com/docs/home/account/project#create-projects) to work in, and the
  [SDK key](https://launchdarkly.com/docs/home/account/environment/keys.md#view-or-copy-sdk-credentials) for the
  environment you'll use. This tutorial uses the test environment.

**Tools**

* A coding agent that supports Model Context Protocol (MCP) and agent skills, such as Cursor or Claude Code,
  running on your computer.
* [Node.js](https://nodejs.org) long-term support (LTS) and npm.
* Optional: Git, to clone the [companion repo](https://github.com/launchdarkly-labs/prompt-ab-test).

**Connections and skills**

* The [hosted LaunchDarkly MCP server](https://launchdarkly.com/docs/home/getting-started/mcp-hosted.md), connected and
  signed in to your account. This lets your agent create the flag, metric, and experiment.
  If you have more than one LaunchDarkly plugin or MCP server installed, keep only
  the hosted server on. This means your agent uses the current skill.
* The [`launchdarkly-experiment-setup`](https://github.com/launchdarkly/ai-tooling/tree/main/skills/experiments/launchdarkly-experiment-setup)
  agent skill, version 0.3.0 or later. You install this in Step 1.

> [!NOTE]
>
> **Skip ahead with the companion repo**: Clone the repo, check out the `finish` branch,
> run `npm install`, and add your SDK key to `.env`. Then install the experiment skill
> in the project folder, as described in Step 1, and start at Step 3. In Step 6, skip
> building the simulator, and run `npm run simulate`.

## Step 1: Plan what to build with your agent

Start by deciding what to test, then ask your agent for a plan rather than code. Reviewing the
plan first lets you check what the agent intends to build before it changes any files.

To set up your project:

1. Create an empty folder for the project.
2. Open the folder in your coding agent.
3. Run `git init` in a terminal in that folder.
4. Install the experiment skill in the same folder:

   ```bash theme={null}
   npx skills add https://github.com/launchdarkly/ai-tooling --skill launchdarkly-experiment-setup
   ```

   The skill is added to a `.agents/skills/` folder in your project, where your agent
   reads it.
5. Start a new chat in your agent. This ensures it picks up the skill.

Send this prompt to your agent:

```text title="Prompt 1: Create a plan" lines wrap theme={null}
I want to run my first A/B test with LaunchDarkly.
Build a tiny Node.js app with one signup page and one button.
A LaunchDarkly flag called cta-button-text should pick the button text:
"Start free trial" or "Try it free for 14 days".
When someone clicks, send a LaunchDarkly custom event called cta-clicked.
For now, just plan the app; we'll create the flag and set up the experiment later.
Look up the official LaunchDarkly docs for every SDK detail, starting at
https://launchdarkly.com/docs/llms.txt. Plan first, don't write code yet.
```

Your agent replies with a plan and doesn't create any files. The plan describes a small
Node.js server that uses the LaunchDarkly Node.js server SDK. The signup page reads the
`cta-button-text` flag for each visitor, and clicking the button sends a `cta-clicked`
event for that same visitor. The plan ends by asking whether to build the page.

Your agent's plan may be worded differently, or use different file names. That's
expected. What matters is that it uses the flag key `cta-button-text` and the event key
`cta-clicked` exactly as written, because the later steps depend on them.

To learn more about the SDK your agent uses, read [Node.js SDK reference (server-side)](https://launchdarkly.com/docs/sdk/server-side/node-js).

## Step 2: Build the signup page

Next, have your agent build what it planned. You add your SDK key yourself. Do
not paste it in chat.

Send this prompt to your agent:

```text title="Prompt 2: Build demo app" lines wrap theme={null}
Looks good, build the app. We'll create the flag and set up the experiment in later steps.
Keep it as small as possible.
Create a .env file with a placeholder for my SDK key; I'll paste the real key in myself.
Add a .env.example with placeholders, and keep .env, .agents/, and skills-lock.json out of git.
Don't commit yet.
Add a README with setup steps. Then tell me how to run it.
```

Your agent builds the page, installs the SDK, and tells you how to start it.

> [!WARNING]
>
> **Keep your SDK key out of the chat**
>
> Paste the key into your `.env` file yourself. Your agent sends everything in the chat to
> its model provider. Never paste the key there.

To run the page:

1. Copy the SDK key for your test environment from LaunchDarkly. It's the server-side key.
2. Replace the placeholder in your `.env` file with the key.
3. Save the file.
4. Run `npm start` in your terminal.
5. Open `http://localhost:3000`. The button reads "Start free trial."
6. Click the button a few times. A thank-you page appears after each click.

![The signup page your agent builds, with the button showing "Start free trial."](https://mintcdn.com/launchdarkly/R1usXG6yY_Qx_Id2/images/tutorials/prompt-ab-test/application.png?fit=max\&auto=format\&n=R1usXG6yY_Qx_Id2\&q=85\&s=4935cfb7d95604c3263cc8dc9fffa8a8)

The signup page your agent builds, with the button showing "Start free trial."

To confirm the clicks are reaching LaunchDarkly, open **Live events** for your test
environment. A `cta-clicked` event appears within a few seconds of each click.

To learn more, read [Live events](https://launchdarkly.com/docs/home/releases/live-events.md) and
[SDK credentials](https://launchdarkly.com/docs/home/account/environment/keys.md).

## Step 3: Create the flag

Your agent creates the flag through the LaunchDarkly MCP server, and you confirm it in
LaunchDarkly.

Send this prompt to your agent. If your project has a different key, replace
`prompt-ab-test-tutorial` with it:

```text title="Prompt 3: Create the flag" lines wrap theme={null}
Using the LaunchDarkly MCP server, create a string flag called cta-button-text
in my prompt-ab-test-tutorial project with two variations:
"Start free trial" and "Try it free for 14 days". Don't change targeting.
```

Before it creates the flag, your agent may ask you to approve the MCP tool call. Read
what the call does, then approve it.

Your agent creates a string flag named `cta-button-text` with the two variations. The
flag is off in every environment and has no targeting rules. Your agent may also mark the
flag as temporary or add tags. Those labels don't affect the experiment.

To check the flag in LaunchDarkly:

1. Click **Flags**. The flags list appears.
2. Click `cta-button-text`. The flag's page appears.
3. Confirm the flag has two variations, "Start free trial" and "Try it free for 14 days."
4. Confirm the flag is off in your test environment.

![The "cta-button-text" flag in the test environment. It's off, with two variations: "Start free trial" and "Try it free for 14 days."](https://mintcdn.com/launchdarkly/R1usXG6yY_Qx_Id2/images/tutorials/prompt-ab-test/flag-in-test.png?fit=max\&auto=format\&n=R1usXG6yY_Qx_Id2\&q=85\&s=bdabe1a16779b93832872e3fbea487c2)

The "cta-button-text" flag in the test environment. It's off, with two variations: "Start free trial" and "Try it free for 14 days."

> [!NOTE]
>
> **The flag starts off on purpose**
>
> While the flag is off, LaunchDarkly serves its off variation, "Start free trial," to
> everyone. You turn the flag on in Step 5, right before the experiment starts.

If you cloned the companion repo and started at this step, run `npm start`, open
`http://localhost:3000`, and click the button a few times. The clicks send `cta-clicked`
events, which the experiment skill checks for in Step 4.

To learn more, read [Creating new flags](https://launchdarkly.com/docs/home/flags/new.md) and
[LaunchDarkly hosted MCP server](https://launchdarkly.com/docs/home/getting-started/mcp-hosted.md).

## Step 4: Design and approve the experiment

This is the step you don't hand off. Your agent can set up an experiment, but it can't
decide what's worth testing or what success looks like. You write the design, and the
`launchdarkly-experiment-setup` skill builds exactly that, only after you approve it.

### Write your design

An experiment design answers a few questions before any data comes in. This table
describes each part of the design in this tutorial:

<table>
  <thead>
    <tr><th>Component</th><th>In this tutorial</th><th>Why it matters</th></tr>
  </thead>

  <tbody>
    <tr><td>Hypothesis</td><td>If we change the button to "Try it free for 14 days", more users will click it, because it tells them exactly what they're getting.</td><td>States what you expect</td></tr>
    <tr><td>Flag and variations</td><td>`cta-button-text`, with "Start free trial" and "Try it free for 14 days"</td><td>Decides what each visitor sees</td></tr>
    <tr><td>Control</td><td>"Start free trial"</td><td>The baseline that the other variation is compared against</td></tr>
    <tr><td>Primary metric</td><td>The share of users who sent the `cta-clicked` event</td><td>The one number that decides the result</td></tr>
    <tr><td>Randomization unit</td><td>User</td><td>Each visitor sees one version consistently</td></tr>
    <tr><td>Traffic split</td><td>50/50</td><td>Sends equal traffic to each variation</td></tr>
    <tr><td>Statistics</td><td>LaunchDarkly defaults</td><td>Frequentist with sequential testing, which lets you check results early without inflating false positives</td></tr>
  </tbody>
</table>

Send this prompt to your agent. If your project has a different key, replace
`prompt-ab-test-tutorial` with it:

```text title="Prompt 4: Design the experiment" lines wrap theme={null}
Using the launchdarkly-experiment-setup skill, set up an experiment in my
prompt-ab-test-tutorial project, in the test environment. Here's my design:
- Hypothesis: If we change the button to "Try it free for 14 days", more users
  will click it, because it tells them exactly what they're getting.
- Flag: cta-button-text. Variations: "Start free trial" and "Try it free for 14 days".
  Control: "Start free trial".
- Primary metric: the share of users who sent the cta-clicked event.
  Create it if it doesn't exist.
- Randomize by user, 50/50 split. Keep the default statistics settings.
```

### Review the agent's design

Your agent checks that `cta-clicked` events are reaching your test environment, then
shows you the complete design. It doesn't create anything yet.

The design lists every part from your prompt, plus the details the skill fills in. Any
value the skill chose for you, like the experiment and metric names and keys, is
labeled as suggested. Statistics appear as "LaunchDarkly default (frequentist,
sequential testing)."

Before you approve, check these parts of the design:

* The metric uses the event key `cta-clicked`.
* The metric measures the share of users who clicked, not the total number of clicks.
* The control is "Start free trial."
* The randomization unit is `user`.

If anything is wrong, tell your agent what to change. It shows you the updated design and
asks again.

> [!NOTE]
>
> **What happens if something is missing**
>
> If your prompt leaves out part of the design, the skill asks for everything that's
> missing before it shows a design. If the `cta-clicked` event isn't arriving yet,
> the design includes a warning and lists the events that are arriving.

### Approve and check the draft

When the design looks right, reply:

```text title="Prompt 5: Create the experiment" lines wrap theme={null}
Yes, create it.
```

Your agent creates the metric and the experiment. The experiment is a draft and doesn't
collect any data until you start it in Step 5.

To check the draft in LaunchDarkly:

1. Click **Experiments**. The experiments list appears.
2. Click the experiment your agent created. Its **Design** tab appears.
3. Confirm the hypothesis, the two variations at 50% each, and "Start free trial" as
   the control.
4. Confirm the primary metric is the one your agent created for `cta-clicked`.
5. Confirm the experiment shows as not started.

![The Design tab for the experiment your agent created. It shows the hypothesis, the two variations at 50% each, and the primary metric.](https://mintcdn.com/launchdarkly/R1usXG6yY_Qx_Id2/images/tutorials/prompt-ab-test/exp-design.png?fit=max\&auto=format\&n=R1usXG6yY_Qx_Id2\&q=85\&s=026c9599e53ef39678f330b401aae41f)

The Design tab for the experiment your agent created. It shows the hypothesis, the two variations at 50% each, and the primary metric.

To learn more, read [Creating experiments](https://launchdarkly.com/docs/home/experimentation/create.md) and
[Bayesian versus frequentist statistics](https://launchdarkly.com/docs/guides/experimentation/bayesian-frequentist.md).

## Step 5: Turn on the flag and start the experiment

Turning on the flag and starting the experiment are two separate decisions. Turning on
the flag changes what visitors see. Starting the experiment tells LaunchDarkly to begin
collecting data. The skill never makes either decision for you.

To turn on the flag:

1. Click **Flags** in LaunchDarkly. The flags list appears.
2. Click `cta-button-text`. The flag's page appears.
3. Choose your test environment from the environment menu.
4. Toggle the flag **On**.
5. Click **Review and save**.
6. Click **Save changes**.

Turn on the flag right before you start the experiment. In between, the flag serves its
default variation without the experiment counting anyone. That doesn't matter in this
tutorial, but it's good practice with real traffic.

Then, send this prompt to your agent:

```text title="Prompt 6: Start the experiment" lines wrap theme={null}
I've turned the flag on in test. Start the experiment.
```

Your agent may read the flag again and find a new flag version, because turning on the
flag changed it. If so, it updates the draft to that version and tells you, then starts the
experiment. It may ask you to confirm once more. If it does, reply "Start."

When the experiment starts, your agent reports its status, the time it started, the
50/50 split, the primary metric, and a link to the results.

> [!NOTE]
>
> **If the flag is still off**
>
> The skill never turns on the flag without your permission. If the flag is still off when
> you ask to start, your agent explains that turning it on serves the variations to visitors,
> and asks whether you want to turn it on yourself or have it do it. It waits for your answer
> before it changes the flag.

To confirm the experiment is running:

1. Click **Experiments** in LaunchDarkly. The experiments list appears.
2. Click your experiment. It shows as running.
3. Reload `http://localhost:3000`. The button now shows either "Start free trial" or
   "Try it free for 14 days," depending on which variation your browser receives.

To learn more, read [Turning flags on and off](https://launchdarkly.com/docs/home/flags/toggle.md) and
[Managing experiments](https://launchdarkly.com/docs/home/experimentation/manage).

## Step 6: Send simulated traffic

A real experiment needs real visitors over days or weeks. To see how results work in
this tutorial, you simulate 500 visitors instead. The simulator makes visitors who see
"Try it free for 14 days" click about 30% more often. This way, the results will
show a clear difference.

If you cloned the companion repo, the demo box and simulator are already built. Skip the
prompt and continue with "Try the demo box."

Send this prompt to your agent:

```text title="Prompt 7: Add demo box" lines wrap theme={null}
Add a small "Demo only" box that switches between test users so I can show
different users getting different button text.
Then add `npm run simulate`: 500 fake users, some clicking, one variation 30%
more often. Make sure the events are sent before it exits. Label it as simulated data.
Don't run it yet; I'll run it myself.
Use the SDK key already in .env (don't print it), and don't change the flag,
metric, or experiment in LaunchDarkly.
```

Your agent adds a "Demo only" box to the signup page and an `npm run simulate` command.
It doesn't run the simulator or change anything in LaunchDarkly. It may restart the page
to show the box.

### Try the demo box

The demo box shows how the flag splits visitors. LaunchDarkly assigns each test user to
one variation, and the same user always gets the same one.

To try it:

1. Reload `http://localhost:3000`.
2. Choose a test user in the "Demo only" box. The page reloads as that user.
3. Note which button text the user sees.
4. Choose other test users until you find one who sees the other button text.

### Run the simulator

Run the simulator in a second terminal. This keeps the signup page running in the first.

To run it:

1. Open a new terminal in your project folder.
2. Run `npm run simulate`.

The simulator prints a summary when it finishes: about 500 users, split roughly evenly
between the two variations, with "Try it free for 14 days" clicked more often. It also
confirms that it sent all of its events before it exited.

Results take a few minutes to appear in LaunchDarkly. Wait three to five minutes before
you continue to Step 7.

To learn more about how many visitors a real experiment needs, read
[Sample size calculations for frequentist experiments](https://launchdarkly.com/docs/guides/experimentation/sample-size-calc.md).

## Step 7: Read the results

The results page compares the two variations on your primary metric and shows how
confident you can be in the difference.

To open the results:

1. Click **Experiments** in LaunchDarkly. The experiments list appears.
2. Click your experiment.
3. Click the **Results** tab.

If the results show zero users, wait a few more minutes and reload the page. LaunchDarkly
processes events in batches. Results may take several minutes to appear.

![The results table after 500 simulated visitors, with "Try it free for 14 days" ahead of the control.](https://mintcdn.com/launchdarkly/R1usXG6yY_Qx_Id2/images/tutorials/prompt-ab-test/exp-result-table.png?fit=max\&auto=format\&n=R1usXG6yY_Qx_Id2\&q=85\&s=48b838fce277a407fafb48aa41b2fea1)

The results table after 500 simulated visitors, with "Try it free for 14 days" ahead of the control.

Here is how to read each variable, using numbers from one run of this tutorial:

<table>
  <thead>
    <tr><th>Variable</th><th>Example</th><th>What it means</th></tr>
  </thead>

  <tbody>
    <tr><td>Sample size</td><td>260 and 245</td><td>How many users saw each variation. This includes the simulated users plus your own visits.</td></tr>
    <tr><td>Conversion rate</td><td>39.85% and 53.88%</td><td>The share of users in each variation who clicked the button.</td></tr>
    <tr><td>Relative difference</td><td>+34.69%</td><td>How much better the treatment did than the control. 53.88% is about 35% higher than 39.85%.</td></tr>
    <tr><td>Confidence interval</td><td>9.29% to 60.10%</td><td>The range of relative differences that fits the data. The whole range is above zero, which means the data points to the treatment being better.</td></tr>
    <tr><td>p-value</td><td>0.007</td><td>How surprising the difference would be if both buttons performed the same. A value below the significance level of 0.05 means the result is statistically significant.</td></tr>
  </tbody>
</table>

In this run, "Try it free for 14 days" won. Its relative difference of about 35%
is close to the 30% the simulator was set to produce. Your numbers may differ slightly,
because each simulated click is random.

> [!WARNING]
>
> **Simulated data isn't proof**
>
> You told the simulator to favor "Try it free for 14 days," so it was always going to
> win. This result shows how the results page works, not which button real visitors
> prefer.

With real traffic, you read the results the same way. Before you choose a winner, make
sure that:

* The experiment has run long enough to include different days of the week.
* The primary metric shows a statistically significant result.
* Other metrics you care about, like completed signups, haven't gotten worse.

To learn more, read [Analyzing experiments](https://launchdarkly.com/docs/home/experimentation/analyze) and
[Statistical methodology for frequentist experiments](https://launchdarkly.com/docs/guides/statistical-methodology/methodology-frequentist.md).

## Recap and next steps

In this tutorial, you ran an A/B test without writing the code yourself. Your agent
built the signup page, created the flag, and set up the metric and experiment.
You wrote the design, approved it, turned on the flag, and chose when the experiment
started. The `launchdarkly-experiment-setup` skill waited for your approval at
each of those points.

That split is the point. Your agent handles the setup that often stops people from
testing at all, and you keep the decisions that make an experiment worth running.

Here are some ways to keep going:

* **Test a real decision.** Point the same prompts at a decision in your own app, like a
  headline, an onboarding message, or a pricing page. Change the flag, the variations,
  and the event to match.
* **Let a real experiment run.** With real visitors, give the experiment enough time to
  include different days of the week before you read the results.
* **Stop the experiment and choose a winner.** When you're ready to decide, stop the
  experiment in LaunchDarkly and choose the winning variation. To learn how, read
  [Managing experiments](https://launchdarkly.com/docs/home/experimentation/manage).
* **Clean up the tutorial experiment.** Stop the experiment, then turn off the
  `cta-button-text` flag in your test environment. This returns the demo page to
  "Start free trial."

To find every prompt from this tutorial, read `PROMPTS.md` in the
[companion repo](https://github.com/launchdarkly-labs/prompt-ab-test). The `finish`
branch has the completed signup page and simulator.

To learn more about Experimentation, read:

* For the video tutorial, watch [Prompt an A/B test using Cursor and LaunchDarkly](https://launchdarkly.com/docs/tutorials/videos/prompt-ab-test.md).
* [Designing experiments](https://launchdarkly.com/docs/guides/experimentation/designing-experiments.md)
* [Bayesian versus frequentist statistics](https://launchdarkly.com/docs/guides/experimentation/bayesian-frequentist.md)
* [Measuring Experimentation impact with holdouts](https://launchdarkly.com/docs/guides/experimentation/holdouts.md)