Prompt an A/B test using Cursor and LaunchDarkly

Product decisions often come down to opinion. “Which button text should we use?” usually gets settled by whoever has the strongest argument, not by which option performs best. The data-driven answer to these questions is to run an experiment. But designing, running, and analyzing an experiment can feel like a big project, especially if you’re not a data scientist. That friction is often enough to skip testing altogether. When teams ship changes based on guesses, they ship without a way to show the impact of their work. A coding agent can set up an experiment for you, but handing it everything is risky. An agent doesn’t know what’s worth testing or what success looks like. Left on its own, it can act haphazardly, starting an experiment or turning on a flag before you’re ready, which changes what your users see. In this tutorial, you use a coding agent and the launchdarkly-experiment-setup agent skill to run your first A/B test. Your agent handles the busywork: it builds a small app, creates the flag, and sets up the metric and experiment. You make the important decisions: you write the experiment design, approve it, turn on the flag, and decide when the experiment starts. When you’re done, LaunchDarkly results show which version performed better and how confident you can be. That way, the next debate about button text ends with data instead of opinion.

Prerequisites

Accounts and information

A LaunchDarkly account with Experimentation, and permission to create flags, metrics, and experiments.
A LaunchDarkly project to work in, and the SDK key for the environment you’ll use. This tutorial uses the test environment.

Tools

A coding agent that supports Model Context Protocol (MCP) and agent skills, such as Cursor or Claude Code, running on your computer.
Node.js long-term support (LTS) and npm.
Optional: Git, to clone the companion repo.

Connections and skills

The hosted LaunchDarkly MCP server, connected and signed in to your account. This lets your agent create the flag, metric, and experiment. If you have more than one LaunchDarkly plugin or MCP server installed, keep only the hosted server on. This means your agent uses the current skill.
The launchdarkly-experiment-setup agent skill, version 0.3.0 or later. You install this in Step 1.

Skip ahead with the companion repo: Clone the repo, check out the finish branch, run npm install, and add your SDK key to .env. Then install the experiment skill in the project folder, as described in Step 1, and start at Step 3. In Step 6, skip building the simulator, and run npm run simulate.

Step 1: Plan what to build with your agent

Start by deciding what to test, then ask your agent for a plan rather than code. Reviewing the plan first lets you check what the agent intends to build before it changes any files. To set up your project:

Create an empty folder for the project.
Open the folder in your coding agent.
Run git init in a terminal in that folder.
Install the experiment skill in the same folder:
npx skills add https://github.com/launchdarkly/ai-tooling --skill launchdarkly-experiment-setup
The skill is added to a .agents/skills/ folder in your project, where your agent reads it.
Start a new chat in your agent. This ensures it picks up the skill.

Send this prompt to your agent:

I want to run my first A/B test with LaunchDarkly.
Build a tiny Node.js app with one signup page and one button.
A LaunchDarkly flag called cta-button-text should pick the button text:
"Start free trial" or "Try it free for 14 days".
When someone clicks, send a LaunchDarkly custom event called cta-clicked.
For now, just plan the app; we'll create the flag and set up the experiment later.
Look up the official LaunchDarkly docs for every SDK detail, starting at
https://launchdarkly.com/docs/llms.txt. Plan first, don't write code yet.

Your agent replies with a plan and doesn’t create any files. The plan describes a small Node.js server that uses the LaunchDarkly Node.js server SDK. The signup page reads the cta-button-text flag for each visitor, and clicking the button sends a cta-clicked event for that same visitor. The plan ends by asking whether to build the page. Your agent’s plan may be worded differently, or use different file names. That’s expected. What matters is that it uses the flag key cta-button-text and the event key cta-clicked exactly as written, because the later steps depend on them. To learn more about the SDK your agent uses, read Node.js SDK reference (server-side). Next, have your agent build what it planned. You add your SDK key yourself. Do not paste it in chat. Send this prompt to your agent:

Looks good, build the app. We'll create the flag and set up the experiment in later steps.
Keep it as small as possible.
Create a .env file with a placeholder for my SDK key; I'll paste the real key in myself.
Add a .env.example with placeholders, and keep .env, .agents/, and skills-lock.json out of git.
Don't commit yet.
Add a README with setup steps. Then tell me how to run it.

Your agent builds the page, installs the SDK, and tells you how to start it.

Keep your SDK key out of the chatPaste the key into your .env file yourself. Your agent sends everything in the chat to its model provider. Never paste the key there.

To run the page:

Copy the SDK key for your test environment from LaunchDarkly. It’s the server-side key.
Replace the placeholder in your .env file with the key.
Save the file.
Run npm start in your terminal.
Open http://localhost:3000. The button reads “Start free trial.”
Click the button a few times. A thank-you page appears after each click.

To confirm the clicks are reaching LaunchDarkly, open Live events for your test environment. A cta-clicked event appears within a few seconds of each click. To learn more, read Live events and SDK credentials.

Step 3: Create the flag

Your agent creates the flag through the LaunchDarkly MCP server, and you confirm it in LaunchDarkly. Send this prompt to your agent. If your project has a different key, replace prompt-ab-test-tutorial with it:

Using the LaunchDarkly MCP server, create a string flag called cta-button-text
in my prompt-ab-test-tutorial project with two variations:
"Start free trial" and "Try it free for 14 days". Don't change targeting.

Before it creates the flag, your agent may ask you to approve the MCP tool call. Read what the call does, then approve it. Your agent creates a string flag named cta-button-text with the two variations. The flag is off in every environment and has no targeting rules. Your agent may also mark the flag as temporary or add tags. Those labels don’t affect the experiment. To check the flag in LaunchDarkly:

Click Flags. The flags list appears.
Click cta-button-text. The flag’s page appears.
Confirm the flag has two variations, “Start free trial” and “Try it free for 14 days.”
Confirm the flag is off in your test environment.

The "cta-button-text" flag in the test environment. It's off, with two variations: "Start free trial" and "Try it free for 14 days."

The flag starts off on purposeWhile the flag is off, LaunchDarkly serves its off variation, “Start free trial,” to everyone. You turn the flag on in Step 5, right before the experiment starts.

If you cloned the companion repo and started at this step, run npm start, open http://localhost:3000, and click the button a few times. The clicks send cta-clicked events, which the experiment skill checks for in Step 4. To learn more, read Creating new flags and LaunchDarkly hosted MCP server.

Step 4: Design and approve the experiment

This is the step you don’t hand off. Your agent can set up an experiment, but it can’t decide what’s worth testing or what success looks like. You write the design, and the launchdarkly-experiment-setup skill builds exactly that, only after you approve it.

Write your design

An experiment design answers a few questions before any data comes in. This table describes each part of the design in this tutorial:

Component	In this tutorial	Why it matters
Hypothesis	If we change the button to “Try it free for 14 days”, more users will click it, because it tells them exactly what they’re getting.	States what you expect
Flag and variations	`cta-button-text`, with “Start free trial” and “Try it free for 14 days”	Decides what each visitor sees
Control	”Start free trial”	The baseline that the other variation is compared against
Primary metric	The share of users who sent the `cta-clicked` event	The one number that decides the result
Randomization unit	User	Each visitor sees one version consistently
Traffic split	50/50	Sends equal traffic to each variation
Statistics	LaunchDarkly defaults	Frequentist with sequential testing, which lets you check results early without inflating false positives

Send this prompt to your agent. If your project has a different key, replace prompt-ab-test-tutorial with it:

Using the launchdarkly-experiment-setup skill, set up an experiment in my
prompt-ab-test-tutorial project, in the test environment. Here's my design:
- Hypothesis: If we change the button to "Try it free for 14 days", more users
  will click it, because it tells them exactly what they're getting.
- Flag: cta-button-text. Variations: "Start free trial" and "Try it free for 14 days".
  Control: "Start free trial".
- Primary metric: the share of users who sent the cta-clicked event.
  Create it if it doesn't exist.
- Randomize by user, 50/50 split. Keep the default statistics settings.

Review the agent’s design

Your agent checks that cta-clicked events are reaching your test environment, then shows you the complete design. It doesn’t create anything yet. The design lists every part from your prompt, plus the details the skill fills in. Any value the skill chose for you, like the experiment and metric names and keys, is labeled as suggested. Statistics appear as “LaunchDarkly default (frequentist, sequential testing).” Before you approve, check these parts of the design:

The metric uses the event key cta-clicked.
The metric measures the share of users who clicked, not the total number of clicks.
The control is “Start free trial.”
The randomization unit is user.

If anything is wrong, tell your agent what to change. It shows you the updated design and asks again.

What happens if something is missingIf your prompt leaves out part of the design, the skill asks for everything that’s missing before it shows a design. If the cta-clicked event isn’t arriving yet, the design includes a warning and lists the events that are arriving.

Approve and check the draft

When the design looks right, reply:

Yes, create it.

Your agent creates the metric and the experiment. The experiment is a draft and doesn’t collect any data until you start it in Step 5. To check the draft in LaunchDarkly:

Click Experiments. The experiments list appears.
Click the experiment your agent created. Its Design tab appears.
Confirm the hypothesis, the two variations at 50% each, and “Start free trial” as the control.
Confirm the primary metric is the one your agent created for cta-clicked.
Confirm the experiment shows as not started.

The Design tab for the experiment your agent created. It shows the hypothesis, the two variations at 50% each, and the primary metric.

To learn more, read Creating experiments and Bayesian versus frequentist statistics.

Step 5: Turn on the flag and start the experiment

Turning on the flag and starting the experiment are two separate decisions. Turning on the flag changes what visitors see. Starting the experiment tells LaunchDarkly to begin collecting data. The skill never makes either decision for you. To turn on the flag:

Click Flags in LaunchDarkly. The flags list appears.
Click cta-button-text. The flag’s page appears.
Choose your test environment from the environment menu.
Toggle the flag On.
Click Review and save.
Click Save changes.

Turn on the flag right before you start the experiment. In between, the flag serves its default variation without the experiment counting anyone. That doesn’t matter in this tutorial, but it’s good practice with real traffic. Then, send this prompt to your agent:

I've turned the flag on in test. Start the experiment.

Your agent may read the flag again and find a new flag version, because turning on the flag changed it. If so, it updates the draft to that version and tells you, then starts the experiment. It may ask you to confirm once more. If it does, reply “Start.” When the experiment starts, your agent reports its status, the time it started, the 50/50 split, the primary metric, and a link to the results.

If the flag is still offThe skill never turns on the flag without your permission. If the flag is still off when you ask to start, your agent explains that turning it on serves the variations to visitors, and asks whether you want to turn it on yourself or have it do it. It waits for your answer before it changes the flag.

To confirm the experiment is running:

Click Experiments in LaunchDarkly. The experiments list appears.
Click your experiment. It shows as running.
Reload http://localhost:3000. The button now shows either “Start free trial” or “Try it free for 14 days,” depending on which variation your browser receives.

To learn more, read Turning flags on and off and Managing experiments.

Step 6: Send simulated traffic

A real experiment needs real visitors over days or weeks. To see how results work in this tutorial, you simulate 500 visitors instead. The simulator makes visitors who see “Try it free for 14 days” click about 30% more often. This way, the results will show a clear difference. If you cloned the companion repo, the demo box and simulator are already built. Skip the prompt and continue with “Try the demo box.” Send this prompt to your agent:

Add a small "Demo only" box that switches between test users so I can show
different users getting different button text.
Then add `npm run simulate`: 500 fake users, some clicking, one variation 30%
more often. Make sure the events are sent before it exits. Label it as simulated data.
Don't run it yet; I'll run it myself.
Use the SDK key already in .env (don't print it), and don't change the flag,
metric, or experiment in LaunchDarkly.

Your agent adds a “Demo only” box to the signup page and an npm run simulate command. It doesn’t run the simulator or change anything in LaunchDarkly. It may restart the page to show the box.

Try the demo box

The demo box shows how the flag splits visitors. LaunchDarkly assigns each test user to one variation, and the same user always gets the same one. To try it:

Reload http://localhost:3000.
Choose a test user in the “Demo only” box. The page reloads as that user.
Note which button text the user sees.
Choose other test users until you find one who sees the other button text.

Run the simulator

Run the simulator in a second terminal. This keeps the signup page running in the first. To run it:

Open a new terminal in your project folder.
Run npm run simulate.

The simulator prints a summary when it finishes: about 500 users, split roughly evenly between the two variations, with “Try it free for 14 days” clicked more often. It also confirms that it sent all of its events before it exited. Results take a few minutes to appear in LaunchDarkly. Wait three to five minutes before you continue to Step 7. To learn more about how many visitors a real experiment needs, read Sample size calculations for frequentist experiments.

Step 7: Read the results

The results page compares the two variations on your primary metric and shows how confident you can be in the difference. To open the results:

Click Experiments in LaunchDarkly. The experiments list appears.
Click your experiment.
Click the Results tab.

If the results show zero users, wait a few more minutes and reload the page. LaunchDarkly processes events in batches. Results may take several minutes to appear.

The results table after 500 simulated visitors, with "Try it free for 14 days" ahead of the control.

Here is how to read each variable, using numbers from one run of this tutorial:

Variable	Example	What it means
Sample size	260 and 245	How many users saw each variation. This includes the simulated users plus your own visits.
Conversion rate	39.85% and 53.88%	The share of users in each variation who clicked the button.
Relative difference	+34.69%	How much better the treatment did than the control. 53.88% is about 35% higher than 39.85%.
Confidence interval	9.29% to 60.10%	The range of relative differences that fits the data. The whole range is above zero, which means the data points to the treatment being better.
p-value	0.007	How surprising the difference would be if both buttons performed the same. A value below the significance level of 0.05 means the result is statistically significant.

In this run, “Try it free for 14 days” won. Its relative difference of about 35% is close to the 30% the simulator was set to produce. Your numbers may differ slightly, because each simulated click is random.

Simulated data isn’t proofYou told the simulator to favor “Try it free for 14 days,” so it was always going to win. This result shows how the results page works, not which button real visitors prefer.

With real traffic, you read the results the same way. Before you choose a winner, make sure that:

The experiment has run long enough to include different days of the week.
The primary metric shows a statistically significant result.
Other metrics you care about, like completed signups, haven’t gotten worse.

To learn more, read Analyzing experiments and Statistical methodology for frequentist experiments.

Recap and next steps

In this tutorial, you ran an A/B test without writing the code yourself. Your agent built the signup page, created the flag, and set up the metric and experiment. You wrote the design, approved it, turned on the flag, and chose when the experiment started. The launchdarkly-experiment-setup skill waited for your approval at each of those points. That split is the point. Your agent handles the setup that often stops people from testing at all, and you keep the decisions that make an experiment worth running. Here are some ways to keep going:

Test a real decision. Point the same prompts at a decision in your own app, like a headline, an onboarding message, or a pricing page. Change the flag, the variations, and the event to match.
Let a real experiment run. With real visitors, give the experiment enough time to include different days of the week before you read the results.
Stop the experiment and choose a winner. When you’re ready to decide, stop the experiment in LaunchDarkly and choose the winning variation. To learn how, read Managing experiments.
Clean up the tutorial experiment. Stop the experiment, then turn off the cta-button-text flag in your test environment. This returns the demo page to “Start free trial.”

To find every prompt from this tutorial, read PROMPTS.md in the companion repo. The finish branch has the completed signup page and simulator. To learn more about Experimentation, read:

For the video tutorial, watch Prompt an A/B test using Cursor and LaunchDarkly.
Designing experiments
Bayesian versus frequentist statistics
Measuring Experimentation impact with holdouts

Prerequisites

Step 1: Plan what to build with your agent

Step 2: Build the signup page

Step 3: Create the flag

Step 4: Design and approve the experiment

Write your design

Review the agent’s design

Approve and check the draft

Step 5: Turn on the flag and start the experiment

Step 6: Send simulated traffic

Try the demo box

Run the simulator

Step 7: Read the results

Recap and next steps