Getting started with funnel experiments

If experimentation is a way of measuring the impact of a change made, funnel experiments calculate the best versions in a series of events or user flows. If you’re playing drop-off detective, funnel experiments can pinpoint exactly where users vanish into the abyss.

This tutorial will teach you how to set up, run, and analyze funnel experiments in LaunchDarkly. This tutorial pairs well with our guide to experimentation, but that guide is not a prerequisite for learning from this tutorial.

The relationship between metrics and experimentation

To declare any type of experiment victorious, we need clearly defined metrics. Metrics provide the data necessary to evaluate the impact of our experiments and make informed decisions. Metrics are like the heartbeat of your application’s performance. They measure audience behaviors influenced by feature flags. Metrics align with your business goals and help you track the impact of different flag variations over time, such as how often customers access a URL or how long a page takes to load.

LaunchDarkly has four primary flavors of experimentation events: custom events, conversion events, click events, and page view events.

Follow along with this tutorial using the Galaxy Marketplace sample application.

Experimentation in Galaxy Marketplace

First, clone the Galaxy Marketplace Example App from LaunchDarkly Labs.

All requirements are included in this project’s configuration; however, you will need to have NPM installed on your machine.

Run npm-v to verify if it’s installed. If not, follow these instructions to install it on your machine.

Next, install the Galaxy Marketplace app locally by running npm install on your terminal.

Then, run the application by running npm run dev. This will start the Next.JS app.

When successful, the following should appear in your terminal.

If you navigate to http://localhost:3000, you should see the Galaxy Marketplace pictured below.

We’ll now need to connect this environment to our LaunchDarkly application.

First, open your LaunchDarkly application or create a new account through a 14-day free trial.

Under the application, create a new project. Name it Galaxy Marketplace. Watch as this auto-populates into the project key.

Now that the project has been created, we must integrate the relevant keys into our application.

By default, you’ll be presented with two environments and two sets of keys: Production and Test.

Copy the Production SDK Key, then navigate to the application’s .env.example file.

First, rename the file to .env, then paste the SDK key in the respective LD_SDK_KEY place. Subsequently, repeat the process for the NEXT_PUBLIC_LD_CLIENT_KEY.

Setting up your feature flags

LaunchDarkly has already been installed within this application, and the marketplace application’s code already has the flags in place. We’ll need to add the flags to our LaunchDarkly application.

<<Want to know how to set up your first feature flag? Check out our feature flag docs.>>

First, navigate to the flags tab within your LaunchDarkly application and click Create flag.

Our first flag will turn on a new store header on the VR Galaxy store within the Galaxy Marketplace application.

Here are the details behind the first flag you should create:

Flag Name: Featured Store Headers
Key: storeHeaders
Description: Headers to drive engagement on stores
Flag Type: Custom Flag
Type: boolean
Variation 1:
Name: Available, Value: true
Variation 2:
Name: Unavailable, Value: False
Client-side SDK: checked

Next, we’ll create a flag that adds text to these banners, highlighting various promotions that we run. This will allow us to compare different versions or promotions.

Name: Store Highlight Text
Key: storeAttentionCallout
Description: header text for store headers
Type: string
Variations
First Variation
Name: Control
Value: New Items
Second Variation
Name: Sale
Value: Sale
Third Variation
Name: Final Hours
Value: Final Hours!

Outlining our experiment

Before any decisions or metrics are established, it’s important to take a page from our 6th-grade science books and ask: What is the hypothesis?

In this scenario, you’re working on an ecommerce site (Galaxy Marketplace, to be exact!) and seeking to understand two primary questions:

Does a promotional banner improve the rate of someone completing the checkout flow
What messaging improves the average cart order value the most?

Considering these two questions, we hypothesized that adding a ‘Final Hours’ banner to an item may increase the average cart checkout rate.

Now that we’ve established our hypothesis—it’s time to put it to the test.

The expected “successful” user journey from website visit to checkout is as follows.

We must identify these events within the application’s code, track them, and build a connection between the application’s code and LaunchDarkly.

This process first starts by creating a metric in LaunchDarkly.

Navigate to Metrics Tab: Head over to the metrics tab in LaunchDarkly to see all existing metrics.

Creating a metric in LaunchDarkly

Create a Metric: Click on the option to create a new metric. Define its event kind (custom conversion, click, or page view) and set an event key for tracking.

Be sure not to forget the event key—we’ll be referencing it within our project’s code!

Each metric will include the following.

Set Success Criteria: For numeric metrics, define what success looks like. Is it a higher number or a lower one? It will be higher today, and we want to see how many more people did the following action.
Define your Event Key: This unique identifier will track specific user actions. Set Success Criteria: For numeric metrics, define success (e.g., an increase in the average conversion rate).
Configure Measurement: Configure how the metric is measured. Choose the analysis method; for funnel experiments, this must be set to “mean.”

Tracking events in your application

LaunchDarkly uses events within your application to create metrics. From a development perspective, this looks like a .track event call to log specific user actions or events within the application.
Running a .track event call allows you to better understand user behavior by pulling context into LaunchDarkly.

In the existing codebase, this is the .track call that pulls in data from the application and begins to affiliate it with the “store-accessed” key.

async function storeOpened() {
   LDClient?.track("store-accessed", LDClient.getContext(), 1);
 }

Creating your experiment

The Experiments tab displays all experiments in your environment. This is where we’ll build and review existing experiments.

Let's now build out our funnel experiment.

Navigate to the Experiments Tab: Go to the Experiments section in LaunchDarkly.
Create an Experiment: Click ‘Create Experiment.’
Enter Experiment Details: Name your experiment “Checkout Flow Text” and fill in the hypothesis for the experiment created.
For this experiment, adding a ‘Final Hours’ banner to an item may increase the average cart checkout rate.
Select Experiment Type: Choose ‘Funnel optimization’ as the experiment type.
Choose Randomization Unit: From the Randomization unit menu, select a context kind. For today’s experiment, we’ll use the ‘user’ context where you want to evaluate an individual user’s behavior.
Select Attributes: You can optionally choose up to five context attributes to slice results. If you’re curious about how two groups may perform against one another—say, mobile users vs. desktop users—you can add additional attributes here to evaluate. For this first funnel experiment, we’ll leave this blank.
Select Metric Group: ‘Create Metric Group’ to create a new metric group. (We’ll dive deeper into this part below.)
Add Metrics: To the group, add the metrics representing each step in the customer journey. Ensure that each step is mandatory in the user’s journey and exists in the correct order.
For our checkout flow, we’ll be adding each step in the user journey from landing on the page to completing checkout: Store Accessed, Item Added, Shopping Cart Accessed, and Customer Checkout.
Choose Flag Variations: Select a flag to use in the experiment or create a new flag.
undefined
Set Audience: Select the ‘user’ audience; we want to target this to all users and specify each variation's traffic percentage. For simplicity’s sake—let's move forward with the 33%, 33%, 33%, and 1% variation. Then, select which variation LaunchDarkly should serve to the remaining population.
Finish Experiment Setup: Click ‘Finish’ to complete the setup. You are returned to the experiment’s Design tab.

About Funnel Metric Groups

For a funnel experiment, you’ll need a Funnel Metric Group. A Funnel Metric Group is a reusable, ordered list of metrics representing mandatory steps in the customer journey. Follow these steps to create a funnel metric group:

Navigate to Metric Groups: Find the Metric Groups tab in the Metrics section.
Create a New Metric Group: Click ‘Create Metric Group’ and and give it a name.
Add Metrics: To the group, add the metrics representing each step in the customer journey. Ensure that each step is mandatory and in the correct order.
Save the Group: Save your funnel metric group for use in experiments.

Connecting metrics to features

We can demonstrate impact by connecting metrics to feature flags.

Feature flags allow you to control the rollout of different feature versions, providing a mechanism to test changes with a subset of users safely. When combined with experimentation, feature flags become potent tools to validate hypotheses and measure the impact of those changes.

On the final step of setting up your experiment, you’ll be prompted to connect a flag, determine the percentage of folks who will receive each version of the flag, and choose the percentage of users who will participate.

In the scenario below; the feature flag creates a promotional banner on the item, with the “Control” without any banners and the other banners being “Sale” or “Final Hours.”

Running the experiment

For an experiment to begin recording data, the flag used in the experiment must be on, and the experiment must also be started.

Toggle the Flag: Navigate to the Feature Flags section and turn on both of the feature flags that you have created.
Turn on Experiment: Return to the Funnel Experiment and click start.

Reviewing experiment data

Once your experiment has concluded, it’s time to dive into the data. In LaunchDarkly, navigate to the Experiments Tab where your experiment is listed, then click on Results.

View Metrics: Look at the metrics you linked to your feature flag. These metrics provide the data needed to evaluate the performance of each variation.
Compare Variations: Compare the performance metrics of the different variations.

Check out our docs for a full deep dive into our experiment data.

Key metrics to consider

Conversion Rate: The percentage of users who completed the desired action (e.g., clicking a button, making a purchase).
Click-Through Rate (CTR): The percentage of users who clicked on a specific element, such as a CTA button.
Engagement metrics measure user interactions, such as time spent on a page or the number of pages visited.
Bounce Rate: The percentage of users who leave the site after viewing only one page.

Analyzing the data

With the data in hand, the next step is to analyze it to determine which variation performed better.

Statistical Significance: Ensure that the results are statistically significant. This means the observed effect is likely not due to random chance. LaunchDarkly provides tools to help assess this.
Performance Comparison: Compare the critical metrics for each variation. Look for significant differences that support your hypothesis.
Contextual Factors: Consider external factors that might have influenced the results, such as seasonality or concurrent marketing campaigns.

Drawing conclusions

After analyzing the data, conclude your hypothesis.

Confirm or Refute Hypothesis: Determine whether the data supports your original hypothesis. What banner led to the highest conversion rate?
Identify Winning Variation: Identify which variation performed better based on the metrics.
Document Findings: Document your findings and insights gained from the experiment. This will be useful for future reference and for sharing with your team.

Deciding what's next

Based on the conclusions drawn from your experiment, decide on the next steps.

Roll Out Winning Variation: If one variation outperformed the other, consider rolling it out to all users. In LaunchDarkly, you can update the feature flag to serve the winning variation to 100% of users.
Iterate and Improve: If the results are inconclusive or suggest further testing, plan additional experiments. Based on the insights gained, refine the variations and metrics.
Communicate Results: Share the results and decisions with your team. Ensure everyone is aligned on the implemented changes and the reasons behind them.

Implementing changes in LaunchDarkly

To implement changes based on your experiment results in LaunchDarkly.

Update Feature Flag: Modify the feature flag settings to reflect the winning variation. In this scenario, it may be that the winning variation is the `Final Hours` banner. In this situation, we could update the feature flag to show the `Final Hours` banner on all items.
Adjust Targeting Rules: Update the targeting rules to include all users or specific user segments as needed.
Monitor Post-Launch: After rolling out the change, monitor key metrics to ensure the desired outcomes are achieved and there are no unforeseen issues.

Why use funnel experiments?

Unlike traditional A/B testing focusing on a single variable change (such as a checkout cart button or headline tweaks), funnel experiments examine user flow. To keep the detective analogy going, A/B testing is like a magnifying glass looking at a single piece of evidence, while funnel experiments are like a crime scene investigation.

Funnel experiments help determine the best user flow from initial engagement to the final desired action. Through implementing funnel experiments, we better understand where folks are dropping off in a process, increase conversion rates, and prioritize team efforts where they matter most.

Identifying drop-off points

Funnel experiments spotlight where users are bailing. Maybe they’re leaving at checkout or dropping off after the second page. Identifying these points helps you zero in on problem areas.

Improve conversion rates

Understanding where users get stuck or frustrated can lead to targeted improvements, such as smoother user journeys and higher conversion rates. Think of it as transforming pothole-ridden roads into smooth highways.

Prioritizing team efforts and resources

Funnel experiments help prioritize your team’s work. Instead of guessing where improvements are needed, you’ll have precise data directing you to the most impactful areas. It’s like having a GPS for your development roadmap.

Congratulations—you just set up your first experiment with LaunchDarkly

Knowing how to use funnel experiments turns you into a full-fledged drop-off detective, prepared to crack the case of where your users vanish into the digital ether.

Funnel experiments are more than just another tool in your experimentation toolbox—they’re essential for those who want to understand and optimize for the entire user journey. While A/B testing gives you a magnifying glass for single changes, funnel experiments provide relevant data across a user flow.

If you have any questions, comments, or curiosities about this tutorial, feel free to contact me in the LaunchDarkly Discord or via email at emikail@launchdarkly.com! I’d love to help.
In the meantime, experiment with confidence, and may your metrics guide the way!

How to Use Funnel Experiments in LaunchDarkly

Like what you read?