At LaunchDarkly our mission is to empower all teams to deliver, control, and measure their software.
With feature flags at the core of everything we do, we’ve continued pushing new limits and expanding the power of our platform. Today, we’re excited to announce a major evolution—LaunchDarkly Experimentation.
Driven by feedback from hundreds of customers looking for a better way to run practical, meaningful experiments, LaunchDarkly Experimentation blends our expertise in seamless feature delivery with experimentation best practices to provide flexible and intuitive experiments that any team can interpret and take action on.
Why experiment? And… why aren’t more teams experimenting?
Elite organizations have been running hundreds or thousands of experiments a year, finding optimizations and new opportunities that accelerate their business growth and provide an edge over their competition. Even small wins pile up for big gains in revenue, activation, or performance rates. But this success comes with a steep price tag—these same organizations also hire massive teams of data scientists and engineers dedicated to building and maintaining homegrown experimentation tools.
For the vast majority of organizations, the full benefits of experimentation remain out of reach. Taking the time and resources to build out a specialist team is a distraction from core business goals, and experimentation tools currently on the market introduce overhead and constraints that often kill experimentation programs before they get off the ground.
In our conversations with product teams looking for a better way to unlock the potential of experimentation, we’ve seen them run into three main challenges:
- Experiments are disconnected from the engineering process, adding complexity and risk
- Setting up valid experiments requires too much pre-existing expertise, burning time and trust
- Results aren’t easily actionable, and leave teams lost, frustrated, or winning experiments un-launched
We have the solution.
LaunchDarkly removes experimentation barriers
By offering an easy-to-use experiment setup in a tool your development team already has in their workflow, LaunchDarkly allows all teams to be part of the experimentation process.
Experimentation as a natural extension of your software delivery workflow
All experiments require the same three components: a change you’re introducing, the audience who will experience the change, and the metric(s) to track how the change impacted your audience.
With feature flags at the core, experimenting becomes a natural extension of your existing software delivery workflows. Each feature flag is a control point in your code used to manage when a feature is made available, and to whom it is available. Anywhere you use a feature flag is an opportunity to run an experiment.
LaunchDarkly allows you to set up, and manage, an experiment all within the same tool your development team is already using. And rolling out a winning variation takes just a few clicks to instantly launch your improvements.
Anyone can set up a valid experiment with LaunchDarkly’s help
Setting up an experiment should be easy. You just need a change, an audience, and a metric and you have an experiment—right?
It turns out, things aren’t always so simple for aspiring experimenters. Teams of all types shared with us that most tools required an unrealistic level of statistics and data science knowledge just to get started.
LaunchDarkly takes the hard work out of setting up a valid experiment so anyone can do it. With our in-house experiment experts (and regular input from customers) we’ve baked best practices for experiment design into our experiment setup flow and added protections to help teams avoid common pitfalls when editing an experiment’s configuration or audience allocation.
In the screenshots below, you can see LaunchDarkly’s experiment builder that guides experimenters through a simplified workflow to create their experiment, and the resulting experiment details page that provides a single place to review and manage an experiment.
Experiment results your team can easily understand and react to
Experimentation offers the promise of true data-driven decision making: increase your level of certainty in every feature release, and quantify the impact any one change has on your goal metrics.
But, many teams report challenges when it comes to attaining and interpreting traditional experiment results. Estimating sample sizes, interpreting p-values, and reaching statistical significance require more statistics knowledge than most teams want—or need—to know. At the least, these are barriers to entry that require lots of training or self-teaching. Worse, many teams can never reach statistical significance in experiments because they have low usage volumes or noisy metrics.
LaunchDarkly’s experiment results provide teams easy-to-understand probabilities—if any team sees “Variation #2 is 72% most likely to be the best,” they will be empowered to make reasonable business decisions. Furthermore, if your team decides to roll out a winner, it’s just a few clicks in the experiment to instantly turn it on for your customers. No code changes required.
In the screenshots below, you can see an example of LaunchDarkly’s simple decision-making probabilities that make experiment results approachable for most teams. Bring Product, Engineering, Marketing, and other leadership stakeholders into the conversation with actionable insights in easy-to-read displays.
What’s next?
This is only the start of our work to provide better ways for teams to measure the impact of their releases and react to what they learn more quickly. Next up, LaunchDarkly will give teams the ability to monitor metric performance thresholds and automate release rollbacks, along with more advanced metric analysis and ingestion capabilities to extend what’s possible to measure. Stay tuned!
How do I get access to Experimentation in LaunchDarkly?
Experimentation is available to purchase as part of LaunchDarkly’s Pro and Enterprise plans. If you’d like to start measuring the impact of your releases, contact our Sales team or your customer success rep.
And, as always, we’re always listening for your feedback at product@launchdarkly.com
