If you want your experimentation program to grow and create tangible business results, you need a clear understanding of what to test, how to test it, and how to analyze the results. Some teams fall into the trap of testing everything, or testing the wrong things entirely. This can waste resources, generate inconclusive results, and damage stakeholders' faith in the experimentation process.
Experimentation at scale is about value, not volume. The goal is to run the right tests, using the right approach.
Why scaling experimentation matters
Let's be honest: none of us is clever enough to knock it out of the park with every experiment. A realistic goal for experimentation is to scale it so that marginal wins accumulate over time, creating a meaningful business impact. The more friction you can remove from the process, the easier it is for your team to implement experiments, and the more value you'll extract from them.
This means being strategic about three key areas: knowing when to experiment (and when not to), selecting the right tools and measurement approach, and translating results into action.
What makes an experiment high-impact?
Before you start testing, identify the characteristics that distinguish high-impact experiments from busywork. Here are four essential criteria:
1. High uncertainty
The sweet spot for experimentation is when you're unsure whether a change will help or hurt, but it could have a meaningful impact on customer experience or key metrics. These are typically highly visible changes or major modifications to something that has existed for a long time.
You may be certain there will be a high impact, but uncertain about the direction of that impact. That uncertainty is exactly what makes the experiment valuable.
2. Conflicting opinions or assumptions
When stakeholders disagree on the right path forward and there’s no clear evidence for the best approach, experimentation provides objective data to inform subjective decisions. This is where you can combat the "HiPPO effect" (the Highest-Paid Person's Opinion) and confirm that decisions are based on data rather than influence.
3. Random audience and exposure
This might seem obvious, but it's what makes experimentation scientifically valid. Without the ability to randomly allocate audiences and measure the results in parallel, you're just making comparisons among things that aren't exactly comparable.
The feature or experience should be randomly assigned to users, and the more exposure you can get, the faster and more reliable your results will be.
4. Measurable outcomes
If you can't clearly define what "better" looks like—whether that's conversion, engagement, latency, or another metric—you can't run a meaningful experiment. You need to identify measurable outcomes during your validation process, starting with the primary metric embedded in your hypothesis: "I think if I do this, this thing will occur."
If you can't measure the outcome, you're left with just feelings or assumptions. And while these are valuable, they won't help you measure the success of your experiment.
When NOT to experiment
Knowing what not to test is as important as knowing what to test. Here are four situations where you should skip the experiment:
- When the improvement is obvious. If a change is clearly positive or you're just fixing a bug, don't waste time testing it. You can always dig deeper with an experiment later if you see unexpected outcomes.
- Low-risk or low-impact change. Small UI tweaks or copy changes that won't move meaningful metrics probably aren't worth testing. However, be careful here; context matters. Changes to navigation text can have a significant impact, while changes to footer text may not. To make this determination, you need to understand how users engage with your site.
- Time-sensitive launches. If you don't have time to wait for results, you're wasting effort running the experiment. Understand the time impact and determine whether testing is feasible.
- Lack of data or metrics. If you can't track success or don't have enough users to detect a difference, testing won't be useful. Sometimes you know what you need to measure but can't track it yet (like leads that go to field sales). In these cases, add measurement capability to your roadmap and include the experiment in your problem library (defined below) for future use.
The experimentation lifecycle
After you've identified high-impact experiments, you need a consistent process for running them. Here's what a typical experimentation lifecycle looks like:
- Maintain a backlog of untested ideas (this is your “problem library”)
- Implement the new feature with proper instrumentation
- Design the experiment with clear success metrics
- Launch and monitor results as they come in
- Present findings back to stakeholders
- Roll out the winning version to all users
- Iterate on what you learned and start the cycle again
The key is running multiple experiments simultaneously at different points in this cycle, constantly iterating on what you learn.
Choosing the right tools and measurement approach
Your experimentation tool should work where your analysis is happening. There are typically three paths:
- Standard experimentation, where your tool handles both traffic assignment and analysis end-to-end. This works well for teams getting started or those who want a turnkey solution.
- Bring your own analysis for organizations that already have notebooks and BI tools in place. You still need reliable traffic assignment and risk mitigation, but you handle the analysis with your existing tools.
- A hybrid approach, where you warehouse all your event data in one place (like Snowflake) and your experimentation platform runs analysis against that dataset. This gives you the best of both worlds.
Turning results into informed decisions
Great experiment results require two things:
- Flexible statistical approaches
Different situations call for different statistical models. Sometimes you may never get enough traffic to justify a frequentist approach, and you’ll need to use Bayesian methods. Other times, organizational preference will drive the choice. The key is having both options available and understanding when to use each. - Consistent, comparable dashboards
Unlike product analytics, where you might create custom views for each analysis, experimentation dashboards should be consistent from one experiment to the next. This allows you to make direct comparisons and quick decisions across all your tests.
The power of data slicing
One advanced technique that can dramatically increase the value of your experiments is data slicing. While you typically want to run experiments on the widest possible group of users, you also know that subgroups within your sample might behave differently.
The most common slice is mobile versus desktop. People simply use phones differently from how they use computers, and in many cases, changes have completely different effects across these platforms.
You can also consider slicing by any categorical data, such as user tier, account age, location, or any other meaningful segment. The real power comes when you discover that different groups prefer different experiences—you can then create targeting rules to deliver the optimal experience to each segment without additional deployments.
Building your experimentation practice
Experimentation takes discipline and consistency. It requires building organizational muscle around hypothesis formation, measurement, and decision-making.
Start by auditing your current approach against these principles. Are you testing high-uncertainty, high-impact changes? Do you have clear measurements in place? Can you turn results into action?
Most importantly, focus on reducing friction in your process. The simpler it is for teams to run quality experiments, the greater the return on your work.
Like what you read?
Get a demo
