I attended a really energizing conference (Unreasonable) recently that was focused on where behavioral science leaves the lab and meets real customers. The final speaker of the day was a professor of marketing – Ryan Hamilton – who helped crystalize something that had been missing from the way most people talk about the power of experimentation.
His closing argument told the now-famous story of the group of MIT students who crafted a system to beat the casino odds.
“Everything I know about counting cards, I read about in a book. This is an interesting book, and a mediocre movie, made about this real story of these MIT students who would go and count cards in Vegas. And what they would do is they would run a series of small experiments. So they would have somebody sit at the table and place very small bets and keep track of the cards that came through. And when the deck shifted such that they knew that there was a disproportionate share of face cards, they would signal their buddy who would then come down and sit at the table and place very large bets, right? You get to make the small bets if you set out to do so. Run your A-B test. Test these things in small increments so you can get an insight into reality before you place the large bets. And they really will dramatically reduce your risk.“
In my first article, I referred to a famous study that showed that trusting your gut can be wrong up to 90% of the time. But that fact shouldn’t be depressing: think of how much room you have to improve!
If you had trusted, high-quality data at your fingertips that showed you a direct comparison between control and treatment, do you think you could do better than making 1 correct decision in every 10? I bet you could. In fact, that is the kind of bet that we should be making. Often.
What if you could improve that ratio to 2 in 10, or 3 in 10, or even (gasp) 5 in 10 decisions? That should not be a difficult target to hit when you have good experimental data!
Think of the value that is out there to be captured if you could improve your win rate from 10% to 50%. Could you get that higher? With good data, you can do better than a coin toss when it comes to decision making.
The importance of making more, better decisions
What I’m talking about here is making a better decision: using causal data to inform your decision making, so that your decision aligns most closely with the customer behaviors you’re looking to change.
We want to be able to make more, better decisions.
There’s another reason to talk about making more decisions, and that has to do with how much better those decisions actually are. How much lift did we get from the experiment?
There is a saying in the experimentation world, which is also a universal truth: big changes are hard to make.
This saying applies to setting and measuring our expectations for our work as well. It is hard to roll out something so revolutionary that it moves any needle more than a few percent. Furthermore, those kinds of big changes tend to be more complicated to implement. They are larger, cross more technology and business barriers, and take a long time to get in front of customers.
When we spend a ton of time trying for perfection, getting a result that is just “better” can feel like a disappointment. But that’s where we need to flip the thinking a little because what makes an experimentation program really super-powered is…
Velocity: small wins add up to large gains
A really powerful experimentation program is looking to increase the velocity of small, incremental wins that add up to large overall changes to the business.
A simple example that demonstrates the power of a culture of experimentation shows the difference between a year of small (1%) increases compared to a year of small (1%) decreases.
Better, data-informed decisions result in positive business impact. Worse, gut-based decisions result in negative business impact
Taking action on this approach requires a bit of a change in the development process. It requires less certainty and more adaptability built into your applications. But, on the positive side, you should be able to work more scrappily!
You don’t need weeks of meetings to align upon “perfect” for any given feature. Instead, you can focus on where you might want to test different variations, build those decision points into your code, and set up experiments to test different versions and make fast decisions.
Think about the confidence you need to make your decisions. Not every decision holds the same risk, and that can be used to your advantage. You know your business, you know your application, and you likely have a really good idea of what changes are riskier than others. Not all decisions need to be made on the same level of confidence.
For less risky changes, you should be able to try—and try again and again—to find the best possible option. Make small, incremental improvements. Learn what is actually driving your customers’ behaviors, and then take those learnings into your next big project!
Which brings me back to the opening quote.
“Test these things in small increments so you can get an insight into reality before you place the large bets. And they really will dramatically reduce your risk.“
I’m not suggesting that you need to stop taking large chances. But your experimentation program should be leveraged to mitigate risk on bigger, more costly items, by giving you a lot of small, incremental improvements to make while larger, more complicated features development progresses.
That way, you have the ability to learn often, adapt quickly, and use those learnings as you design your larger, more impactful (and riskier) features.
Limit your risk by making more, smaller, faster decisions. Increase your bottom line incrementally with smaller wins, and use the learnings to drive decision-making on the bigger things.
It’s a winning strategy.
.webp)
