Every A/B test should have a hypothesis. This may sound obvious, but not all hypotheses are created equally.
The simple step of writing down hypotheses can elevate testing so that it becomes a more precise practice, where A/B tests play an integral role in product development. This can turn your A/B tests into another tool for using experimentation to your benefit—to stay in lockstep with what users actually want.
A hypothesis shouldn’t just be some blind statement about whether you think your metric will go up or down; it should also include a reason you think the metric will go up or down.
In other words, a good experiment hypothesis should make a statement about some mechanism of action:
- OK: “Treatment A will increase metric X”
- Better: “Treatment A will increase metric X because of Y”
Here’s why: the more thought and intentionality you put into even simple A/B tests, the better you will understand your product—and your users. You can carry these learnings into a deeper and more informed relationship with every part of the product experience.
For busy teams, A/B tests can sometimes become a “box to be checked” rather than a focused pursuit of valuable data. It’s tempting to just experiment unconsciously, coming up with ideas randomly and using feedback from an A/B test as a crutch to see what sticks.
Part of the danger has to do with the imperfection of metrics. A single metric cannot reasonably capture all the nuances of how your users might get value from your product. Letting A/B tests completely drive your decision-making puts you at the mercy of that metric. For that reason, experimentation requires some amount of taste (or intuition) to bridge that gap between getting feedback with respect to a single metric and delivering an overall delightful experience for users in the long run.
A simple measure for more focused A/B testing
The best safeguard against an unconscious, “see what sticks” A/B testing practice is to always have strong hypotheses about how your changes will impact your chosen metric. Grounding your experiment in user behavior prior to measurement allows the A/B test to play a more pointed role in confirming or challenging your product intuition.
For example, consider a simplified A/B testing scenario where I want to see if a blue button outperforms a red button with respect to click-through rate:
- Hypothesis A: “The blue button experience will have a higher CTR than the red button experience.”
- Hypothesis B: “The blue button experience will have a higher CTR than the red button experience because the background of the site is already red, so the blue button will visually jump out more.”
If the result of the experiment was to confirm the hypothesis, you might argue that the end result is the same. In hypothesis A, you would probably think about the result and might even come to the conclusion that the blue button was performing better because of visual contrast.
However, the crucial difference is that in hypothesis B, you had this idea prior to measurement, so it allows you to further refine your thinking and gain a deeper understanding. Rather than just thinking, “huh, that’s interesting, I wonder why it performed better,” your thought process would be more like:
- (positive result) “That’s exactly what I thought, and this leads me to my next idea for exploiting visual contrast to do [X]”
- (negative result) “I guess users don’t really find visual contrast that important for navigation, so we should pivot to exploring [Y] instead.”
When you experiment consciously, A/B test results contribute more explicitly to building your product intuition, rather than just helping you optimize towards a number.
Like what you read?
Get a demo
