Feature Flags and Product Analytics for Better Experimentation and Testing

This piece was written by John Cook, Senior Product Marketing Manager at Heap. Heap is a product analytics platform that lets you track every touchpoint of a user’s journey across your web and mobile experiences.

If you’re reading this, you likely know something about feature flags. And it’s likely you’ve heard a little bit about product analytics. (If not, here’s a good intro: What Is Product Analytics?) What you may not know is that when you use both tools together across your mobile and desktop experiences, you get a very powerful mix of user data and product flexibility. What once might have taken an entire team days to figure out can now be understood in a few simple clicks.

By deploying product analytics across your web and mobile experiences, you can track every single touchpoint of a user’s journey and get in-depth user intelligence via session replays and heatmaps. Using a product analytics tool also lets you port your user data to a data warehouse, where it can be analyzed alongside other types of business data. 

Once product analytics are integrated with a robust feature management system, you’ll get real-time insights into every piece of your product’s lifespan, from cohort testing to full rollout. In this post, we’ll tell you why that matters for your company. We’ll also cover:

• The difference between feature flags and product analytics
• How feature flags and product analytics can help you iterate quickly, spot trends and anomalies, run experiments, and more
• How these two tools together can help you supercharge your A/B tests
• How different teams can make the most of using feature flags and product analytics together
• Why now is the perfect time to implement this product strategy

Feature flags vs. product analytics

A feature flag is a conditional statement (e.g., if-else) that controls the behavior of your software applications in runtime. Product delivery teams use flags to de-risk and accelerate releases, deliver targeted software experiences, and create a foundation for experimentation. 

Product analytics, on the other hand, show you exactly how users engage with your app and whether new features are being used in the way you initially intended. You’ll get clear quantitative data breaking all this information down in a way that removes friction from your quantitative analysis. The best platforms also give you qualitative tools like session replays, which let you view how exactly users have interacted with your product—what they clicked, what follow-up actions they took, and whether they were able to achieve their intended goals. 

Product analytics also give you critical information for tracking key metrics that show how your digital product is serving your company’s users, team goals, and bottom line.

Why add product analytics to feature flags?

While feature flags allow you to test new features with real subsets of users in a production environment, incorporating product analytics will give you an instant read on whether the new features are working as intended and can be ramped up to full release. This will help you: 

1. Run tests faster. You can use feature flags to test new features and validate (or invalidate!) your idea based on what product analytics show. If your feature is working, you can either instantly or gradually roll it out to your entire user base. If not, you can roll it back and develop it further.
2. Quickly spot trends and anomalies. Once you’ve toggled on a new feature for a particular set of users, you can use analytics to help you spot enlightening patterns in user behavior that either validate your hypotheses or show you paths toward improvement. If you detect any anomalies or problems, feature flags will let you quickly revert back to your original state. 
3. Run really robust experiments. You can also use feature flags to segmentize users into control groups and experimental groups before toggling a feature on. With product analytics, you can then see if your experimental group is experiencing the intended benefits of your new tool. This type of A/B testing can be done in multiple areas of your product at the same time without interfering with your app’s main trunk of live code.
4. Test web against mobile. If you use feature flags to deploy the same new feature across both your web and mobile experiences, you can use product analytics to see how user behavior differs across both experiences. This can help you deploy features for each experience in a way that complements user behavior (as opposed to forcing the same UX across both experiences, even if it isn’t the ideal solution).
5. Localize. Given that user segmentation is a key benefit of feature flags, you can build them into your localization and internationalization strategies to accommodate different regulations and behaviors in different countries. By adding product analytics into the mix, you can also test your product in different locales to make sure foreign audiences are engaging with it in the ways you expect.

Ways to combine product analytics and feature flags

When it comes to experimenting with a combination of feature flags and product analytics, your only limits are the parameters of your product. Below are a few examples of things you can do when combining these tools to maximize both your offerings and revenue:

1. Test free trials. If you’re offering free trials of your product, you might wonder how long these trials should be to maximize their chances of seeing full conversions. With feature flags, you can divide users into segments and show each group a different free-trial duration. With product analytics, you can then track which trial duration options get the most clicks and ultimately bring in the most revenue.
2. Improve QA (quality assurance). If you use feature flags to ramp up the release of a new feature, you can use product analytics to make sure users are interacting with it the way you intended. If everything seems to be going as planned, you can slowly increase the percentage of users who can see the new feature until it is released 100%.
3. A/B test package tiers. If your product or service offers multiple package tiers with differing features for users who want to pay varying amounts, you can A/B test different versions of your packages along with different price offerings. With product analytics, you can then track which packages or prices generate the most clicks, conversions, and revenue before you choose which offerings will ultimately be released 100 percent.

Ultimately, experimentation is all about proving the validity of your ideas. For software companies, user testing is key to iterating in a way that solves people’s problems and increases revenue.

How different teams can use feature flags and product analytics together

Using feature flags and product analytics together can benefit multiple teams at your SaaS company, including product, engineering, and marketing (particularly product marketing). 

Product

When product teams use feature flags and product analytics together, they can quickly identify pain points at different points in your funnel, design and build new alternatives, and test how effective they are by releasing them to small segments of users, whose every click can show whether the feature is adding value. 

Engineering

Engineers, who execute the product team’s designs and vision with code, are responsible for implementing feature flags and product analytics so they can work together. There are multiple types of flags they can implement, including:

• Boolean. These simple true-false flags can be built into code and used easily to turn features on or off for all users or segments of users.   
• Multivariate. These allow teams to define two or more custom variations to dynamically configure flag behavior. These are great for when you need a more nuanced approach to controlling what parts of the product your different segments of users see. 
• Temporary. These flags can be incorporated into lines of code on a temporary basis and removed after you fully release a feature.
• Permanent. You can mark flags as permanent when you intend to keep them in your code long-term, which can be great for toggling non-essential features on and off during spikes in traffic or other pressing situations. 
• Client-side. These enable flags on front-end applications with client-side SDKs for desktop clients and mobile devices.

Feature flags allow engineers developing in Continuous Integration and Continuous Delivery (CI/CD) pipelines to decouple deployment and release, which helps reduce the risk of system crashes when they roll out new features. Combining feature flags with product analytics uncovers additional benefits to feature rollouts. Whether users are participating in a simple A/B test or seeing more than two variations of a new feature, analytics will be key in helping engineers identify user behavior differences between each version. These behavior logs can expose bugs, unforeseen UX problems, and opportunities for improvement that eventually become part of a feature’s lifecycle. 

Product Marketing

If your company’s conversions are product led, it’s essential that you have a product marketing team that understands every aspect of your users behavior. Apart from needing to know what data points to track and how to manage the tracking process, product marketers can help their teams customize targeting of particular features; limit feature flags to instances that match particular business or data models; and release features at the user, device, and store levels.

Product analytics can then help marketers gain insights into whether these controlled rollouts are working correctly, whether they need to roll back, or whether further communication might be necessary to help users navigate the new features.

Why now is the time to start using feature flags and product analytics together

These days, software production costs are up, and teams need to do more with less. Feature flags and product analytics offer a shortcut to gaining essential knowledge that lets you do more with lower headcounts. 

For example, when using these tools together, teams can create consistent testing and development across both mobile and web. Data gathered from product analytics can also be used in scripts written for automation testing, which can allow teams to run preprogrammed quality assurance tests on all the touchpoints users interact with most. This will allow team members to focus on higher value-add tasks.

Bringing feature flags and product analytics together

Ready to start harnessing the combined power of feature flags and product analytics? Start integrating LanchDarkly with Heap to gain immediate insights from new features that you introduce into your product. Together, these tools can help you develop and test app features smarter and faster while always measuring their functionality against your company’s KPIs. 

Curious how to do product analytics the right way? Check out this article from Heap.