Loom runs more experiments, increases product engagement

Loom is the video communication platform for async work that helps companies communicate better at scale. Loom makes it easy to record quick videos of your screen and camera and instantly share them with a link. More than 14M users across more than 200k companies around the world trust Loom to share feedback, updates, intros, training, and more – every day. Founded in late 2015, Loom has raised $203M from world-class investors including Andreessen Horowitz, Sequoia, Kleiner Perkins, Iconic, and Coatue. To learn more please visit www.loom.com

Loom’s Growth team, a band of product managers, software engineers, and data scientists, has a singular focus: to increase user acquisition and engagement. With that end in mind, they devise the product roadmap, build new functionality, and improve existing features.

Early on, the team wanted to adopt a culture of experimentation and inform product decisions with data. What reinforced this desire was, in the past, they had run experiments on features they were confident would perform well. But the experiment results disproved their assumptions. Experiences like these strengthened the Growth team’s resolve to pursue data-driven product development.

In the past, they weren’t shipping experiments as often as they would have liked. One manual procedure, in particular, slowed them down. 

Developers had to write custom logic when setting the inclusion criteria for an experiment—i.e., the targeting rules dictating which end-users should and should not see the features in an experiment. This alone was a hassle. But to make things more challenging, the logic would sometimes malfunction. It would bar intended users from experiments while granting access to unintended users. Developers would then have to troubleshoot the issue.

The Growth team had already been using LaunchDarkly for release management. They leveraged feature flags to progressively deliver new functionality, which increased their deployment velocity and lowered their risk profile.  

After gaining a deeper understanding of experimentation and targeting in LaunchDarkly, they began getting more creative with their experiments in the platform. 

Here’s a typical experiment workflow they now follow with LaunchDarkly. Product managers set the parameters for an experiment. Developers then build one or multiple variations of a new feature and wrap them in feature flags. They set fine-grained targeting rules in LaunchDarkly to ensure the right users see the right feature variations in the experiment. Moreover, with LaunchDarkly, they can restrict the experiment to a small subset of users. 

Then, developers deploy the experiment to production. If a bug arises, they can hit a kill switch in LaunchDarkly (i.e., toggle a flag) and resolve the issue in real-time. Otherwise, they proceed with the experiment and gather key performance metrics as they go. The data generated from the experiment gets streamed to Segment, then to Snowflake, and then ultimately to Tableau, where data scientists analyze the results. 

This workflow is more efficient than what they had previously. 

In using feature flags to run experiments, Loom is able to integrate experiments with the software delivery process. That is, the experimentation and software delivery workflows are one; they aren’t siloed. The Growth team can thus gradually roll out features and run experiments on those features—all at once, all in production, all in LaunchDarkly. 

Not long ago, they planned to unveil a substantial product change that would have a broad impact on the user experience. Rather than release it to the full audience in one big precarious splash, they rolled it out in phases. Using custom targeting in LaunchDarkly, they gradually released the new features to 75K specific Loom Workspaces over the course of a few days. 

Simultaneously, they ran experiments on those features within the designated Workspaces. They gathered customer feedback, tracked usage metrics, and measured the impact on system performance. They then iterated on those features based on the qualitative and quantitative data they collected. 

The launch was a success. And it was less stressful than it might otherwise have been.   

Loom now runs more overall experiments on application features than before. And they deploy the winners from those experiments in less time.

In the past, when engineers had to troubleshoot targeting issues, it could delay an experiment by 2-3 weeks in some cases. But now leveraging LaunchDarkly’s powerful targeting rules, the team can ship experiments in a single day.

LaunchDarkly helps Loom continue to deliver value to millions of people across the globe who rely on—and love—their video communication platform.