SmugMug increases conversions with experimentation and A/B testing

SmugMug, Inc. is the largest, most comprehensive photo management platform in the world, serving millions of photographers and billions of photographs. It provides photographers a safe, easy, and convenient way to protect, share, showcase, and even sell their photographs. In 2018, SmugMug acquired Flickr, a thriving global photography-centric community, from Yahoo, Inc.

SmugMug is a subscription-based service, and its revenue is based on the number of people who convert from a free trial to a subscription plan. In order to improve its subscription rate, the site constantly experiments with different trial offers, site designs, and features, and compares the sign-up rates among them. For example, it may vary the site flow that people follow to upgrade from a trial to a subscription. SmugMug constantly performs these kinds of experiments and A/B tests. They carefully record and analyze all metrics to determine which flows and features should be put in place permanently.

SmugMug also uses A/B testing to improve the usability of its site, for example, cutting down the time it takes for users to perform actions such as uploading photos or creating photo websites. It compares metrics for different flows and chooses the ones that are most effective.

SmugMug had built a homegrown A/B testing framework using PHP, but for each experiment, a good deal of new code needed to be written in order to customize the experiment. This took a substantial amount of time, and product managers needed to rely on engineers in order to have flexibility and granular control over the tests they ran.

SmugMug wanted a solution that would not just make its A/B testing more effective but would also help manage releases and overall operations. It was looking for a platform that would allow it to use feature flags to test releases in production before they are turned on for customers. To do that, it chose LaunchDarkly.

With LaunchDarkly, product managers now have full control over A/B testing without having to ask engineers to write code. Product managers only have to log into a control panel, set the conditions of the tests, and then make them live. They get full granular control over how the tests are run, down to the level of determining which customers get the new features or design being tested, and which customers will be used as a control group.

"Whenever someone does something as basic as click a button, the click is tracked, and we can use that information in our A/B testing," Sheryl Lun, Senior Front-End Engineer at SmugMug, explained. "That gives our data team a lot more information, so they can get faster and more accurate results about the tests."

SmugMug also uses LaunchDarkly when pushing new code to production. It wraps new features in a feature flag and keeps them turned off while QA tests them. Once any bugs or issues are fixed, the flags are switched so the new features go live, bug-free, at a faster pace than previously possible.

With LaunchDarkly, product managers can run more A/B tests and quickly iterate on them. They collect more and better data about what features, designs, and workflows increase user satisfaction. For example, they have used A/B testing to help determine whether offering new users a survey about why they signed up for SmugMug would increase or decrease engagement.

In addition to A/B testing, rolling out new features to specific subsets of users via feature flags has helped improve overall SmugMug usability. When SmugMug decided to improve its Lightbox feature that displays photos, it used feature flags to first introduce the new capabilities to its most advanced users, gathered feedback from them, and then used that feedback to improve Lightbox before turning it on for everyone else.

Releases are now higher quality because they have been fully vetted by QA in production and are only turned on via flags when QA ensures all issues have been resolved.

The end result of all this? A better, more usable interface and service for millions of photographers—and turning happier customers into subscribers.