Galaxy Brain is a blog post series covering previous talks at our annual user conference, Galaxy.
How do you launch a major feature to all of your customers simultaneously without risk? How do you manage that launch across platforms, device types, and app versions without compromising your customers' experience?
The answer is through better feature management, which allows developers to deploy code to production more frequently and release without risk. Your feature management platform should also enable targeted product experiences for any customer segment based on any attribute—giving you more control and better customer experiences without burdening engineers.
In the case of streaming service Hulu, the team had outgrown its homebuilt feature flag system and wanted to move to a release process that was more scalable and safe.
Senior Software Engineer Matthieu Labbe and Client Architect Dominic Hopton shared the before and after of their release process leveraging LaunchDarkly Custom Attributes at our Galaxy conference. Watch in full below and keep reading for the highlights of their talk.
"That wasn't fun": What Hulu's release process used to look like
At the time, Hulu had five distinct code bases with five different applications, running in many different situations on many different devices. They also have a lot of versions—as many as 30— in production at the same time, which makes managing their features and experiences more complex than it may otherwise seem.
Dominic described launch day activities that may sound familiar: "At 11:59 on the day of launch, we wanted to be able to get to the new experience just a minute later … We wanted a big bang release, we wanted to be able to control that layout, that rollout, and we wanted to be able to pause it, or roll it back if we had any issues."
What does a major new UI release look like when you have around five applications available through multiple app stores, running multiple pre-release experiments, with many versions, and with complex CMS configurations to make the right experience available to the right customers?
"We had an 18-page agenda, starting at midnight," Dominic explained. "We had minute-by-minute responsibilities with people doing certain things in certain places to try and get our apps out the door. That wasn't fun. No, thank you."
"A crude tool": The trouble with Hulu's homebuilt flag solution
"The reality is, with no flags, you have no scale," said Dominic. "We ship to millions of users. Not all of our features are available on all of our platforms. Our customers don't receive the updates simultaneously. And when you include third-party app stores and the potential for issues after releasing to customers, it's just not possible to keep doing that."
Hulu was using feature flags, but as Matthieu explained, the practice for years had been to stash flags in its device config service.
Here's how it looked before: "We represent each device type by a unique integer and we have about 200 so far, with about 50 still active in production. And we use that to map device type to device properties using service configuration. And that information is used across the whole ecosystem for many different use cases, ads, and payback. Only a few services directly ask for the information, but it's used everywhere.
"So, after 10 years and thousands of commits, what does it look like? We have more than a hundred top level config keys, and we have over 300 total config items."
In addition to what looked like flags, there were also endpoints, strings, JSON blobs, and so on. "Config is code, so flipping a flag should be easy. Just one line change, one PR. No," said Matthieu. "It's at least a few PRs because we have a staging and a production PR … it works, but it's a crude tool."
Matthieu explained that making any change had to be all or nothing on a per-device-type basis. No gradual rollout or fine-grained targeting. This was a slow, human-driven workflow, even for simple changes.
"When we find a bug in the client," Matthieu said, "it can take just a few hours to fix it but then releasing the feature can take weeks because we need to let the client be updated."
"Can we scale this to a billion devices? Yes."
Hulu set up LaunchDarkly users to represent devices rather than people. Because those devices are used by people (Hulu customers), they defined custom attributes for their LaunchDarkly users to identify customers without personal identifiable information (PII). They mapped Hulu concepts to LaunchDarkly concepts, using custom attributes again to define information like device type and anonymous user data per LaunchDarkly user.
"Can we scale this to a billion devices?" Matthieu posed the question. "Yes, because it doesn't scale with the number of devices or the number of users. The state just grows with the number of LaunchDarkly rules and flags that we have."
What did these changes mean for that big UI launch?
"We had six clients with multiple versions in multiple states of implementation and experiments running all at the same time," Dominic said. "This is gonna be really hard, right? No. We were able to use five rules with five segments configured using custom attributes from our servers to target all the rules in the right places at the right time."
Dominic goes into more detail about how those segments and rules were structured for the launch of their new UI in the talk, so check out the video to see how those worked. Dominic summed up the lesson they learned from moving to LaunchDarkly and shipping their new UI with just a few clicks:
"LaunchDarkly users don't have to be people—they can be whatever you want them to be," he said.
