IBM Enjoys Significant Cost-Savings and “Unceremonious” Releases with LaunchDarkly

IBM has been an enthusiastic LaunchDarkly customer for several years now. Most recently, for example, Michael McKay, a Senior Development Manager at IBM, spoke about his love of feature flags at LaunchDarkly’s annual software industry conference. We are exceedingly grateful for IBM’s business.

In their early days as a customer, the IBM Kubernetes Service team, a branch of IBM Cloud, used LaunchDarkly to increase their deployment frequency from twice a week to 100+ times a day. Since then, IBM has expanded its use of LaunchDarkly to the wider IBM Cloud product delivery team.

Most recently, they’ve used LaunchDarkly feature flags for two use cases in particular:

1. Blocking bad actors from accessing the free tier of IBM Cloud; and
2. Performing phased rollouts of new data centers.

Blocking web domains that abuse the free tier of IBM Cloud

IBM offers a free version of its IBM Cloud infrastructure as a service (IaaS) offering. Periodically, bad actors will abuse this free service and spin up clusters for mining cryptocurrency. 

In the past, IBM engineers would have to follow a series of sometimes inefficient steps to resolve the issue. Roughly 10 people, including developers, site reliability engineers (SREs), and team leads, would drop what they were doing to put out these fires. Upon identifying the offending web domains, engineers would have to change environment variables and update configuration files to deny access to these domains, submitting these changes as pull requests (PRs). Once a senior person approved the changes, engineers would merge the changes with the master code branch (trunk) and then deploy to production at last. 

This resolution process took more time than IBM would have liked. And the company lost money every minute that a bad actor abused its free service. The organization was thus motivated to find a new solution. 

Thankfully, with LaunchDarkly, IBM can use feature flags as a kill switch and repel bad actors almost instantly. And they can do this without having to change configuration files and reboot their service, both of which eat up valuable time. Moreover, once a bad domain has been identified, IBM can create a feature flag rule that automatically denies service to anyone associated with the initial bad domain in the future. 

When it comes to “free tier abuse” incidents, IBM has dramatically improved its mean time to recover (MTTR) with LaunchDarkly. As such, the team saves considerable time whenever such incidents arise. Moreover, they have saved—and continue to save—a great deal of money by shrinking the duration in which unauthorized users eat up IBM’s computing resources. But perhaps the most significant benefit of using LaunchDarkly is IBM’s engineers have reduced the opportunity costs associated with these bad actor incidents.

“The main issue with our former approach to dealing with the free tier abuse cases had to do with opportunity costs,” explained Jeff Sloyer, a Senior Technical Staff Member (STSM) on the IBM Cloud Kubernetes/OpenShift team. “When we were putting out these fires, it meant we weren’t building new valuable products that could push the business forward. LaunchDarkly gives us our time back. It helps us avoid losing money in the short-term while creating more opportunities for us to make money in the long-term.”

Progressively delivering new cloud computing products and services

IBM Cloud’s product delivery team also uses LaunchDarkly for release management. Whenever they launch a new data center or cloud service, engineers will progressively deliver the service with LaunchDarkly feature flags. This gives them a profound level of control over the rollout.

First, the team will perform a canary launch, in which internal IBM developers test the new data centers in production. Once the service passes through this initial gate, the team will then expand the rollout to, say, a few local regions. And when the data centers meet or exceed pre-defined performance metrics at this stage, engineers will broaden the rollout further still. They do this until all intended users have access to the new service.

Such an approach embodies the heart of Progressive Delivery, a new software development lifecycle enabled by feature management. By progressively delivering its infrastructure offerings, which involves testing changes in production, IBM virtually removes any shred of risk from its releases. In fact, the product delivery team has reduced its “change failure rate” to zero percent, putting them among the elite of the elite software teams, according to the DevOps Research & Assessment (DORA) group (see chart on page 18).

Time and again, customers tell us that, before LaunchDarkly, software releases were agonizing and stressful. But now, they are so safe and predictable as to be boring—in the best way, of course. IBM, too, has enjoyed such an exodus of stress.

“In the past, we’d roll out a new data center,” said Sloyer, “and the whole time we’d be stressed out, thinking, is this going to work or not?. But with LaunchDarkly, our releases have become unceremonious. Whereas before, we would have waited till the middle of the night to go live, now, we’ll flip a feature flag in the middle of the day. We’ll do a full launch without any fear whatsoever. In fact, we haven’t had a bad launch day in over two years.”

IBM Cloud provides critical infrastructure to clients. And LaunchDarkly plays a critical role in ensuring IBM Cloud maintains peak performance for all users at all times.

Innovators in cloud infrastructure

LaunchDarkly has caused developer productivity at IBM to rise and development costs to fall. Moreover, LaunchDarkly has made releases safe, uneventful, and easy to manage. Lastly, it has cut down on system disruptions, thus allowing engineers to devote more time and energy to building valuable cloud computing products.

The team over at IBM Cloud continues to innovate and make great strides in the world of cloud infrastructure. We are fortunate to share in this journey with them.

Read the full IBM case study and more LaunchDarkly customer case studies.