While hiking with my quarantine partner recently, our conversation turned towards our work, and I found myself explaining LaunchDarkly over the course of about 20 minutes. While my partner is a brilliant woman with an engineering degree from arguably the best engineering school on earth, she doesn't have a background in software development, so this was an interesting challenge.
It occurred to me how useful it would be to capture some of what we discussed, as there is a surprising lack of content out there to help non-technical business people understand software development best practices, DevOps, and related topics. How can this be? Doesn't everyone agree that “every company is a software company” these days?
Business leaders of all types must know the basics of software creation, maintenance, and improvement if they are going to stay relevant and successful in this economy.
So please, allow me, a Philosophy grad from Alabama, to teach you, the non-technical business rockstar that you are, about one aspect of software delivery that has far-reaching consequences: feature flags.
What is so hard about software?
"Software" is basically just a bunch of text files. Some of them are more human-readable than others, some software has a greater or lesser number of these files and so on, but at the end of the day that's what it is.
Creating a software application is very literally all about collaborating with teams to write a bunch of these files, putting them in specific virtual locations, and connecting them together in different ways. When you are learning about systems, problems, and solutions in software development, you will have an easier time if you keep this idea very present in your mind.
Consider: writing an article is easy. Writing an article that gets wide readership is hard. Writing an article that gets published in Nature, a publication with millions of readers, is so hard almost no one achieves it despite a lifetime of trying. But at the end of the day, it's still just text. What makes getting published in Nature so much harder is the expectations of the audience.
Software is the same way. Anyone can write an app these days. Writing one that meets the expectations of users is extremely hard. Most fail. And unlike Nature articles, where getting published represents a kind of "finish line," in software, the equivalent success—say, getting in the upper right of a Magic Quadrant or #1 spot in the App Store—is really only the beginning, as users' expectations continually change, and competitors emerge hot on your heels. Writing a commercial software app is like writing a Nature article that you have to constantly update to avoid getting de-published.
Software factories
Imagine writing that big article with one other person and what kind of problems might emerge. Who does what part of the piece? How do you avoid stepping on each other's toes or accidentally contradicting each other in the article? Are you using Google Docs or Microsoft Word? Now imagine having hundreds or thousands of co-authors. Yeah, those problems only get worse. That's what software teams have to deal with.
To create apps that meet users' expectations well enough and consistently enough, without driving employees completely bonkers in the process, certain techniques are used. Much of this was inspired by the world of physical factories and, as you might expect, these techniques change over time as new ideas become more practical and fashionable.
For now, all that's important to understand is that, as code is written, it is carefully controlled with different systems to minimize bad things like costs and risk, while maximizing good things like revenue and customer satisfaction. It's also important to note that, like with physical factories, software factories also try to automate as much as possible, both because it's cheaper in the long-run and because humans will quit if you make them do too many things that can be automated.
Improvements = (good) changes
If an app gets better, that means the experience of using it has changed for the better. From a user perspective, changes might look like faster load times, better ease of use, or a whole new feature. From a developer's perspective, these changes are called "releases," and there are three fundamental ways to do them, each with pros and cons.
1. Deployments: change the application code itself.
This is the most obvious way of getting changes into the hands of your users but also the riskiest. It is widely understood by developers that any time code changes, there is the unforeseeable risk of problems. Careful methods of modifying code, dividing labor, and testing are done to minimize that risk, but it's far from zero. Changing code is also the most burdensome on users because they have to download an update or refresh the app for the new code to take effect.
When things go wrong, software teams scramble to push fixes or revert back to a previous version of the software (called a "rollback"), but it's not always fast or easy to do. Changing code is also usually all-or-nothing in that the same code goes out to all users at the same time.
2. Configuration change.
Remember how software is a bunch of text files that are connected to one another? Configuration can be thought of as the ways in which those files are connected. When you change a configuration, you're usually changing a file or setting outside of the software itself that controls things like what server to download from, what security credentials to use, or what database to reference. You may detect some similarities between configuration and feature flags in the next section (and there is some overlap), but configuration is best used for changes that occur infrequently and typically only when the user boots up their app initially.
3. Feature flags: remotely enable or disable code that is already in the application.
For most software teams, this is a "best of both worlds" approach to getting changes into users' hands. It entails using a system that hides code until they are ready to un-hide it. By doing this, developers can deploy code all the time without waiting for a feature to be fully done yet, which speeds up their work significantly.
Feature flags also give developers surprising new superpowers. Here are just a few:
- Release a software change to a small set of users first to make sure it doesn't break before releasing to everyone else. That's called a "canary release."
- Disable software components when problems happen without taking down the whole app. For example, an ecommerce company can disable their checkout system (or route to a backup system!) when they see certain errors without taking down their whole website or shopping cart. This is called a "kill switch." This is also way better than having to do one of those rollbacks we mentioned earlier.
- Give features to some users but not others. For example, if you have some Premium users with access to special things, feature flags are a great way to manage that. These are called "entitlements." Also, why not try releasing variations of a feature to different users to see which version performs best? Feature flags let you do this handily too—we call this experimentation at LaunchDarkly.
Let non-engineers control features. I did this at my last role in sales at CircleCI. When someone bought a Premium plan, I could enable those features for the customer without asking an engineer to do anything. With a good feature flag system, marketers, product managers, customer success reps, etc., can do a lot of things themselves that they'd ordinarily need an engineer to do for them.
Show me the money!
This sounds great, you're thinking, but so did the Juicero. What's the real-world evidence that feature flags are worth investing in? I'll paraphrase from a few of our own case studies:
- TrueCar, a massive digital auto marketplace, previously deployed new code once a week, with individual engineering teams taking as long as three months to launch new features. But with LaunchDarkly feature flags, they're now getting new code out the door 20 times a week.
- Intuit (makers of TurboTax) had an in-house feature flagging system that, among other things, would take up to ten minutes to process feature changes (e.g., when an engineer disabled a feature, it could take ten minutes for that change to fully register). After using LaunchDarkly, that ten minutes dropped to 200 milliseconds, and Intuit was also able to run vast experiments involving thousands of users at a time.
- Atlassian (makers of Confluence, Jira, and Trello) used LaunchDarkly feature flags to run experiments that, in turn, boosted customer satisfaction. Net promoter scores (NPS) increased in the very first year. And Atlassian's engineers were happier since the average time to resolve issues (MTTR) went down by 97%.
In the above cases, it's not unreasonable to assume that LaunchDarkly feature flags resulted in millions of dollars of ROI.
Why doesn't everyone use feature flags?
Well, understand first that most developers are using feature flags in one way or another. They may call them by other names ("toggles" or "switches"), but at this point, using feature flags is a best practice. That said, just as how factories in the real world are far from perfect, software teams are forced to deal with tight timelines, limited resources, and systems built by predecessors who may not have been the most competent.
What usually ends up happening is that companies leave it up to individual engineering teams to decide if, when, and how to use feature flagging. You don't have to be an MIT grad to predict the end result of that: inefficient, cumbersome, hard-to-use feature flagging systems that might do one or two things pretty well, but many other things not-so-well.
Until recently, it was incumbent on engineering leadership to care about feature flags, foresee their future importance, and enforce a shared standard across the board. They had to build these systems internally, and only those companies that could throw hundreds of full-time engineers at the problem came out ahead, like Facebook and Amazon.
Luckily, for everyone else, there's LaunchDarkly.
LaunchDarkly enables all teams to deliver and control their software with feature flags. You can create, see, and manage flags at scale as well as use flags for complex use cases and experimentation. And you can do all this in one intuitive platform. This frees you from having to devote several full-time engineers to maintaining an unreliable in-house feature flagging system.
So, there you have it, non-technical people. In sum, feature flags enable software teams to build and deliver new features (i.e., value) faster, with far less risk than would normally be the case.
This translates to dramatic gains in productivity, customer satisfaction, and revenue.
I'm confident you know what those things are.
.webp)
