In a recent blog post, our CEO Dan Rogers outlined a vision for how tech orgs can make painful software releases a thing of the past by releasing progressively, automating monitoring and rollbacks, targeting audience segments, and using runtime configuration management.
Software bugs are inevitable (in other words, ship happens), but they don’t have to cause costly incidents and outages for most of your users. Before we get into mitigation strategies, let’s cover three core feature management capabilities:
- Decouple deploy from release: You can ship a software change to a production location without exposing it to production traffic.
- Targeting: You can release that change to a subset of users for safe, controlled testing in production (shrinking the blast radius).
- Kill switch flags: If that change degrades performance, you can disable it in runtime—without having to roll back an entire release package or shipping a fix through your deployment pipeline.
Using a combination of feature flags and targeting rules (as well as some other features we’ll cover), you can achieve high innovation velocity while reducing risk.
1. Release progressively
You might not be able to prevent a software release from crashing a machine or causing an issue, but you can put strong guidelines in place to minimize any resulting impact.
Releasing progressively means incrementally increasing the number of users exposed to new features so that you can catch issues before they affect a significant portion of your user base. Here’s an example of how it can work using LaunchDarkly:
A new feature starts with a single line of code behind a flag in your Dev environment. After the engineer working on it has completed development of the feature and confirmed that things work as expected, it’s turned on for all developers in that environment. Then, the new feature enters a formal testing phase, where multiple teams QA the feature to uncover bugs and ensure that it passes acceptance criteria.
Once this is complete, the testing phase can be marked as done, and the team can proceed to the next phase, where the feature is released to internal users in production.
After validating that everything looks good, the new feature can then be turned on for a subset of end users. This stage could take the form of a percentage-based release (in which the release is progressively turned on for defined percentage increments of users) or a canary release, in which the release is turned on for specific cohorts of users, machines, or other definable attributes.
Finding the right progressive release strategy requires assessing your organization’s existing release processes, end user base, and time constraints to find a solution that works for you. Not all types of releases require the same release process, and Release Assistant enables engineering leaders to embed best practices into reusable release pipelines so changes (whether it’s a new product or a small feature update) can follow a successful path to production.
2. Proactive monitoring and automatic rollbacks
There are many solutions to help you understand if errors are occurring in your software, like application performance monitoring (APM), infrastructure monitoring, and user feedback monitoring (LaunchDarkly can integrate with many existing APM tools).
But unfortunately, issues within a small rollout will often fail to trigger existing alerts, and the triggered alerts can be difficult to attribute to a feature flag change. So how do you catch issues within all the noise? With Release Guardian, you can proactively uncover and remediate software issues before they become customer issues.
Here’s how it works: If you’re planning to release a new feature, you can define key metrics that help closely track changes and trigger error alerts if something goes wrong. Once your software is released (let’s say, as part of a progressive release to a small subset of your user base), Release Guardian will monitor those metrics, identify exactly which flag change caused the issue, and notify the relevant teams via Slack or email to take action with a one-click rollback.
3. Targeted releases and customized experiences
You can target releases and experiences based on anything you know about a customer. Targeting is a key part of progressive releases, but you can also use targeting in LaunchDarkly to customize software experiences for your end users. For example, you may want to offer different experiences based on:
- Geo regions and compliance: Adapting features to align with local regulations or legal requirements.
- Subscription tiers: Offering different features or entitlements to users based on their subscription level.
- Software and device versions: Customers of your own software or machines running various operating systems.
- Personalization: Customizing the user experience on different user roles or preferences.
- Early access programs (EAP): Selectively exposing a new feature to consumers who have opted in, with the intention of collecting product feedback.
- Bug bounties: You may want to expose functionality to a subset of users to try to break or find exploits in your software.
The ability to turn software on and off for various audiences is a great help when things go wrong. If you’re releasing software to machines using different operating systems and versions, you can define each cohort of machines as an audience segment in LaunchDarkly.
Now, let’s say you have a bug that only appears to be affecting machines running a particular version of one specific operating system. With LaunchDarkly, you can turn that release off for only machines running that operating system version, instead of rolling it back for all machines on an operating system, preserving the user experience for the rest of your audience.
4. Runtime configuration management
Software deployments and releases are far from the only ways a software issue can manifest. Changes to configuration files are another way for bugs to reach end users.
By placing feature flags strategically throughout code, LaunchDarkly allows users to augment configuration management by dynamically updating application configurations (using multivariate flags and targeting rules), which helps when you need one of the following properties:
- Dynamic: The configuration file values can be changed at runtime.
- Delegated: For when you need to delegate control of a configuration to another team. For example, your security team may increase the log verbosity for features that are being tested by bug bounty hunters, or support may use this to manage approval requests.
- Contextual: Sometimes, you need to make decisions based on attributes such as device information, user location, etc.
- Time-bound: You need to temporarily change values in response to changing business conditions.
Managing configuration values via feature flags enables runtime control for situations like API rate limiting or enabling easy-to-use approval processes. It also gives you audit logs and the ability to measure the impact of configuration changes.
If, for example, your API is getting throttled from a DDoS attack, you normally have to change a configuration and redeploy your application. However, if you assign a multivariate feature flag to your rate limiting rules, you can adjust your rate limit during an attack instantly, and without redeploying, so that you can maintain system availability for all customers.
De-risking your software release management practices requires an alignment between the people building and releasing software, the processes they follow, and the tools they use. If you’re interested in learning more about how to mitigate the inherent risk associated with software releases, we’re happy to discuss how we can help.
