Why Decouple Deployments From Releases?

Software developers decouple deployments from releases because it reduces risk in their software release process considerably. This gives them the confidence to ship code to production faster. Decoupling deployments from releases refers to separating the process of deploying code to a production environment from that of releasing the code (feature) to end users. Traditionally, those processes are coupled: you release a new feature by deploying code to production at the same time. 

Decoupling deploy from release offers many benefits:

• Improves stability by controlling timing and having instant remediation in case of errors
• Increases velocity by releasing features immediately without waiting on app builds 
• Lays the foundation for targeted rollouts that let you gradually release features to a subset of users
• Lays the foundation for trunk-based development and continuous delivery
• Reduces developer stress and burdensome inter-team dependencies

In this article, we’ll dive into why tightly coupling deploy and release is the wrong path for modern software delivery, how feature flags help you decouple deploy and release, and what benefits LaunchDarkly feature flags can add to your release process.

In the above image, we've deployed a code artifact to a production server but have only released it to internal test engineers and developers. The code artifact (feature) is hidden from users.

The problem with coupling deploy and release

In today’s software-driven world, organizations tend to center around a key moment in time: the software release. Releases are tricky. Introducing new functionality or capabilities into an application increases the potential for outages and other issues. Developers then spend their time debugging issues and patching problems instead of innovating or further improving their products. The result is that organizations deploy new code to production less frequently, and development starts to stagnate. 

In a lot of instances, deploy and release are considered largely the same process. I release a new feature by deploying the updated code to production. And until recently, that was how things were handled. In order to actually see the changes that developers had been working on, you had to deploy them to a production environment. However, this creates a few problems:

• New code is available to everyone immediately, creating risk 
• Time-consuming rollback processes for remediation 
• Delays and missed deadlines for releases 

Because of these issues, organizations prioritize low-risk releases to ensure a good user experience. They slow down their delivery process and group multiple releases together instead of continuously delivering new features and updates. While this alleviates some of the problems listed above, it actually increases the complexity of the releases meaning that if something does go wrong, resolving that issue becomes even more time consuming and has broader impacts.

How decoupling solves these challenges

Taking a step back, let’s remember why we couple releases with our deployments: we need to deploy the new code to production in order to release the feature. What if instead, we could still deploy the new code to our production environments, but rather than immediately release that new feature, we control when the new code becomes available to customers? We can, with feature flags. 

Feature flags are a gating mechanism for when we deploy multiple variations of code and want to control which variation is available to which customers. In pseudo code, it looks something like:

This is a very basic structure for decoupling deploy from release. Both our new component and our old component have been deployed, but until the feature flag, in this example called flagValue, is set to true, the new component is not released to our users. This process is called dark launching, and is where LaunchDarkly gets its name.

Why does separating deploy from release matter?

Now you may be thinking, “that sounds great, but why does that matter? What benefit does releasing a feature with a feature flag offer vs. my old release process?” These are fair questions, so let’s address them by looking at how feature flags impact releases in two ways: release velocity and release stability.

Release velocity

Without decoupling deploy from release, organizations typically opt for extensive pre-production testing. As a result, release velocity starts to slow down and it takes longer and longer to move new features through a release pipeline. The thought here is sound, more pre-production testing helps reduce the risk of buggy code getting sent to production. But this false choice of having to choose speed or safety is exactly what feature flags help eliminate.

Without feature flags, we don’t have a way of controlling when the new features are actually released outside of deploying the new code. As a result, organizations reduce the number of times they rollout features in order to minimize the number of potential problems a release could cause.

A feature flag in LaunchDarkly with one targeting rule created and adding a second.

With feature flags, we no longer have that concern because the new functionality is not released until we toggle the flag on. Organizations can write, test, and deploy code at whatever speed works for their development teams and then actually roll out features whenever it works best for them. Additionally, because the new code already exists in the application, releases can happen in a matter of milliseconds. LaunchDarkly feature flag changes are delivered in under 200 ms, without having to rebuild an application or re-trigger a CI/CD pipeline. This ability to instantly make new code active actually plays into the second impact, release stability.  

Outside of pure speed improvement, using feature flags enables organizations to expand who can play a role in the release process. In the past, developers not only had to push the code changes, but were the ones ultimately responsible for the release. Since the code has been deployed, enabling the feature flag can be done by other stakeholders including product managers, operations teams, or whoever should have the final say over when a release should happen.

Release stability

Stable releases are safe releases. Now that we know we can release frequently and control when features become available, seeing changes in a matter of milliseconds, the next piece to consider is how do we prevent those frequent releases from causing frequent problems. Well, if we consider that a release is being made available with a feature toggle, then we have to assume the inverse also holds true. You can revert a release with the flip of a feature toggle. And that’s what feature flags enable—instant remediation without having to roll back or redeploy an application. If something goes wrong or we discover a new bug with the code in our production environment, we can revert the changes instantly and go back to a working version of our application. 

Having a low-risk release process is critical as release velocity accelerates. This especially holds true when you think about microservices or Kubernetes environments. In Kubernetes, applications are broken apart into individual components that all rely on one another. If something breaks in the microservice architecture it can have cascading effects on unintended parts of the application. For example, a simple API update might cause an error on one of the frontend services resulting in an outage for some customers. With so many moving parts, it may be difficult to locate the exact service that is causing the issue. Using a feature flag, we can turn off the feature across all replicas of that API without having to redeploy it.

Diving into the details

Let’s further explore how feature flags help improve release stability and velocity. One of the earlier issues with coupling deploy and release that we mentioned earlier was the idea that code becomes instantly available to everyone with access to the production environment. If there’s a bug, everyone sees that bug. We talked about how feature flags help you control the speed and when a feature is released, but we can also determine whom the features get released to.

Targeting rules

LaunchDarkly feature flags have the ability to target end users based on any type of attribute. We do this through a feature called custom contexts (learn more about targeting with custom contexts). The gist of it is that you determine how the accessors of your application, be it users, devices, stores, or anything else, are identified and grouped based on specific characteristics or attributes. 

To put this into perspective, let’s say we are releasing a feature that we want to only give access to users that have opted into an early access program. Using LaunchDarkly we: 

• Group those early access users in a segment 
• Create a targeting rule that serves the new variation only to that segment
• Enable the feature for those users whenever we desire by toggling the flag on

Should we encounter an error, no problem, we turn the flag off and remove the new capability. Targeting rules let us control the impact radius of a feature release, making it even easier to move faster knowing there’s a safety net in place.

Canary releases

At this point, you may be wondering how this differs from another release term: canary releases. In practice, using feature flags is very similar to the purpose of running canary releases. In both cases, you’re pushing out new capabilities to a production environment and slowly introducing that new functionality to users, testing along that way that things are operating correctly. The difference is that with canary releases, you are maintaining two environments: the old version and the new version. Managing this type of overhead can be unnecessary in some cases especially for small changes.

Feature flags, on the other hand, control variations of code within the same application. You can test and release new features without having to redeploy the application or update networking rules (e.g., like using a load balancer). Instead you use the custom contexts feature that we mentioned earlier to control your audience size and progressively deliver new features.

LaunchDarkly contexts unlock limitless possibilities for targeting, this video provides an overview of how to use different contexts for targeting purposes.

Kill switches and flag triggers

The capability of a feature flag to instantly turn off broken code is sometimes referred to as a kill switch. However, kill switches serve a slight different purpose beyond feature releases. Kill switches are typically permanent flags, meaning that they will live on in the code base and cover a broader feature set, like putting a site into maintenance mode or closing access to a third party service. But kill switches and release flags both serve a purpose in helping stabilize our release processes.

LaunchDarkly supports a variety of flag types including release, kill switch, experiment, and migration flags.

A couple of times we’ve discussed the benefits of being able to instantly remediate issues without redeploying applications by toggling flags on and off, but what if we could automate this process? Using a tool called flag triggers, we can have LaunchDarkly listen for events that indicate there is an error from the application either through a direct API call or a third-party integration with an APM (Datadog, Dynatrace, Honeycomb, etc.). We already use APMs to detect abnormalities or outages in our application and with flag triggers we can extend those monitoring capabilities to eliminate the cause of the problem without a bigger incident.

LaunchDarkly flag triggers support integrations with many popular Application Performance Monitoring tools (APMs).

Culture impact of decoupling deploy and release

We’ve discussed why we should decouple deploy from release, how we decouple them and the benefits of doing so, but we haven’t touched on what decoupling deploy from release means for your organization. To put it bluntly, decoupling deploy from release helps organizations achieve the vision described in the promise of DevOps. 

Truly embracing DevOps means committing to a continuous deployment process. As outlined in the State of DevOps report, elite teams deploy new code daily, if not multiple times a day. Not only is this a process shift for many organizations, but a cultural shift as well. In order to encourage development teams to deploy code with more regularity they need to know there are guardrails and safety nets for their production environments. Knowing that deploying new code will not automatically trigger a release provides that safety net. Developers work at their own speeds and release can still be on a controlled cadence to the right audience at the right time.

Success stories of development teams decoupling deploy and release

1. Paramount improves developer productivity by 100x
2. HP standardizes and scales fast, low-risk releases
3. Bayer’s digital farming arm improves reliability for critical apps
4. Jack Henry drives innovation in financial services
5. The Centers for Medicare & Medicaid Services (CMS) builds better software faster with less risk

If reading this article has inspired you to decouple your deploys from your releases, but aren’t sure how to get started, read this article on the "Five Stages of Feature Flag Adoption". Or visit our website: launchdarkly.com