Continuous integration and continuous delivery—or deployment—have become mainstays within the DevOps pipeline. Users expect functional deployment of new features, and developers expect these changes to keep their applications competitive against market alternatives. Therefore, it’s critical that these deployments are bug-free and don’t disrupt existing services.
A canary release helps avoid common problems associated with poor rollouts. These releases place a subset of users on the version with new features or content, while the majority of users remain on older production builds. This means that only a minimal number of users are potentially impacted by bugs and vulnerabilities, enabling development teams to collect useful feedback and refine their apps accordingly.
Keep in mind, though, that this testing environment isn’t a clone of your production environment. It’s a sliver of your existing infrastructure and has real-world significance.
This article explores the ins and outs of canary releases and how to implement them among your other CI/CD processes.
Why are canary releases important?
Developers and testers surveyed in 2020 found that untested or broken code accounted for sixty percent of known software bugs. While some launches are flawless, it can be impossible to know how new features behave until users put them through their paces. A nightmare scenario for teams is when an outwardly flawed application version is pushed to everyone simultaneously. Canary releases may also help developers test revamped infrastructure deployments within which their applications live—analyzing how user behavior impacts resource usage, horizontal scalability, and load management.
Canary releases progressively route more users to the updated software version as internal confidence in it grows. Eventually, all users are migrated over and the older version is decommissioned. Teams can rinse and repeat with other feature updates as required. You might’ve heard this called an “incremental” or “phased” rollout, as it happens gradually over time. While the old infrastructure remains unused, it assumes a crucial role during fallback. If real-world usage uncovers major issues with newer builds, you can then migrate users backward to previous versions.
How to use canary releases
There are numerous ways to approach canary releases, depending on your user demographics. Large companies like Apple or Google tend to incrementally introduce new features to users in certain regions, countries, or even cities. These specific areas can be ideal test beds for new features (for example, 3D map layouts and street-view capture). Companies might stage their releases based on priority or overall roadmap.
Another method involves the use of multiple canary builds over time, resulting in a canary pipeline of sorts. New builds are first distributed to internal testers, who employ automated testing, usage patterns, and unique device-application pairings to assess functionality. Developer-side elements, like feature toggles, can help technical teams evaluate feature readiness. Once QA testing returns favorable results, that build is pushed live to select users. A router and load balancer work in tandem to distribute users. A small portion is directed to the canary release, while most user activity is centered within an alternative version. The release package eventually migrates to this primary build.
You should note, though, that deploying these builds becomes challenging with mobile or desktop apps that aren’t updated automatically. The onus is on the user to update in situations where server-side changes aren’t possible. Static code changes are tricky to apply reliably via manual updates; therefore, it’s harder to gather feedback and distribute your canary releases.
Just like it’s used for applications, canary testing can be used for web pages and browsers. For Google Chrome and Microsoft Edge updates, developers used flags to enable new features such as “Force Dark Mode for Web Contents” and “Dark Mode to New Tab Page,” respectively. Active flags and toggles like these help internal teams catch glaring issues.
It’s preferable to push canary builds to highly trusted users, or those who aren’t as sensitive to deployment snags. New users can be more fickle, but long-standing users have more loyalty to an application. That’s why those who’ve used a service for years might be more likely to receive a canary release than newer adopters. They’re more likely to provide feedback and become involved in the evolution of useful updates.
Enhancing the feedback loop
Canary releases represent a key portion of the DevOps feedback loop. These connected processes influence how developers, IT teams, and others within an organization continuously function by relying on internal and external input.
For example, let’s say that the QA team has discovered a showstopping bug within a new feature’s code. They relay that information to their developers, who work on a fix. A canary release allows developers to capture usage metrics, gather direct user feedback, and determine how polished their builds (or backends) are before approving a widespread rollout. This process happens rapidly and is equally reliant on cultural buy-in and sound processes. That’s in stark contrast to something like A/B testing, which may take days or even weeks to gather useful feedback. DevOps teams can successfully monitor canary features over a few hours—or even minutes.
Canary releases help you build better software and improve internal operations. If multiple issues keep sneaking into production builds, it’s clear that better testing or automation is needed to prevent future problems. The goal of the feedback loop is to bring software functionality level with customer expectations.
This is why canary testing is valuable throughout CI/CD. New code is integrated into an existing codebase, refined with small changes, and deployed via canary releases. Since features tend to be smaller and decoupled in the microservices realm, they’re easier to isolate and test. Canary releases make continuous delivery easier by facilitating realistic testing via partitioning, as opposed to something like sandboxing.
While security is important, it’s not the chief goal of canary testing. The real prize is ensuring compatibility, functionality, and usability without driving users away.
Git release management
Git was introduced to streamline the process of version control for applications. It’s ideal for both small and large deployments, which is why it pairs well with canary-release strategies. Git helps teams control how new deployments are pushed or rolled back in response to feature updates or known issues, respectively. With an intermediary service that integrates with Git, you can even determine how traffic is distributed between your general availability (GA) and canary builds. This is a form of GitOps.
It’s possible to tag your unique releases in Git, then create a deployment pipeline for each. Patches are pushed automatically or manually as required. Since canary releases live within the infrastructure networking layer, you can leverage some sort of gateway that controls how users receive experimental features. Once new commits are added and builds are updated, a companion service like GitLab can help deploy those canary versions to your Kubernetes clusters. Accordingly, Git and its components make it easy to launch canary releases within a microservices ecosystem.
Killing features
Sometimes a feature release may be detrimental and require immediate revocation. Canary releases can contain built-in feature flags that essentially act as toggles. If a canary feature is causing problems, it’s easy for developers to disable that flag. This is handy in a live production environment because the application doesn’t need to restart.
Feature flags thus function as convenient kill switches. They help preserve uptime and reduce friction stemming from broken code. This feeds into the validation stage of the CI/CD pipeline, in which code quality and compatibility must be solid prior to canary promotion.
Canary releases vs. blue-green deployments
In blue-green deployment, you simultaneously deploy two versions of an application as user traffic is switched to the new one. Canary releases, as noted above, are a smaller portion of your current setup provisioned for live feature experimentation. Each approach is focused on safety and risk management. However, canary releases can follow a node-based deployment strategy; a portion of nodes are converted to run the canary build (say fifteen percent), and this proportion increases in stages until one hundred percent of nodes are swapped. Canary releases typically require fewer resources to run effectively, making them cheaper to support.
Additionally, DevOps teams running blue-green testing—also known as A/B testing—gather feedback on the active environment while the other environment remains idle. No users interact with the inactive version. A canary release dynamically splits all traffic between two active ecosystems.
Conclusion
Canary releases offer value both within the CI/CD pipeline and to the DevOps world overall. The results of a canary testing session can impact validation, testing, delivery, and deployment. Rapid feedback gathered throughout is used to accelerate software development in a responsible manner—giving users a “preview” of what’s to come, without making them suffer unnecessarily from bugs.
Ready to jump-start your own canary release process? LaunchDarkly helps you deploy experimental features to customers safely and easily. Should problems arise, you can disable these elements and perform a rollback without needing to manually restore older software. LaunchDarkly makes it easier to do the following:
- Release experimental features and measure their sales or business impacts
- Test new infrastructure without switching all traffic
- Release features for beta testing
- Release new features to VIP users first
LaunchDarkly can streamline your scaling process and make the testing and monitoring phases of your pipeline more user-friendly. If you want to learn more, give LaunchDarkly a try today, or explore our free demo.
.webp)
