Canary Testing

Overview

According to Aslan Brooke, “companies use “Canary Deployments” or “Canary Testing” to test software in production by routing a subset of users to new functionality as part of continuous delivery according to Nolio in their 1st video in a series about DevOps Best Practices.” A canary release is, therefore, a partial release to a subset of production nodes with sticky (custom) sessions turned on. Thus, you are able to minimize the number of users/customers that get impacted if you end up releasing a bad bug.

Origin of Canary

“Canary” was chosen because canaries were once used in coal mines to alert miners when toxic gases reached dangerous levels. In other words, if dangerous gases such as carbon monoxide collected in the mine, the gases would kill the canary before killing the miners, thus providing a warning to exit the tunnels immediately.  As long as the bird kept singing, the miners knew their air supply was safe. A dead canary signaled an immediate evacuation.

The word canary was selected to describe the code push because just like canaries that were once used in coal mining to alert miners when toxic gases reached dangerous levels, end users selected for testing are unaware they are being used to provide an early warning. Canary tests, which are often automated, are run after testing in a sandbox environment has been completed. Because the canary is only pushed to a small number of users, its impact is relatively small should the new code prove to be buggy and changes can be reversed quickly.

How it Works

A canary deployment / canary test allows you to gradually release new features to a subset of your users while still serving your current branch to the rest of your users.  It basically allows you to test things in parallel without having to make major merges/deployments.  This allows you to A/B test features to assess performance before releasing them to a majority of your users.

According to Nolio’s DevOps Best Practices, canary deployments consist of the following steps:

  1. Stage artifacts for deployments, including: build artifacts, test scripts, config files, and deployment manifests.
  2. Remove “Canary” servers from load balancing.
  3. Upgrade “Canary” application (drain and deploy).
  4. Automated testing of application.
  5. Restore “Canary” servers to load balancing (connectivity and sanity checks).
  6. Upgrade the rest of the servers if the “Canary” testing with live usage is successful. (Otherwise rollback)

LaunchDarkly Canary Deployment

Canary Deployment Resources

Pivotal – Canaries Are Great!

  • “Our production deployment runs multiple instances of the marketplace services broker. Enter the sacrificial canary. When BOSH started the upgrade it only took down one of the brokers, upgraded the bits and tried to restart it. In this case the air in the coal mine was toxic and our little bird did not survive :-(. As a result, we sent no additional souls in, leaving the other service brokers fully functional :-).”

Danilo Sato – Canary Release

  • Canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.  Similar to a BlueGreenDeployment, you start by deploying the new version of your software to a subset of your infrastructure, to which no users are routed.”

Christian Posta – Blue-green Deployments, A/B Testing, and Canary Releases

  • “A lot of teams I talk to recently are very interested in “DevOps” (whatever that means… seems to mean different things to different people?) and when we sit down and talk about what that really means, the direction of the conversation can go down many interesting paths. And some times, the path it goes down makes people feel very uncomfortable. I was talking with a team a while back about deployment best practices, hot deployments, rollbacks etc and when I mentioned blue-green deployments, they became a bit queasy. Another team couldn’t understand why doing something they’ve always done was not such a very good thing.”

Jez Humble – Four Principles of Low-Risk Software Releases

  • “One key goal of continuous deployment is to reduce the risk of releasing software. Counter-intuitively, increased throughput and increased production stability are not a zero-sum game, and effective continuous deployment actually reduces the risk of any individual release. In the course of teaching continuous delivery and talking to people who implement it, I’ve come to see that “doing it right” can be reduced to four principles.”

Lori MacVittie – Programmability in the Network: Canary Deployments

  • “The Canary deployment pattern is an incremental upgrade methodology. First, the build is pushed to a small set of servers to which only a select group of users are directed. If that goes well, the release is pushed to a larger set of servers with a limited set of users. Finally, if that goes well, then the release is pushed out to all servers and all users. If issues occur at any stage, the release is halted – it goes no further. Hence the naming of the pattern – after the miner’s canary, used because “its demise provided a warning of dangerous levels of toxic gases”.”

Conviva – Canary Deployment with NGINX

  • Canary deployment is a very useful technique and there are multiple mechanisms that one can use to implement it. In this post, I describe how to do it if you are using NGINX as your front web server, with the examples specifically for a Django uWSGI-based web service, although a similar configuration can be used for other kinds of web services. In a future post, I will describe a slightly more complex way to achieve the same using HAProxy.”

Edith Harbaugh – Canary Release is the New Beta

  • “Canary release – exposing features to some subset of users (whether it be opt-in, random rollout, or specific segments) is now used to describe what was once a beta.
    • Microsoft: In development of Windows 10, Microsoft used “canary” releases to test with internal users within Microsoft. Gabe Aul, who leads the Data & Fundamentals Team in the Operating Systems Group (OSG), said “our Canary ring probably sees 2X-3X as many builds as OSG because we catch problems in Canary and don’t push to OSG.”
    • Instagram: “Using ‘canary’ releases, updates go out to a subset of users at first, limiting the ability of buggy software to do damage.” Mike Krieger, Instagram co-founder and CTO, said he uses canary releases because “If stuff blows up it affects a very small percentage of people”.
    • Google: For Chrome, Google offers Chrome Canary, which it labels with “Get on the bleeding edge of the web, Google ChromeCanary has the newest of the new Chrome features. Be forewarned: it’s designed for developers and early adopters, and can sometimes break down completely.” “