Are canary releases the new beta? What does beta even mean?
By LaunchDarkly CEO – Edith Harbaugh
A #canary launch (aka canary deployment or canary release) is rolling #features out to a small number of users to assess the reaction of the overall system. Are users happier? Is more money made? Does the new feature increase system load? A canary launch allows you to roll out a feature slowly, and measure the reaction from real user “canaries”, looking for early indicators of danger. If a feature is not good, it can rolled back. Canary launches are a best practice for agile development organizations practicing #continuous delivery to move faster.
Canary launches are named after the “canary in a coal mine” – miners would carry canaries into mines, as canaries were more sensitive to fatal gases like methane or carbon monoxide. If the canary stopped singing, miners knew to exit the mine immediately. Canary launches are different than internal betas (dog fooding), user testing, QA or performance testing in that all of those methods are done in controlled test environments. Canary launching is done on real users in a production environment, either in a randomized fashion or with users explicitly assigned or opted in to the early launch. For example, Google always releases new versions of Chrome as “Chrome Canary” to allow early adopters to try new features.
Traditional pre-launch activities to improve product quality like internal betas, user testing, QA and performance testing are all helpful, but they all lack the key component of testing a feature in actual usage. For example, recruiting people off Craigslist to look at your new user experience in a lab is extremely different than a harried traveler running for a plane seeing your new user experience. QA can try to run every test they can think of, but not be able to know or think of some real life test cases. With the explosion of devices and browsers, it’s difficult to test across platform. And for performance testing, running tests in a lab misses real world latencies and hiccups. I worked at an Internet of Things company where we’d test uploads of our data – sitting in our office with a high speed internet connection. Our real users sometimes had dial-up!
Traditionally, companies have “rolled-their-own” platforms to enable canary launches. The key issues with homegrown systems is whether the systems are effective and maintainable. If your framework fails, you’re in a bad state. In addition, maintenance can be a hassle. LaunchDarkly offers “canaries as a service” – we are a blazingly fast, completely scalable platform for you to roll out features, see the reaction on your users, and either continue rollout or completely shut off. Here’s a video demonstrating canary launching with LaunchDarkly.
Now, you might be thinking – isn’t in bad to test potentially imperfect features on real users? Shouldn’t we do everything beforehand to prevent defects? Of course we should. Every good product development org will do their darndest to product high quality code. However, canary launches provide risk mitigation. If an issue IS to occur, do you want 100% of your users to encounter the issue, or 1%? And, if an issue is to be found, don’t you want a mechanism to quickly recover? To quote Sting of the Police “First to fall over when the atmosphere is less than perfect Your sensibilities are shaken by the slightest defect – you’re a canary in a coal mine”. Canary launches allow you to quickly identify issues that might impact your entire user base, roll back easily to a known good version, and fix the issues in a controlled environment. The net result is a better product for all.
When I was an Engineering Manager at Vignette, I’d run beta programs for our new releases. The beta programs had a dual purpose. First, we wanted to get feedback on the stability and validity of our features. But the beta also fed marketing with happy reference customers for our launch announcement. Customers liked being part of a beta because it gave them early access to features they had been waiting for, as well as an opportunity to influence product direction.
What had changed? The word beta has been overloaded to mean “we’re not entirely ready for prime time, so please be patient”. Gmail was in beta for five years! At TripIt, we had a beta tag for multiple years.
Canary release – exposing features to some subset of users (whether it be opt-in, random rollout, or specific segments) is now used to describe what was once a beta.
- Microsoft: In development of Windows 10, Microsoft used “canary” releases to test with internal users within Microsoft. Gabe Aul, who leads the Data & Fundamentals Team in the Operating Systems Group (OSG), said “our Canary ring probably sees 2X-3X as many builds as OSG because we catch problems in Canary and don’t push to OSG.”
- Instagram: “Using ‘canary’ releases, updates go out to a subset of users at first, limiting the ability of buggy software to do damage.” Mike Krieger, Instagram co-founder and CTO, said he uses canary releases because “If stuff blows up it affects a very small percentage of people”.
- Google: For Chrome, Google offers Chrome Canary, which it labels with “Get on the bleeding edge of the web, Google ChromeCanary has the newest of the new Chrome features. Be forewarned: it’s designed for developers and early adopters, and can sometimes break down completely.”
So yes, canary is the new beta.