What is canary deployment, and what makes it actually work?

A canary deployment sends a small share of traffic to a new version, watches what happens, and either continues or rolls back. The appeal is obvious: a bad release affects a fraction of users for a few minutes rather than everybody for as long as it takes somebody to notice.

The part that decides whether it works is not the traffic split. It is whether something is watching a signal that would actually reveal the problem, and whether rollback is automatic — because a canary that requires a person to notice is a normal deployment with extra steps.

What we'll cover

• What it is, precisely
• Canary, blue-green, and rolling
• The signal is the hard part
• Sizing and timing the canary
• Automatic rollback
• What cannot be canaried
• Feature flags alongside
• When it is not worth it
• Automating the release process with CodeWords
• What to get right first
• Frequently asked questions

What it is, precisely

A small share of traffic goes to the new version while everything else stays on the current one.

Both run simultaneously, which is the requirement that makes it possible and the constraint that makes some changes ineligible.

Something observes the difference between the two populations.

The share increases in stages if the signal is healthy — a small percentage, then larger, then everything.

Rollback is routing. Sending the traffic back to the old version is fast because the old version is still running, which is the property that makes the whole approach safe.

Canary, blue-green, and rolling

Three strategies that get conflated, with meaningfully different properties.

Blue-green runs two complete environments and switches all traffic at once. Rollback is instant and there is no gradual exposure — a bad release affects everybody, briefly.

Rolling replaces instances gradually, so both versions serve traffic during the transition, but the split is a side effect rather than a control and there is usually no analysis gate.

Canary deliberately controls the share and gates progression on a signal. It is the only one of the three that is a decision rather than a mechanism.

Canary costs more to operate, needing traffic control, comparison, and automation. Blue-green is simpler and coarser; rolling is the default in most orchestrators and is not a release strategy on its own.

The signal is the hard part

Everything else is plumbing. The question is what would reveal that the new version is worse.

Error rate is the obvious one and it is necessary rather than sufficient — plenty of bad releases produce no errors.

Latency, at a high percentile. The median hides a regression affecting a minority; the tail does not.

Business metrics catch what technical ones miss. Checkout completions, signups, messages sent — a release that breaks a form produces perfect infrastructure metrics and a collapse here.

Compare canary against control, not against history. Traffic varies by hour and by day, so comparing the canary to the simultaneously running old version removes that entirely.

Require enough volume. A canary at one per cent on a low-traffic service produces numbers too small to distinguish from noise, which is the commonest reason a canary tells you nothing.

Decide the thresholds before deploying, because deciding during is how a marginal result becomes a judgement call under pressure.

Sizing and timing the canary

Start small enough to contain the damage, large enough to produce a usable signal. Those two constraints frequently conflict on a low-traffic service, and the honest answer there may be that canarying is not worth it.

Hold long enough to see the problem. A memory leak or a connection exhaustion appears after time, not after requests, so a canary that runs for ninety seconds cannot catch either.

Progress in stages, with an analysis gate at each, rather than going from a small share straight to everything.

Route consistently per user rather than per request, or a user bounces between versions and sees inconsistent behaviour — which is both confusing and a poor test.

Cover the traffic mix. A canary receiving only traffic from one region, one client version, or one time of day is testing a subset.

Stop the clock on deployments during the canary, so the comparison is not confounded by another change.

Automatic rollback

Manual rollback means somebody has to be watching, which reduces the canary to a slower deployment with more steps.

Define the conditions in advance — error rate above a threshold relative to control, latency percentile degraded by some margin — and let the system act.

Roll back first, investigate after. The cheapest moment to reverse is immediately, and the investigation is easier without production pressure.

Rehearse it. A rollback path that has never executed is a plan rather than a capability, and finding that out during an incident is expensive.

Make rollback safe. If the new version wrote data the old cannot read, rolling back is not a reversal, which is the constraint discussed next.

Alert on the rollback, loudly. An automatic rollback that nobody is told about means a broken release is quietly parked and possibly redeployed.

What cannot be canaried

The limits, which are mostly about state.

Database schema changes. Both versions share one database, so a migration that only the new version understands breaks the old one — which is why expand-migrate-contract exists, splitting the change across releases so each is individually compatible both ways.

Anything writing data the old version cannot read, since rollback then loses or corrupts it.

Changes to shared infrastructure — a message format, a cache key scheme — affecting consumers that are not part of the canary.

One-way operations. An email sent by the canary cannot be unsent, so a bad release that communicates with customers is not recoverable by routing.

Stateful sessions, unless routing is consistent per user, since a user moving between versions mid-session sees inconsistency.

Feature flags alongside

Flags decouple deploying from releasing. The code ships to everybody; the behaviour is enabled separately.

That is a finer instrument than traffic routing, since it can target specific users, plans, or regions rather than a random share.

They compose well. Canary the deployment to prove the build is sound, then flag the behaviour to control who experiences the change.

Flags accumulate and become a source of untested combinations, so removing them after a release should be part of the release rather than a later intention.

A flag is also a rollback that does not require redeploying, which is frequently the faster remedy.

When it is not worth it

Worth saying, because progressive delivery gets adopted as a standard rather than as a fit.

Low traffic. A service handling a few requests a minute cannot produce a signal from a small share in a reasonable window, and waiting long enough to gather one defeats the purpose.

Internal tools where the blast radius is small and somebody will simply say if it breaks.

Changes with no runtime behaviour — copy edits, documentation, configuration with no code path — which gain nothing from a gradual rollout.

Where nobody will maintain the analysis. Thresholds that were tuned once and never revisited produce either constant false rollbacks or a gate that passes everything.

Where the honest alternative is fine. Blue-green with a rehearsed rollback is simpler, coarser, and adequate for a great deal of software.

Adopt it where the cost of a bad release is high and the traffic supports the analysis. Everywhere else, it is ceremony that looks like rigour.

Automating the release process with CodeWords

The connective work around a release — recording what deployed, telling the teams who need to know, gathering the context when something rolls back — is where automation helps. CodeWords connects to more than 3,000 integrations.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which release events, what should be recorded, and who should be told.
3. Describe the exceptions: a rollback, a canary held longer than expected, a deployment during an incident.
4. Connect the systems involved once.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

The signal and the automatic rollback, together. A canary without an automated decision is a deployment that takes longer, and a canary watching only error rate misses the large category of releases that break something without erroring.

Two habits make the difference. Compare the canary against the simultaneously running control rather than against yesterday, since that removes every confounding variation in traffic. And include at least one business metric alongside the technical ones, because a broken checkout produces perfectly healthy infrastructure numbers.

Frequently asked questions

What makes a canary deployment worth the complexity?

Automatic analysis and automatic rollback. Without both, it is a normal deployment that takes longer and still depends on somebody noticing — which is the thing it was supposed to improve.

What signal should gate the canary?

Error rate and a high-percentile latency, plus at least one business metric. Compare against the simultaneously running control rather than against historical values, which removes variation by hour and by day.

How small should the canary be?

Small enough to contain damage, large enough to produce a statistically usable signal. On a low-traffic service those conflict, and the honest conclusion may be that blue-green with fast rollback suits you better.

Can database migrations be canaried?

Not directly, because both versions share the database. Split the change with expand-migrate-contract so each release is compatible in both directions, which makes each individual deployment canary-safe.

What is the difference from blue-green?

Blue-green switches all traffic at once between two environments — simple, with instant rollback and no gradual exposure. Canary controls the share deliberately and gates progression on analysis, which is a decision rather than a mechanism.

Should feature flags replace canary deployments?

They complement each other. Canary proves the build is sound in production; flags control who experiences the behaviour, with finer targeting. Using both is common, and flags are often the faster rollback.

What happens after an automatic rollback?

Tell people, loudly. An automatic rollback nobody is informed about means a broken release is quietly parked, and the same change gets redeployed later by somebody who never knew it failed.

When is canarying not worth it?

On low-traffic services where a small share cannot produce a usable signal, on internal tools with a small blast radius, and for changes with no runtime behaviour. Blue-green with a rehearsed rollback is simpler and adequate for a lot of software.

What happens to thresholds over time?

They rot. Tuned once and never revisited, they end up either rolling back constantly or passing everything — and a gate that passes everything is indistinguishable from no gate while looking like rigour.

Related reading

• CI/CD pipelines explained
• Shift-left testing
• Platform engineering