How to automate feature flag rollouts: the flags you never remove

Progressive rollout gets all the attention: one per cent, then five, then twenty-five, with metrics gating each step. It is worth automating and it is the easy half of the problem.

The hard half is what happens after the rollout finishes. Flags accumulate, each one doubling the notional number of code paths, and a codebase with two hundred live flags has a combinatorial testing problem nobody can reason about. The teams that get value from flags are the ones that automated removal, not just rollout.

What we'll cover

• What a flag costs after it has shipped
• Deciding the flag's lifetime up front
• Automating the rollout itself
• Building it with CodeWords
• What should halt a rollout
• Automating the cleanup
• Flags that are not rollouts
• Who should be able to change a flag
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

What a flag costs after it has shipped

Every flag doubles the paths, nominally, and the combinations are what actually breaks.

Tests either cover one branch or both. Covering both doubles the suite for that area; covering one means the other is untested in production.

The dead branch rots. Code behind a flag that is permanently off stops being maintained and starts being a hazard for anyone refactoring near it.

Reading the code gets harder. Every conditional is a question the reader has to answer about whether this path is live.

Nobody knows which are safe to remove, which is the reason they accumulate — removal requires knowing the flag's state everywhere, and that knowledge is nowhere.

The evaluation cost is real at scale, though it is rarely the binding constraint. The binding constraint is comprehension.

Deciding the flag's lifetime up front

Give every flag a type at creation: release, experiment, operational, or permission. Each has a different expected life and only one of them is permanent by design.

Set an expiry date on release and experiment flags, recorded where the automation can read it.

Name the owner, an individual, since a flag owned by a team is a flag owned by nobody.

Record what it gates in a sentence, because six months later the flag name will not be enough.

Link it to the ticket so the removal work has somewhere to go.

Refuse creation without these, which sounds bureaucratic and takes thirty seconds and is the only thing that prevents the accumulation.

Automating the rollout itself

Stage by percentage, with a soak period at each step long enough for the metrics to be meaningful.

Segment before percentage where possible — internal users, then beta customers, then a percentage — since the early failures are better found by people who expect them.

Make each step conditional on the guard metrics rather than only on time.

Keep the steps coarse. One, five, twenty-five, fifty, a hundred is enough, and finer granularity mostly adds waiting.

Require an explicit approval for the final step to a hundred per cent, since that is the one that is hardest to walk back socially.

Announce each step where the team can see it, so an unrelated incident can be connected to a rollout in progress.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the flag service, the metrics, the repository and your channels can meet in one automation.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the stages, the guard metrics, the halt conditions, and who is told.
3. Authorize the connections — the flag management service, the metrics source, the repository and Slack.
4. Describe the exceptions: a flag with no metrics configured, a rollout during a change freeze, an operational kill switch.
5. Run one low-risk rollout through it manually-approved at every step before letting it proceed on its own.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What should halt a rollout

Error rate for the exposed cohort against the unexposed one, which is the comparison that matters and is not the same as the overall rate.

Latency at the tail, not the mean, since the mean absorbs exactly the degradation you want to catch.

The business metric the feature was meant to improve, moving the wrong way, which is a different and slower signal.

Support contact rate from the exposed cohort, which frequently moves before any technical metric does.

A related incident, automatically, since continuing a rollout during an unrelated incident makes the investigation considerably harder.

Halt rather than roll back by default. Stopping at the current percentage preserves the evidence; automatic rollback destroys it and is occasionally the wrong call.

Automating the cleanup

This is the part that determines whether flags are an asset.

Report flags past their expiry, weekly, to their named owner.

Report flags at a hundred per cent for a defined period, which is the clearest removal candidate there is.

Report flags at zero per cent, which are dead features whose code is still present.

Detect flags not referenced in the code at all, which are safe to delete from the service and are pure noise until then.

Open the removal ticket automatically, assigned to the owner, rather than sending a report that requires somebody to create work.

Track total flag count as a team metric, since it is the number that shows whether cleanup is keeping pace with creation.

Flags that are not rollouts

Worth separating, because applying rollout automation to them causes problems.

Operational kill switches are permanent by design and should never expire or be cleaned up.

Permission and entitlement flags are product configuration rather than release tooling, and they frequently belong in a different system entirely.

Experiment flags have a statistical lifecycle — they end when the experiment concludes, not when a percentage reaches a hundred.

Long-lived migration flags are legitimate and should carry a visible reason and a review date rather than an expiry.

Mark the type in the flag service so the automation treats each appropriately and nobody's kill switch is auto-removed during a cleanup sweep.

Who should be able to change a flag

A flag is a production configuration change that bypasses deployment, which is its value and its risk. Deciding the permissions deliberately is what stops it becoming an unlogged back door.

Separate the environments. Permission to change a flag in staging should not imply permission in production.

Distinguish the kill switch from the rollout. Turning something off in an incident should be available to on-call immediately; turning something on for everybody should not be.

Require a second person for the final step, where the flag gates something significant, which is the same standard the equivalent deploy would face.

Log to somewhere outside the flag service, since an audit trail held only by the system being audited is weak.

Alert on changes outside working hours that were not part of an incident, because that pattern is worth looking at whether it turns out to be routine or not.

Review the permission list periodically, as it accumulates people who needed access once for a launch and have kept it since.

Making it survive

Fail safe. If the flag service is unreachable, the default value must be defined and must be the safe one.

Do not let the rollout automation be the only way to change a flag, since an incident needs a direct manual override.

Log every change with who or what made it and why, which is the first thing anyone asks during an incident.

Handle metric gaps. No data is not a pass, and a rollout should hold rather than proceed when the guard metric is missing.

Report the outcome: rollouts advanced, halted, flags past expiry, and the total count.

Limits worth knowing about

Small cohorts are statistically weak, so a one per cent stage on a low-traffic feature cannot detect much and the soak time needs to reflect that.

Client-side flags lag, since cached configuration means a change does not take effect immediately for everyone.

Cohort comparison needs consistent assignment, or the exposed and unexposed groups differ in ways that invalidate the comparison.

Flag evaluation in tests is its own problem, and a test suite that does not pin flag states is nondeterministic in a way that looks like flakiness.

Removal is a code change with its own risk, which is why it accumulates, and why making the ticket automatic matters more than making the report pretty.

What to build first

The weekly flag report: flags past expiry, flags at a hundred per cent for over a month, flags at zero, and flags not referenced in the code — each addressed to its named owner. It touches no production behaviour and it addresses the half of the problem that actually compounds.

Two habits make the difference. Require a type, an owner and an expiry at flag creation, since retrofitting that metadata onto two hundred existing flags is a project nobody will fund. And open the removal ticket automatically rather than sending a report, because a report asks somebody to create work and a ticket asks them to do it.

Frequently asked questions

What does a feature flag cost once the feature has shipped?

A code path that may be untested, a conditional every reader has to reason about, and a combination with every other live flag. The comprehension cost is what binds, not the evaluation cost.

Why do flags accumulate?

Because removing one requires knowing its state everywhere, and nobody records that. Without a type, an owner and an expiry set at creation, the knowledge needed to remove a flag safely does not exist.

What should gate each rollout stage?

Error rate for the exposed cohort against the unexposed one, tail latency rather than the mean, support contact rate from the exposed group, and the business metric the feature was meant to move.

Should a failed rollout halt or roll back?

Halt by default. Stopping at the current percentage preserves the evidence for investigation; automatic rollback destroys it and is occasionally the wrong call.

What happens if the guard metric is missing?

Hold. No data is not a pass. A rollout that proceeds on missing metrics is an unguarded rollout with extra steps.

Which flags should never be cleaned up?

Operational kill switches and permission or entitlement flags. Mark the type in the flag service so a cleanup sweep does not remove somebody's emergency control.

How do I make cleanup actually happen?

Open the removal ticket automatically, assigned to the named owner. A weekly report asks somebody to create work; a ticket asks them to do it, and the difference in completion rate is large.

Why are flags a problem in tests?

A suite that does not pin flag states is nondeterministic in a way that presents as flakiness. Pinning them is straightforward and is usually not done until something has already been debugged for a day.

Who should be allowed to change flags in production?

Separate environments, separate the kill switch from the rollout — on-call needs to turn things off immediately, turning something on for everyone should require a second person — and log changes somewhere outside the flag service.

Are feature flags an audit concern?

They change production behaviour without a deploy, so yes. Log every change with who made it and why, and review the permission list periodically, since it accumulates people who needed access once for a launch.

Related reading

• Automating regression test triggering
• Automating API health checks
• GitHub and CodeWords