How to automate feature flag rollouts: the flags you never remove

Automate feature flag rollouts by staging exposure by percentage, gating each stage on guard metrics, and automating flag removal as well as rollout. Rollout gets the attention: one per cent, then five, then twenty-five, with metrics gating each step. It is the easy half of the problem. The hard half starts after the rollout finishes. Flags accumulate, each one nominally doubling the number of code paths, and a codebase with hundreds of live flags has a combinatorial testing problem nobody can reason about.

Key takeaways

• Give every flag a type, a named owner, an expiry date and a one-sentence description when it is created.
• Gate each rollout stage on guard metrics, not only on time, and hold the rollout when a metric is missing.
• Halt a failing rollout rather than rolling it back automatically, so the evidence is preserved.
• Open a removal ticket automatically for expired flags, because a report asks people to create work and a ticket asks them to do it.
• Mark operational kill switches and permission flags so a cleanup sweep never removes them.

What we'll cover

• What a flag costs after it has shipped
• Deciding the flag's lifetime up front
• Automating the rollout itself
• Building it with CodeWords
• What should halt a rollout
• Automating the cleanup
• Flags that are not rollouts
• Who should be able to change a flag
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

What a flag costs after it has shipped

A shipped flag costs comprehension: an extra code path that may be untested, and a conditional every reader must reason about. The evaluation cost is real at scale, but comprehension is the binding constraint.

Every flag doubles the paths, nominally, and the combinations with other flags are what break.

Tests cover one branch or both. Covering both doubles the suite for that area; covering one leaves the other untested in production.

The dead branch rots. Code behind a flag that is permanently off stops being maintained and becomes a hazard for anyone refactoring near it.

Reading the code gets harder. Every conditional asks the reader whether this path is live.

Nobody knows which flags are safe to remove, which is why they accumulate. Removal requires knowing the flag's state everywhere, and that knowledge is not recorded anywhere.

Deciding the flag's lifetime up front

Decide a flag's lifetime when you create it, by requiring five pieces of metadata.

Give every flag a type at creation: release, experiment, operational, or permission. Each type has a different expected life, and only one is permanent by design.

Set an expiry date on release and experiment flags, recorded where the automation can read it.

Name the owner, an individual, since a flag owned by a team is owned by nobody.

Record what it gates in a sentence, because six months later the flag name will not be enough.

Link it to the ticket so the removal work has somewhere to go.

Refuse creation without these. The check is quick, and it is the only thing that prevents accumulation.

Automating the rollout itself

Automate a rollout in coarse stages, each conditional on guard metrics.

Stage by percentage, with a soak period at each step long enough for the metrics to be meaningful.

Segment before percentage where possible: internal users, then beta customers, then a percentage. People who expect early failures find them best.

Make each step conditional on the guard metrics rather than only on time.

Keep the steps coarse. One, five, twenty-five, fifty, a hundred is enough, and finer steps mostly add waiting.

Require an explicit approval for the final step to a hundred per cent, since that step is the hardest to reverse socially.

Announce each step where the team can see it, so an unrelated incident can be connected to a rollout in progress.

Building it with CodeWords

You build a rollout automation in CodeWords by describing the stages and guard metrics to Cody, the automation builder. CodeWords connects to 3,000+ integrations (CodeWords pricing page, 2026), so the flag service, the metrics, the repository and your channels can meet in one automation.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody: the stages, the guard metrics, the halt conditions, and who is told.
3. Authorize the connections: the flag management service, the metrics source, the repository and Slack.
4. Describe the exceptions: a flag with no metrics configured, a rollout during a change freeze, an operational kill switch.
5. Run one low-risk rollout with manual approval at every step before letting it proceed on its own.

You describe the outcome; Cody builds it, connects it, and deploys it. The Free plan costs $0 and covers light use, Pro is $39 per month ($33 billed yearly) and Business is $100 per month (CodeWords pricing page, 2026). Details are on the pricing page.

What should halt a rollout

A rollout should halt when the exposed cohort does worse than the unexposed one on any guard metric. Five signals matter.

• Error rate for the exposed cohort against the unexposed one. This comparison matters, and it differs from the overall rate.
• Latency at the tail, not the mean, since the mean absorbs the degradation you want to catch.
• The business metric the feature was meant to improve, moving the wrong way. This signal is slower.
• Support contact rate from the exposed cohort, which can move before any technical metric.
• A related incident, detected automatically, since continuing a rollout during an incident makes the investigation harder.

Halt rather than roll back by default. Stopping at the current percentage preserves the evidence; automatic rollback destroys it and is occasionally the wrong call.

Automating the cleanup

Automating cleanup is what makes flags an asset instead of debt. Run five checks on a schedule.

Report flags past their expiry, weekly, to their named owner.

Report flags at a hundred per cent for a defined period, the clearest removal candidates.

Report flags at zero per cent, which are dead features whose code is still present.

Detect flags not referenced in the code, which are safe to delete from the service and are noise until then.

Open the removal ticket automatically, assigned to the owner, rather than sending a report that requires somebody to create work.

Track total flag count as a team metric, since it shows whether cleanup is keeping pace with creation.

Flags that are not rollouts

Not every flag is a rollout, and applying rollout automation to the others causes problems.

Flag type	Lifecycle	Automation rule
Operational kill switch	Permanent by design	Never expire or clean up
Permission or entitlement	Product configuration, not release tooling	Often belongs in a different system
Experiment	Ends when the experiment concludes	Do not end it when a percentage reaches a hundred
Long-lived migration	Legitimate and long	Carry a visible reason and a review date instead of an expiry

Mark the type in the flag service so the automation treats each one appropriately and nobody's kill switch is removed during a cleanup sweep.

Who should be able to change a flag

Restrict flag changes by environment and by action, because a flag is a production configuration change that bypasses deployment. That is its value and its risk, and deliberate permissions stop it becoming an unlogged back door.

Separate the environments. Permission to change a flag in staging should not imply permission in production.

Distinguish the kill switch from the rollout. On-call should be able to turn something off in an incident immediately; turning something on for everybody should need more.

Require a second person for the final step, where the flag gates something significant, which is the standard the equivalent deploy would face.

Log to somewhere outside the flag service, since an audit trail held only by the system being audited is weak.

Alert on changes outside working hours that were not part of an incident.

Review the permission list periodically, as it collects people who needed access once for a launch.

Making it survive

A durable rollout automation fails safe and keeps a manual override.

Fail safe. If the flag service is unreachable, the default value must be defined and must be the safe one.

Do not let the rollout automation be the only way to change a flag, since an incident needs a direct manual override.

Log every change with who or what made it and why, the first thing anyone asks during an incident.

Handle metric gaps. No data is not a pass, and a rollout should hold when the guard metric is missing.

Report the outcome: rollouts advanced, halted, flags past expiry, and the total count.

Limits worth knowing about

Small cohorts are statistically weak, so a one per cent stage on a low-traffic feature cannot detect much and the soak time needs to reflect that.

Client-side flags lag, since cached configuration means a change does not take effect immediately for everyone.

Cohort comparison needs consistent assignment, or the exposed and unexposed groups differ in ways that invalidate the comparison.

Flag evaluation in tests is its own problem: a test suite that does not pin flag states is nondeterministic in a way that looks like flakiness.

Removal is a code change with its own risk, which is why flags accumulate, and why making the ticket automatic matters more than making the report polished.

What to build first

Build the weekly flag report first: flags past expiry, flags at a hundred per cent for over a month, flags at zero, and flags not referenced in the code, each addressed to its named owner. It touches no production behaviour and it addresses the half of the problem that compounds.

Two habits make the difference. Require a type, an owner and an expiry at flag creation, since adding that metadata to hundreds of existing flags later is a project nobody will fund. And open the removal ticket automatically rather than sending a report, because a report asks somebody to create work and a ticket asks them to do it.

Frequently asked questions

What does a feature flag cost once the feature has shipped?

A shipped flag costs an extra code path that may be untested, a conditional every reader has to reason about, and a combination with every other live flag. Comprehension is the binding cost, not evaluation.

Why do flags accumulate?

Flags accumulate because removing one requires knowing its state everywhere, and nobody records that. Without a type, an owner and an expiry set at creation, the knowledge needed to remove a flag safely does not exist.

What should gate each rollout stage?

Gate each stage on the exposed cohort's error rate against the unexposed one, tail latency rather than the mean, support contact rate from the exposed group, and the business metric the feature was meant to move.

Should a failed rollout halt or roll back?

Halt by default. Stopping at the current percentage preserves the evidence for investigation, while automatic rollback destroys it and is occasionally the wrong call.

What happens if the guard metric is missing?

Hold the rollout. No data is not a pass, and a rollout that proceeds on missing metrics is an unguarded rollout.

Which flags should never be cleaned up?

Never clean up operational kill switches and permission or entitlement flags. Mark the type in the flag service so a cleanup sweep does not remove an emergency control.

How do I make cleanup actually happen?

Open the removal ticket automatically, assigned to the named owner. A weekly report asks somebody to create work, while a ticket asks them to do it.

Why are flags a problem in tests?

A suite that does not pin flag states is nondeterministic in a way that presents as flakiness. Pinning the states is straightforward but is often not done until something has already cost a day of debugging.

Who should be allowed to change flags in production?

Separate the environments and separate the kill switch from the rollout. On-call needs to turn things off immediately, turning something on for everyone should need a second person, and changes should be logged outside the flag service.

Are feature flags an audit concern?

Yes, because flags change production behaviour without a deploy. Log every change with who made it and why, and review the permission list periodically, since it collects people who needed access once for a launch.

Related reading

• Automating regression test triggering
• Automating API health checks
• GitHub and CodeWords