CI/CD pipelines explained: what continuous actually requires

Continuous integration means everybody merges to a shared branch frequently, and something verifies each merge. Continuous delivery means every verified change is releasable. Continuous deployment means it is actually released, automatically.

The reason these are worth distinguishing is that organisations commonly have a tool that runs tests and describe themselves as doing continuous integration while merging once a fortnight. The tool is not the practice, and the practice is where the benefit comes from.

What we'll cover

• Three terms, three commitments
• What a pipeline has to earn
• Flaky tests, the expensive problem
• Speed, and why it changes behaviour
• What belongs in which stage
• Deploying without a maintenance window
• Secrets in the pipeline
• Metrics that describe the pipeline honestly
• Automating around the pipeline with CodeWords
• What to get right first
• Frequently asked questions

Three terms, three commitments

Continuous integration is about merging: small changes, frequently, into a shared branch, each verified automatically. The verification is the visible part and the frequent merging is the part that produces the benefit.

Continuous delivery is about readiness: every change that passes is deployable, which requires the deployment itself to be automated and rehearsed even if a person presses the button.

Continuous deployment removes the button. Every passing change goes out.

They build on each other in order. Continuous deployment on top of a suite nobody trusts is a mechanism for shipping bugs quickly.

Most teams should target continuous delivery and adopt continuous deployment where the change type and the risk make it sensible — which varies by service rather than by organisation.

What a pipeline has to earn

A pipeline is only useful if people believe it, and belief has preconditions.

A failure must mean something is broken. The moment a red build routinely means "try again", the signal is gone and every subsequent failure is ignored.

It must be fast enough to wait for. A pipeline nobody waits for is a pipeline people merge around.

Its results must be reproducible. A build that passes locally and fails in the pipeline, or vice versa, for environmental reasons, erodes trust quickly.

Failures must be legible. A red mark with a log file to search is a worse signal than a failure that names the test and shows the assertion.

It must block what it claims to block. A required check that can be overridden routinely is documentation, not a gate.

Flaky tests, the expensive problem

The single biggest destroyer of pipeline value, and the least tracked.

A flaky test fails intermittently for reasons unrelated to the change, usually timing, shared state, or an external dependency.

Its cost is not the re-run. It is that every failure now requires a judgement about whether it is real, and people stop making that judgement carefully.

Measure it directly. Count jobs that failed and then passed on re-run with no code change. No instrumentation is needed and the result is a ranked list.

Rank by frequency times job duration, since a flaky test in a twenty-minute job costs far more than one in a two-minute job.

Quarantine with an owner and a date. Moving a flaky test out of the blocking path is reasonable as a temporary measure; without an owner and a date it is deletion with extra steps, and the coverage gap is invisible because the suite still passes.

Treat a rising flakiness rate as a decay signal, because it is much cheaper to address early than after people have stopped reading failures.

Speed, and why it changes behaviour

Under ten minutes and people wait, watching the result before moving on.

Beyond that they context switch, which means a failure is addressed later, with less context, and often after another change has been merged on top.

Beyond an hour they merge around it, and the pipeline becomes a thing that reports on history rather than a gate.

Parallelise and split by stage. Fast checks first, so the common failures surface in a minute rather than at the end of a long run.

Fail fast on cheap checks — formatting, linting, type checking — before spending money on an integration suite that will fail for the same reason.

Cache deliberately, and measure the hit rate. A cache key based on something that changes every commit never hits, and the build simply does several minutes of avoidable work every time with nothing reporting it.

What belongs in which stage

On every commit: format, lint, type check, unit tests. Everything cheap and everything deterministic.

On every pull request: the above plus integration tests, plus any build artefact that proves the thing compiles.

On merge to the main branch: the full suite, the artefact that will actually be deployed, and the deployment to a staging environment.

On release: deployment to production, with whatever gate your risk appetite requires.

Nightly: the expensive things — long-running suites, security scans, dependency checks — that would make the fast path unusable.

Nothing in the pipeline that a developer cannot run locally. A check that only exists in CI is a check people cannot fix without pushing repeatedly.

Deploying without a maintenance window

Backwards-compatible changes first. A deployment that requires the database and the application to change simultaneously is a deployment that needs a window.

Expand, migrate, contract. Add the new column, write to both, backfill, switch reads, remove the old — several deployments rather than one, each individually safe.

Feature flags decouple deploying from releasing, which is what makes frequent deployment compatible with careful release.

Rolling or canary deployment limits exposure, with automated rollback on a health signal.

Rehearse the rollback. A rollback path that has never been executed is a plan, not a capability, and the moment you need it is not the moment to find out.

Secrets in the pipeline

Worth its own section, because CI holds credentials that reach everything.

Scope per environment. A pipeline that can deploy to production from any branch is a pipeline where a branch is a production deployment.

Restrict who can use production credentials, using whatever grouping your platform supports, rather than making them available to every job.

Forked pull requests must not receive secrets. Check the default behaviour rather than assuming it.

Inventory them. Nothing tells you a credential is three years old and belongs to a system that no longer exists.

Rotate and confirm the next run passed, since a rotation that breaks a nightly job is found days later.

Metrics that describe the pipeline honestly

Most CI dashboards count things that are easy to count, and a few numbers are genuinely worth reporting.

Pass rate on the main branch, which is a direct measure of whether the gate before it is working.

Time from push to feedback, as a distribution rather than an average, since the tail is what people experience as "slow".

Re-runs that passed with no change, which is flakiness expressed as a number somebody can act on.

Queue time separately from run time. Waiting for an available runner is a capacity problem and running slowly is an efficiency problem, and a combined figure hides which you have.

Cost per pipeline run where your platform bills by usage, broken down by job, since it is almost always one or two jobs.

Report at team level, not individual. These describe a system, and pointed at people they get gamed — smaller commits for the sake of the number, tests deleted rather than fixed.

Automating around the pipeline with CodeWords

Most of what people build outside a pipeline is reporting across runs — which is not something a single job can do. CodeWords connects to more than 3,000 integrations, including the common CI platforms.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which projects, what to measure, and who should hear about it.
3. Describe the exceptions: a project with no recent runs, a job with no test metadata.
4. Connect the CI platform and wherever the report should go, once.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

A suite people trust, which in practice means eliminating flakiness before adding coverage. Every other improvement on this page — speed, staging, automated deployment — is worth less than nothing on top of a pipeline whose failures are routinely ignored.

Two habits make the difference. Measure flakiness by counting re-runs that pass with no code change, since that produces a ranked list without any instrumentation. And keep the fast path under ten minutes, because that threshold is roughly where people stop waiting and start working around the pipeline entirely.

Frequently asked questions

What is the difference between continuous delivery and deployment?

Delivery means every passing change is releasable, with the deployment automated and rehearsed. Deployment means it is actually released without a person pressing anything. Most teams should target delivery and adopt deployment per service.

Is running tests on every commit continuous integration?

Only half of it. The practice is frequent merging into a shared branch; the automated verification is what makes that safe. A team merging once a fortnight has a test runner rather than continuous integration.

Why are flaky tests such a problem?

Because they destroy the meaning of a failure. Once red routinely means "try again", people stop evaluating failures carefully, and the next real one is ignored along with the noise.

How fast should a pipeline be?

Under ten minutes for the path developers wait on. Beyond that they context switch and fix failures later with less context; beyond an hour they work around it entirely and the gate stops being a gate.

How do I deploy without a maintenance window?

Make changes backwards compatible and split them across deployments — expand, migrate, contract — so no single deployment requires two things to change at once. Feature flags separate deploying from releasing.

What should never be in the pipeline?

A check a developer cannot run locally. It forces people to debug by pushing repeatedly, which is slow, frustrating, and makes the pipeline something to defeat rather than something to use.

Should quarantined tests be re-enabled?

Yes, with an owner and a date at the moment of quarantine. Without both, quarantine is deletion with extra steps and the coverage gap is invisible because the suite continues to pass.

Which pipeline metrics are worth reporting?

Main-branch pass rate, time from push to feedback as a distribution, and re-runs that passed with no code change. Separate queue time from run time, since one is a capacity problem and the other an efficiency one.

Should pipeline metrics be attributed to individuals?

No. They describe a system, and pointed at people they get gamed — commits split for the sake of a number, flaky tests deleted rather than fixed. Report at team level and use them to find friction.

Should the pipeline run the same way locally?

As closely as possible. A check that only exists in CI forces people to debug by pushing repeatedly, and the gap between local and pipeline environments is a steady source of results nobody can reproduce.

What is the first thing to fix in a slow pipeline?

Order. Run the cheap deterministic checks first so common failures surface in a minute, and check the cache hit rate — a key that changes every commit never hits, and the build silently redoes several minutes of work every time.

Related reading

• Canary deployment
• Infrastructure as code
• Shift-left testing