Automating regression test triggering: which tests, when

Every test suite eventually outgrows the patience of the people waiting for it. The suite that took four minutes takes forty, engineers start merging before it finishes, and the feedback loop the suite existed to provide is gone.

The response is to stop running everything on every change, which immediately raises the real question: which tests, when, and what are you accepting by not running the rest? That trade-off is the subject worth automating carefully, because the failure mode is silent — a suite that passes quickly and misses things looks identical to a suite that is working.

What we'll cover

• Why running everything stops working
• The selection strategies, honestly compared
• Tiering by when, not just what
• Building it with CodeWords
• Flakiness, which ruins all of this
• Reporting a result people act on
• Measuring whether the selection is safe
• The tests worth keeping out of the selection entirely
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Why running everything stops working

Wall-clock time is the constraint, not compute cost. A forty-minute suite changes how people work regardless of what it costs to run.

Parallelism has a floor. Setup, fixtures, and shared resources mean a suite does not shrink linearly with runners.

Long feedback loops get routed around. People batch changes, merge optimistically, and stop reading the results, which removes the benefit entirely.

The slow tests are usually the valuable ones. End-to-end coverage is what catches integration failures and it is exactly what makes the suite slow.

Nightly runs defer the problem. A failure found the next morning has already been built on by three people.

So selection is not an optimisation, it is the thing that keeps the suite useful. The question is only how to do it without quietly losing coverage.

The selection strategies, honestly compared

By file path mapping — changed directory to test directory. Trivial to implement, works well in a modular codebase, and misses anything crossing a boundary.

By dependency graph. Which tests exercise code reachable from the change. Considerably better, and it needs a real graph, which not every language makes easy.

By historical failure correlation. Which tests have failed when this area changed before. Effective, and it needs history, and it is blind to newly written tests.

By coverage data. Which tests execute the changed lines. The most precise available signal, and the most expensive to keep current.

By tag. Developer-declared, which is honest about being a guess and is often the practical starting point.

Combine, and always include a core set. Every strategy above has a failure mode; a small always-run tier covers the gap where they all agree wrongly.

Tiering by when, not just what

The more useful framing is not which tests but at which moment, because different moments tolerate different latency.

On every push: the fast, deterministic tests. Minutes, not tens of minutes.

On pull request: the selected set plus the core tier, which is where selection earns its keep.

On merge to main: the full suite, since this is the last point before anything is shared and the wait no longer blocks a person.

Nightly: the long, expensive and environment-dependent tests, plus anything that cannot be parallelised.

Before release: everything, including the tests that need real external systems.

Say what was skipped, at every tier. The developer should be able to see that a selection happened and what it excluded.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the repository, the CI system, the test history, and wherever the team reads results can meet in one automation.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the tiers, the selection rules, the core set, and where results go.
3. Authorize the connections — the repository, the CI provider, and Slack or the tracker.
4. Describe the exceptions: a change to test infrastructure, a dependency bump, a release branch, a known-flaky test.
5. Run it in shadow mode for two weeks — select, but run everything anyway, and record what the selection would have missed.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Flakiness, which ruins all of this

A flaky test teaches people to re-run rather than investigate, and that habit applies equally to real failures.

Quarantine rather than delete. Move it out of the blocking path, keep running it, and report it separately.

Quarantine must expire. A test quarantined indefinitely is a deleted test with extra steps and a false sense of coverage.

Track flake rate per test and publish it, since the worst offenders are usually a small number of tests that everyone knows about informally.

Automatic retries hide the signal. They keep the pipeline green and they remove the evidence that something is genuinely intermittent.

Fix the cause. Most flakiness is shared state, timing assumptions, or real races in the code under test — and the third category is a bug that was found and dismissed.

Reporting a result people act on

Lead with what failed, not with the run status, since the status is one word and the failure is the information.

Show the failure output inline rather than behind a link, because a link is a decision about whether to context-switch.

Say whether this test has failed recently, which immediately distinguishes a real regression from a known flake.

Name the likely change where the selection makes that inferable, since the developer already knows what they touched.

Report duration and the skip count so the trade-off stays visible rather than becoming invisible infrastructure.

Notify the author, not a channel, for their own pull request.

Measuring whether the selection is safe

This is the part that is usually missing, and without it the selection is a guess that nobody ever checks.

Count escaped failures: tests that pass at pull-request time and fail on main. That number is the cost of your selection, stated plainly.

Attribute them. Which selection rule missed it, and whether the same rule is missing things repeatedly.

Report it monthly alongside the time saved, so the trade-off is a decision rather than a drift.

Set a threshold you would act on in advance, since deciding after the fact always resolves in favour of the faster pipeline.

Re-run shadow mode after major changes to the codebase's structure, because a selection strategy tuned to an old architecture degrades quietly.

Expect a non-zero number. The goal is a known and accepted rate, not zero, which is only achievable by running everything.

The tests worth keeping out of the selection entirely

Some tests do not belong in a selection scheme at all, and identifying them early saves arguing about them repeatedly.

Security and authorisation tests. The cost of skipping one is not proportional to the change that triggered it, so they run always.

Migration and schema tests, since a change that looks unrelated can break a migration in a way nothing else catches.

Money. Anything touching pricing, billing, invoicing, or tax runs every time, because the failure is expensive in a way that is not measured in engineering hours.

Smoke tests for the critical paths — sign-in, checkout, the main workflow — which are fast, stable and worth their seconds unconditionally.

Tests for code with no other coverage. Where one test is the only thing standing between a module and nothing, selection should not be allowed to skip it.

Keep this list explicit and short. It grows by default if nobody guards it, and a core tier that has swollen to half the suite has quietly restored the original problem.

Making it survive

Fail open. If selection cannot be computed, run everything rather than running nothing.

Always run everything on release branches, whatever the rules say.

Treat test infrastructure changes as invalidating the selection entirely.

Cache carefully. A stale dependency graph produces a confidently wrong selection, which is worse than no selection.

Report the outcome: what ran, what was skipped, why, and how long it took.

Limits worth knowing about

Selection cannot see runtime coupling — configuration, feature flags, and data-dependent behaviour are invisible to static analysis.

Coverage data ages and maintaining it accurately costs roughly a full-suite run.

Monorepo boundaries are frequently wrong in the direction of being too permissive, which makes path-based selection over-select.

CI concurrency limits apply, so a selection strategy that spawns many small jobs may not be faster in practice.

Newly written tests have no history, so any history-based strategy must include them unconditionally.

What to build first

Shadow mode: compute the selection, run the full suite anyway, and record what the selection would have missed. It changes nothing about the pipeline and it produces the only number that matters — how much coverage your proposed rules actually lose.

Two habits make the difference. Keep a small always-run core tier regardless of what the selection says, since every strategy has a blind spot and the core tier is what covers it. And publish the escaped-failure count monthly, because a selection nobody measures drifts towards faster and less safe one reasonable-seeming tweak at a time.

Frequently asked questions

When should I stop running the whole suite on every change?

When wall-clock time starts changing how people work — when they batch changes, merge before results arrive, or stop reading them. That is the point at which the suite has stopped providing feedback.

Which test selection strategy is best?

A combination, plus a small always-run core set. Path mapping is trivial and misses cross-boundary changes; dependency graphs are better and need real tooling; failure history is effective and blind to new tests; coverage data is most precise and most expensive.

How do I know the selection is not losing coverage?

Count escaped failures — tests that pass at pull-request time and fail on main — attribute them to the rule that missed them, and report the number monthly alongside the time saved.

Should I aim for zero escaped failures?

No. Zero is only achievable by running everything. The goal is a known, accepted, monitored rate, with a threshold you decided in advance rather than after a bad week.

What should I do about flaky tests?

Quarantine them out of the blocking path with an expiry, keep running and reporting them separately, and publish per-test flake rates. Do not paper over them with automatic retries, which keep the pipeline green and destroy the evidence.

Is a nightly full run enough?

Not on its own. A failure found the next morning has usually been built on by several people. Nightly is right for long, expensive, environment-dependent tests, with the full suite running on merge to main.

What invalidates a test selection entirely?

Changes to test infrastructure, dependency bumps, release branches, and anything that alters the codebase's structure enough that the dependency or coverage data is stale.

How should CI results be reported?

Lead with what failed rather than the run status, show output inline, say whether that test has failed recently, and notify the author directly rather than a channel.

Which tests should never be selected out?

Security and authorisation, migrations and schema changes, anything touching money, smoke tests for critical paths, and tests that are the only coverage a module has. Keep the list explicit and guard its size.

How big should the always-run core tier be?

Small enough that it stays fast. It grows by default if nobody guards it, and a core tier that has swollen to half the suite has restored the original problem without anyone deciding to.

Related reading

• GitHub and CodeWords
• Automating API health checks
• CodeWords integrations and templates