How to automate code review without replacing the reviewer

Code review does two things at once. It checks mechanical properties — formatting, obvious bugs, missing tests, risky patterns — and it transfers understanding between people about whether a change is the right change.

Automation is better than humans at the first and cannot do the second at all. Most of the value in automating review comes from taking the mechanical part away entirely, so the reviewer's attention lands on the judgement. Most of the disappointment comes from tools that add volume to the second instead.

What we'll cover

• Two activities, one name
• What automation does better
• What it cannot do
• Building it with CodeWords
• Why comment volume backfires
• Blocking or advising
• The change that matters more than any tool
• Introducing checks to an existing codebase
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Two activities, one name

Mechanical checking is whether the code conforms — style, types, lint rules, test coverage, dependency risk, common insecure patterns.

Judgement is whether the change solves the right problem, fits the architecture, handles the cases that matter, and will make sense to whoever reads it next year.

Humans are unreliable at the first. Attention degrades over a long diff, and formatting comments crowd out substantive ones.

Automation cannot do the second. It has no view on whether this is the right abstraction or whether the requirement was understood.

The goal is separation, not replacement: every mechanical check automated and silent when passing, so the human review is entirely about the judgement.

What automation does better

Formatting, which should never be discussed by a person. A formatter run automatically ends the conversation permanently.

Lint and type checking, which catch a class of mistakes consistently and instantly.

Dependency vulnerabilities and licences, which no reviewer checks by hand and which matter.

Secrets in the diff, which is the check with the highest cost of being missed and the lowest cost to run.

Test coverage on changed lines, which is a more useful signal than overall coverage and is trivially computable.

Migration and schema risks — a dropped column, a non-concurrent index, a change that is not backwards compatible — which is where a checklist beats memory.

Large or unusual diffs, flagged for extra attention rather than for rejection.

What it cannot do

Worth being explicit, because tools are frequently sold as though it can.

Whether the change is necessary, which requires knowing what was asked for.

Whether the abstraction fits the rest of the system, which requires knowing the rest of the system.

Whether the edge cases that matter are handled, since which cases matter is a domain question.

Whether it will be understandable later, which is a judgement about the people who will read it.

Whether the tests test anything. A suite that passes and asserts nothing is invisible to coverage.

Teaching. Review is how people learn a codebase, and that transfer does not happen between a person and a tool.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the surrounding work — routing reviews, chasing what is blocked, reporting on latency — can run without anybody watching a queue.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which repositories, what should be checked or routed, and who should hear about it.
3. Authorize the connections — the repository host and wherever the team works.
4. Describe the exceptions: a bot-authored change, a draft, an author who has left.
5. Test against one repository before extending it.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Why comment volume backfires

The failure mode of automated review, and it is consistent.

Fifteen comments on a change produces nothing. The author scans, dismisses, and the two that mattered go with the thirteen that did not.

Mixing severities makes it worse. A possible null dereference next to a naming suggestion teaches the reader that the tool's comments are all optional.

Repeated false positives end it. Once a tool is wrong a few times, everything it says is discounted, including the finding that was real.

Silence on success is a feature. A check that says nothing when it passes is a check people trust when it speaks.

Fewer, higher-confidence findings beat comprehensive coverage by a wide margin here, because the currency is attention rather than detection.

Measure whether findings are acted on. A check whose comments are routinely dismissed should be tuned or removed, and only that measurement tells you which.

Blocking or advising

Block on the deterministic and the consequential: formatting, type errors, failing tests, secrets, known vulnerabilities above a threshold.

Advise on the probabilistic: possible bugs, complexity warnings, suggestions.

Never block on something that is wrong occasionally. A gate that requires an override becomes a gate everybody overrides, which removes the value of the gates that should be absolute.

Make overriding visible rather than difficult, with a reason recorded, so the pattern is available.

Review what gets overridden. A rule overridden most times it fires is a rule that is wrong, and the override log is the evidence.

Keep the blocking set small enough to be respected, because credibility is the resource being spent.

The change that matters more than any tool

Smaller changes. A four-hundred-line diff gets a worse review than four hundred-line ones, reliably, and no tooling compensates.

Large changes wait longer, because reviewers postpone them, which adds latency on top of worse review.

Report the size distribution rather than the average, since the tail is where the problem is.

Flag unusually large changes for extra attention rather than blocking them — sometimes a large change is genuinely atomic.

Review latency matters as much as review quality. A change waiting two days has cost more than a slightly shallower review would have.

Notify the reviewer directly, which is the single highest-leverage automation here and is covered in the Slack integration work rather than in any review tool.

Introducing checks to an existing codebase

Turning on a new rule across a mature repository produces thousands of findings, and that is how a good check gets abandoned in its first week.

Baseline the existing violations and enforce only on changed lines. New code meets the standard; old code is not a blocker.

Fix by area, not all at once, when you do fix — a directory at a time, with its own change, so review stays possible.

Format the whole repository in one commit, deliberately, and add it to the blame ignore list so history stays readable.

Introduce as advisory first, for a fortnight, and look at what it would have blocked before making it block.

Let the team see the findings before the gate, since a rule that lands as a surprise is a rule that gets argued with rather than adopted.

Retire rules too. A check that has not fired in a year against a pattern nobody writes any more is configuration to maintain for nothing.

Making it survive

Keep check configuration in the repository, versioned and reviewed, rather than configured in a console.

Run the same checks locally, so people fix things before pushing rather than by pushing repeatedly.

Fail fast. Cheap deterministic checks first, so common failures surface in a minute.

Suppress on bot-authored changes, which otherwise generate constant noise.

Report the outcome. Checks run, findings raised, findings acted on, and overrides with reasons.

Limits worth knowing about

Static analysis has false positives, and the rate determines whether the tool is trusted — tuning it is ongoing work rather than setup.

Coverage measures execution, not assertion. A test that runs code without checking anything counts as covered.

Security scanning finds known patterns, not novel logic flaws, which is a useful and limited thing.

Automated suggestions on a large diff are expensive to generate and expensive to read, which is another argument for smaller changes.

None of it reviews the requirement. A perfectly implemented misunderstanding passes every check there is.

What to build first

Secret scanning and formatting, both blocking, both silent when they pass. Formatting ends an argument permanently and frees reviewer attention immediately; secret scanning has the highest cost of being missed and the lowest cost to run. Neither produces false positives worth arguing about, which is what earns the tooling credibility for anything added later.

Two habits make the difference. Keep the tool silent on success, since a check that says nothing when it passes is one people trust when it speaks. And measure whether findings are acted on — a check routinely dismissed is spending attention it is not earning, and should be tuned or deleted rather than tolerated.

Frequently asked questions

What should automation review?

Formatting, lint, types, dependency vulnerabilities, secrets, coverage on changed lines, and risky migration patterns. Everything mechanical, so the human reviewer's attention goes entirely to whether the change is the right change.

What can it not review?

Whether the change was necessary, whether the abstraction fits, which edge cases matter, and whether the tests assert anything. A perfectly implemented misunderstanding passes every automated check there is.

Why do automated review comments get ignored?

Volume and mixed severity. Fifteen comments produces nothing, and a possible bug listed next to a naming suggestion teaches people that all of it is optional. Fewer, higher-confidence findings work far better.

Should automated checks block merging?

Block on the deterministic and consequential — formatting, types, failing tests, secrets. Advise on the probabilistic. A gate that requires frequent overrides devalues the gates that should be absolute.

What improves review quality most?

Smaller changes. A four-hundred-line diff reliably gets a worse review than four hundred-line ones, it waits longer, and no tooling compensates for either effect.

How do I know a check is worth keeping?

Measure whether its findings are acted on. A check whose comments are routinely dismissed is spending attention without earning it, and the override log tells you which rules are simply wrong.

Does automating review replace reviewers?

No. It removes the mechanical half so reviewers spend their attention on judgement — and review is also how people learn a codebase, which is a transfer that does not happen between a person and a tool.

How do I add a check to an existing codebase?

Baseline the current violations and enforce only on changed lines, so new code meets the standard without old code blocking anybody. Run it advisory for a fortnight first and look at what it would have blocked.

What about a repository-wide reformat?

Do it in one deliberate commit and add it to the blame ignore list, so history stays readable. Mixing a reformat into a functional change makes both impossible to review.

Related reading

• CI/CD pipelines explained
• Connecting GitHub to Slack
• Shift-left testing