How to automate pull request reviews: latency is the problem

Teams frustrated with code review usually reach for another automated checker. That helps at the margin, and it does not address what people are actually complaining about, which is that pull requests sit for two days and then get a rubber-stamp approval from someone who did not have time to read eight hundred lines.

The automation with the most leverage is not another reviewer. It is the process around review: how large the pull requests are, who they go to, how long they wait, and what happens when they are stuck. That is measurable, it is unglamorous, and it changes review quality more than any analysis tool.

What we'll cover

• Why reviews are shallow
• The numbers worth measuring
• Size, the upstream fix
• Building it with CodeWords
• Routing to the right reviewer
• Nudging without nagging
• Helping the reviewer start
• The reviews that should not be automated at all
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Why reviews are shallow

Size defeats attention. Beyond a few hundred lines, review quality falls off sharply and reviewers switch from reading to scanning.

Waiting kills context. By the time a review arrives, the author has moved on, and the back-and-forth costs both people a context switch.

Nobody in particular is assigned. A pull request sent to a team is a pull request sent to nobody, and it waits for whoever feels guiltiest.

Reviewers lack the context to review well. The description says "fix bug" and the reviewer is reconstructing the intent from the diff.

There is no cost to a slow review, while there is a visible cost to a slow author, so the incentives point the wrong way.

Approval becomes a formality, at which point the process is consuming time and producing nothing.

The numbers worth measuring

Time to first review, which is the number that determines how the process feels to everybody in it.

Time to merge, separately, since a fast first review followed by three days of iteration is a different problem.

Pull request size distribution, in changed lines and files, which is the upstream cause of most of the rest.

Review depth — comments per review, and how many reviews have none — which is a rough proxy and an honest one.

Rework rate. Changes that come back after merge, which is the closest thing to an outcome measure available.

Report these to the team, not to individuals. The moment these become individual metrics, people optimise them directly and the numbers stop meaning anything.

Size, the upstream fix

Flag large pull requests automatically, at a threshold the team chose, with a suggestion rather than a block.

Measure the substantive diff, excluding generated files, lock files and vendored code, or the flag becomes noise immediately.

Suggest the split where it is obvious — a refactor mixed with a behaviour change is the common case and is easy to identify.

Separate mechanical changes. A rename touching two hundred files is reviewable in a minute on its own and unreviewable mixed with logic.

Encourage stacked pull requests where the tooling supports them, since that is the mechanism that makes small changes practical.

Show the distribution monthly. Teams respond to seeing their own size distribution far more than to a rule about it.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the repository, the ownership data, the calendar and Slack can meet in one automation.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the thresholds, the routing rules, and when a stalled review should be escalated.
3. Authorize the connections — the repository host, Slack, and the calendar if you want availability-aware routing.
4. Describe the exceptions: a hotfix, a draft, a dependency bump, an author who is also the only owner.
5. Run it in report-only mode for a fortnight and look at the review latency distribution before changing anything.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Routing to the right reviewer

Assign a person, always, since an unassigned review is an unowned one.

Use ownership where it exists, and fall back to recent contributors to the files rather than to the whole team.

Respect availability. Assigning to someone on leave is the most common cause of a pull request sitting for a week.

Balance the load, since review work concentrates on a few people invisibly and burns them out quietly.

Deliberately include someone unfamiliar occasionally, because review is how knowledge spreads and pure ownership-based routing prevents that.

Reassign automatically when a review has not started within a defined window, rather than waiting for the author to ask.

Nudging without nagging

Remind the reviewer, not the author, which is the opposite of what most teams do informally.

Once, in their own channel, after a defined period, rather than repeatedly in a shared one.

Escalate to reassignment, not to a manager, since the goal is a review rather than a conversation about a review.

Respect working hours and time zones, or the nudge becomes an out-of-hours message and people mute the system.

Notify on approval too, because authors frequently do not notice that a review arrived.

Do not publish a leaderboard. Naming slow reviewers produces fast shallow reviews, which is exactly the outcome you are trying to avoid.

Helping the reviewer start

Summarise what changed, structurally — which modules, what kind of change — so the reviewer knows what they are about to read.

Link the ticket and the description together, since the intent usually lives in one and the implementation in the other.

Highlight the risky parts: changes to auth, migrations, anything touching money, and anything with no test coverage.

Say what to look at first, which the author knows and almost never writes down.

Surface related recent changes to the same area, which is the context a reviewer would otherwise have to go and find.

Enforce a description minimally — a template with two required lines beats a blank box and beats a long template nobody fills in.

The reviews that should not be automated at all

Some changes need a different process entirely, and treating them as ordinary pull requests is how they get approved without anybody really looking.

Security-relevant changes — authentication, authorisation, anything touching secrets — should route to someone who reviews those specifically, not to whoever owns the file.

Database migrations, which are the changes that are hardest to undo and most often approved quickly because the diff is small.

Anything affecting money, where the review should include somebody who understands the business rule rather than only the code.

Changes to the review automation itself, which can otherwise be used to weaken the checks in the same change that needs them.

First contributions, which deserve a slower and more generous review than the routing rules would produce.

Emergency fixes, which should be merged and then reviewed properly afterwards rather than pretending the normal process happened under pressure.

Making it survive

Handle the repository host's rate limits, particularly on a busy monorepo.

Make reminders idempotent so a retry does not produce a second nudge.

Exclude bots and automated pull requests from every metric, or the numbers are meaningless.

Fail quietly. A broken review automation should not block merging, since that turns an inconvenience into an outage.

Report the outcome: reviews routed, reminders sent, reassignments made, and the latency distribution.

Limits worth knowing about

Ownership data goes stale as teams reorganise, and stale routing is worse than no routing.

Availability data is incomplete unless people keep calendars current, which not everybody does.

Line counts are a poor proxy for complexity — a fifty-line change to a scheduler can be harder than a thousand-line rename.

Comment counts measure activity, not quality, and a review with no comments may be a good review of a simple change.

Process automation cannot fix a culture where review is treated as an obstacle, though it does make that visible.

What to build first

The review latency report: time to first review per pull request, distributed rather than averaged, excluding bots. It changes nothing and it tells you whether your problem is latency, size, or depth — which determines everything else on this list and is usually different from what the team assumes.

Two habits make the difference. Assign an individual rather than a team from the start, since that single change typically removes the largest chunk of waiting. And report the numbers at team level only, because individual review metrics reliably produce fast shallow reviews, which is the outcome this whole exercise exists to prevent.

Frequently asked questions

Why are code reviews shallow?

Mostly size and latency. Beyond a few hundred lines reviewers scan rather than read, and a review that arrives two days later costs both people a context switch. Neither is fixed by adding another automated checker.

What is the single most useful metric?

Time to first review, shown as a distribution rather than an average. It determines how the process feels to everybody and it points at which of the other problems you actually have.

Should pull request size be enforced?

Flagged, not blocked, at a threshold the team chose, measuring the substantive diff only. Showing the team its own size distribution monthly changes behaviour more reliably than a rule.

Who should a pull request be assigned to?

A named individual, never a team. Use ownership where it exists, fall back to recent contributors to those files, respect availability, and balance load — review work concentrates on a few people invisibly.

Who should reminders go to?

The reviewer, once, in their own channel, respecting working hours — then escalate to reassignment rather than to a manager. The goal is a review, not a conversation about a review.

Should review metrics be published per person?

No. Individual review metrics produce fast shallow approvals, which is the failure mode you are trying to fix. Report at team level.

What helps a reviewer most?

Knowing where to start. A structural summary of what changed, the risky parts highlighted, related recent changes surfaced, and one line from the author saying what to look at first.

Does automated review analysis help at all?

At the margin, for the mechanical layer. It does not address size, latency or routing, which is what people are usually complaining about when they say review is not working.

Which changes need a different review process?

Security-relevant changes, database migrations, anything touching money, and changes to the review automation itself. These need a specific reviewer rather than whoever owns the file, and migrations in particular get approved fast because the diff is small.

How should emergency fixes be handled?

Merge, then review properly afterwards. Pretending the normal process happened under pressure produces an approval that nobody actually gave.

Are stacked pull requests worth the overhead?

Where the tooling supports them, yes. They are the mechanism that makes genuinely small changes practical, and small changes are the upstream fix for most review complaints.

How do I stop review work concentrating on a few people?

Balance it explicitly in the routing. It concentrates invisibly otherwise, and the people carrying it rarely say so until they are already burnt out.

Related reading

• Automated code review workflows
• Automating regression test triggering
• GitHub and CodeWords