How to automate bug triage without losing the judgement

Bug triage is two activities wearing one name. There is gathering — finding the reproduction steps, working out how many customers are affected, checking whether this is already a known issue, identifying which team owns the area. And there is deciding — how bad is it, does it displace something, who picks it up.

The gathering is most of the time and none of the judgement. Automating that half and leaving the decision to a person is the arrangement that works; automating the decision is what produces a backlog nobody trusts.

What we'll cover

• Gathering and deciding
• What arrives and what is missing
• Duplicate detection, which is the main win
• Building it with CodeWords
• Seven automations worth building
• Severity, which should stay human
• Routing without a black hole
• Measuring whether triage is working
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Gathering and deciding

Gathering is mechanical. Which version, which browser, which customer, what the logs say at that timestamp, whether anybody has reported this before.

Deciding is not. Whether a data corruption affecting two accounts outranks a layout bug affecting everybody is a judgement about the business, not a rule.

Most triage time goes to gathering, and most of it is the same few lookups performed by hand.

A report that arrives complete is triaged in a minute. One that arrives as "the export is broken" takes twenty, most of it spent finding out what was actually exported.

So automate the assembly and present it, rather than trying to encode the decision. The person then spends their time on the part that needs them.

What arrives and what is missing

The reports that cost the most are the ones missing something, and it is predictable which.

No reproduction steps. The single most common gap, and the reason a bug sits for days waiting for a reply.

No version. Which build, which client, which environment — frequently determines whether the bug still exists.

No customer. Without it, nobody can assess impact or tell the reporter when it is fixed.

No timestamp. Which makes the logs unsearchable, and the logs are usually where the answer is.

No expected behaviour. "It's wrong" without saying what right looks like is a conversation rather than a report.

Ask for these at the point of filing, as a short form rather than a free-text box, and the gathering problem mostly disappears. A form with five fields is answered; a template pasted into a description is ignored.

Duplicate detection, which is the main win

The highest-value automation here and the one that changes the backlog most.

The same bug is reported repeatedly by different customers through different channels, and each arrives as a new issue.

Search before creating, on the error text, the affected area, and the customer-visible symptom.

Add to the existing issue rather than creating a second, and increment a count of how many have reported it.

That count is prioritisation input. "Eleven customers have hit this" changes a decision in a way that eleven separate low-priority issues never do.

Report likely duplicates rather than merging automatically, since near-matches are sometimes genuinely different and merging is destructive.

Link the original reports to the issue, so whoever picks it up can see all of them rather than one.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so a report can be assembled from the support tool, the error tracker, and your own systems in one automation.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: where reports arrive, what should be gathered, and where the result should go.
3. Authorize the connections — the support tool, the issue tracker, the error tracker, and wherever the team works.
4. Describe the exceptions: a report with no reproduction steps, a likely duplicate, a report that is actually a question.
5. Run it against a test project before anything reaches the real backlog.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Seven automations worth building

Assemble the report. Pull the customer, the plan, the version, the recent errors at that timestamp, and any related support history into the issue when it is created.

Duplicate detection, with the count on the existing issue, as described above.

Link the error tracker. If the reported symptom matches a known exception, attach it — that connection is usually made by hand and frequently not at all.

Flag the incomplete report back to whoever filed it, immediately, while they still have the context, rather than three days later.

Impact assessment. How many customers, on which plans, which is data support and billing hold and engineering does not.

Route to the owning team based on the area, so it lands in front of somebody rather than in a general queue.

Close the loop. When the fix ships, tell the people who reported it — the step that determines whether support keeps filing properly.

Severity, which should stay human

Severity is a business judgement, and encoding it as rules produces confident wrong answers.

A rule cannot weigh a data problem against a cosmetic one, or know that this particular customer is mid-renewal.

What automation can do is surface the inputs — affected count, plans, revenue at risk, whether it is a regression — and let a person decide in seconds rather than minutes.

Flag the obvious ones. Anything touching data integrity, security, or money can be marked for immediate attention without claiming to have assessed it.

Watch for severity inflation. When everything filed is urgent, nothing is, and a periodic report of what was marked critical against what was actually worked on is uncomfortable and useful.

Never let automation downgrade. Raising a flag is safe; quietly lowering one is how a real problem gets buried.

Routing without a black hole

Route by area, not by person, since people move and areas do not.

Have a default with an owner. Unroutable reports go somewhere, and that somewhere needs a name attached or it becomes the place things go to be forgotten.

Report the default queue weekly, since its contents are either misrouted or genuinely unowned, and both need somebody.

Acknowledge to the reporter with what happens next, which stops the follow-up message asking whether anybody saw it.

Escalate on age, not on severity alone. A medium-severity bug untouched for a month is a decision nobody has made rather than a decision to do nothing.

Measuring whether triage is working

A few numbers tell you more than the backlog size, and none of them is usually tracked.

Time from report to triaged, as a distribution. The median is usually fine and the tail is where reports sit for a fortnight.

Reports returned as incomplete, which measures the filing route rather than the triagers — a high rate means the form is wrong, not that people are careless.

Duplicates caught, which is the direct evidence the detection is working and is usually higher than anybody expects.

Bugs reopened after being closed, which measures whether the fix or the verification is the weak step.

Age of the oldest untriaged report, which is the single number that shows whether the process is keeping up.

Not bugs closed. It rewards closing things, which is a metric with an obvious and unhelpful response.

Making it survive

Make it idempotent. Store the source identifier on the issue and check before creating, since reports arrive through several channels and the same one can come twice.

Handle the missing field as a case, reporting what could not be gathered rather than filing a half-empty issue.

Keep the original text. A summary loses the detail somebody will need, and the original is a link away.

Do not create from keywords. An automation filing an issue whenever somebody types "bug" produces a backlog nobody trusts.

Report the outcome. Issues created, duplicates avoided, reports returned as incomplete.

Limits worth knowing about

Duplicate detection is imperfect. Two reports of the same symptom can be different causes, which is why a flag beats an automatic merge.

Customer impact data lives outside the tracker, so this automation depends on connections to support and billing being maintained.

Error tracker matching is heuristic. A reported symptom and a logged exception are related by inference, not by identity.

Rate limits apply on both sides at volume, particularly during an incident when reports spike.

An automation cannot tell a bug from a misunderstanding, and a meaningful share of reports are the second — which is what the triage step is for.

What to build first

Duplicate detection with a report count on the existing issue. It is the automation that changes the backlog rather than merely populating it faster, and the count is the piece of information product teams consistently lack when deciding what to fix.

Two habits make the difference. Always search before creating, since an automation that files without looking is a duplicate generator and makes triage worse. And show the evidence for each suspected duplicate — which fields matched — so the person reviewing can judge in seconds instead of opening both.

Frequently asked questions

What part of triage should be automated?

The gathering: customer, plan, version, recent errors, prior reports, owning area. Not the decision about severity or priority, which is a judgement about the business that rules produce confident wrong answers for.

How do I stop the same bug being filed repeatedly?

Search before creating, on error text and customer-visible symptom, and add to the existing issue with a report count rather than creating a second. Flag likely duplicates for review rather than merging automatically.

Why is a report count so valuable?

Because "eleven customers have hit this" changes a prioritisation decision in a way that eleven separate low-priority issues never do. That information exists in support and reaches engineering only if something carries it.

Should automation set severity?

No. It should surface the inputs — affected customers, plans, whether it is a regression — and let a person decide. Flagging data, security, or money issues for attention is fine; assessing severity is not.

What is the most common gap in a bug report?

Reproduction steps, followed by version and timestamp. Asking for them as a short form at filing time is far more effective than a template in a description, which gets ignored.

Where should unroutable reports go?

A default queue with a named owner, reviewed weekly. A default queue belonging to nobody is where reports accumulate, and the weekly review is what distinguishes misrouted from genuinely unowned.

Why tell reporters when a bug is fixed?

Because it determines whether they keep filing properly. Support that never hears what happened to its reports stops writing good ones, and the quality of what reaches engineering degrades accordingly.

What should be measured?

Time from report to triaged as a distribution, incomplete reports returned, duplicates caught, reopen rate, and the age of the oldest untriaged item. Not bugs closed, which rewards closing rather than fixing.

A lot of reports come back as incomplete — whose problem is that?

The filing route's. A high rate means the form is asking for the wrong things or not asking at all, rather than that reporters are careless. Fix the form before asking people to try harder.

Related reading

• Automated ticket routing
• Automating customer feedback
• Linear and CodeWords