Automating product feedback categorisation: the taxonomy problem

Sorting feedback into buckets is the easy part. The hard part is that the buckets determine what the product team can see, and most taxonomies are inherited from whoever set up the tool rather than designed for the decisions they are meant to inform.

The second hard part is quieter: once feedback is categorised, people stop reading it and start reading the counts. That is usually the point of the exercise, and it is also how a category containing one furious enterprise customer and forty trivial mentions comes to look like a priority.

What we'll cover

• Why the taxonomy is the whole problem
• Counting versus reading
• A taxonomy that informs decisions
• Building it with CodeWords
• What to extract beyond the category
• Routing, and who actually acts
• Keeping the taxonomy from drifting
• Closing the loop, which almost nobody does
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Why the taxonomy is the whole problem

Categories determine visibility. A theme without a category is invisible however often it is mentioned, and it will stay invisible for as long as the taxonomy stands.

Most taxonomies mirror the org chart, which tells you who owns something and not what customers are experiencing.

Feature-shaped categories hide the problem. "Reporting" as a category collects complaints about speed, about exports, and about a missing metric, which are three different pieces of work.

A catch-all absorbs the interesting material. Every taxonomy has an "other", and it is systematically where the new things live.

Too many categories is as bad as too few, because nothing accumulates enough volume to be visible and the classifier becomes unreliable at the margins.

Changing it later is expensive since historic data is categorised under the old scheme, which is the argument for spending real time on it at the start.

Counting versus reading

Counts answer "how many", which is rarely the question. The question is usually whether this blocks somebody important, or whether it is the same underlying issue as three other categories.

Frequency and severity are uncorrelated. The most frequent feedback is often cosmetic; the most serious is often mentioned once, by somebody about to leave.

Weight by something. Account value, segment, whether the person is a decision-maker, whether they have said it before — any of these produce a more useful ranking than raw volume.

Keep the verbatim attached. A category with no example is an abstraction people argue about; a category with three quotes is a conversation about a real thing.

Sample deliberately. Somebody should read a random selection every week regardless of what the counts say, because that is how you notice a category is wrong.

Report both. The count for tracking, the quotes for understanding, and never the count alone.

A taxonomy that informs decisions

Separate the type from the subject. Bug, feature request, confusion, and praise are types; the area of the product is the subject. Conflating them produces a list nobody can act on.

Make confusion its own type. It is neither a bug nor a request, it is extremely common, and it is usually the cheapest thing to fix.

Keep the top level small — under a dozen — and put the detail in a second level.

Name by customer experience, not by internal component, since customers do not know your architecture and the classifier only sees their words.

Allow multiple labels. Most real feedback is about more than one thing, and forcing a single choice loses information at exactly the point it is collected.

Include a "cannot tell". Genuinely ambiguous feedback exists and marking it honestly is better than a confident wrong label.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the support desk, the survey tool, the app store reviews and the tracker can feed one pipeline.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which sources, the taxonomy, what to extract, and where each type goes.
3. Authorize the connections — the feedback sources, the CRM for account context, and the tracker.
4. Describe the exceptions: ambiguous feedback, low confidence, anything from a named account.
5. Run it against a few hundred pieces you have already categorised by hand and compare.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to extract beyond the category

The underlying request, stated plainly, since customers describe solutions and the useful thing is the problem behind it.

Severity in their terms — blocked, working around it, or would be nice.

Whether they have a workaround, which changes the urgency more than any other single fact.

The account and its context from the CRM, so a piece of feedback arrives with its weight attached.

Whether this is a repeat from the same account, because the third mention means something the first did not.

A confidence score, surfaced, so low-confidence labels can be reviewed rather than silently counted.

Routing, and who actually acts

Bugs to the tracker, deduplicated against existing issues rather than creating a new one per mention.

Requests to the product backlog as evidence attached to a theme, not as individual tickets, since a backlog of raw feedback items is unreadable within a month.

Confusion to whoever owns documentation and onboarding, which is the cheapest queue in this whole system and the most neglected.

Anything from a named account to the account owner, immediately, regardless of category.

Praise to the team, which sounds soft and measurably affects whether people keep engaging with the pipeline.

Nothing should route to a channel nobody owns. A feedback channel with no owner is where this system goes to die.

Keeping the taxonomy from drifting

Watch the catch-all. A growing "other" is the signal that a new category is needed, and it is the single most useful metric this system produces.

Review misclassifications weekly, from the low-confidence queue, which is where the taxonomy's weak seams show first.

Merge categories that always co-occur, since two labels that always appear together are one category with two names.

Split categories that produce contradictory actions. If a category leads to three different teams doing three different things, it is three categories.

Version the taxonomy and record when it changed, because a step change in a category's volume is otherwise indistinguishable from a real shift in customer behaviour.

Re-run history after a change, or at minimum mark the boundary, so trend lines are not silently wrong.

Closing the loop, which almost nobody does

Feedback arrives, gets categorised, and then nothing goes back. The customer who reported a bug learns it was fixed by encountering the fix themselves, months later, if at all. This is the step that determines whether people keep telling you things.

Record who said what, at the point of ingestion, which is the only moment the link is cheap to keep.

Notify on resolution, tied to the tracker rather than to somebody remembering. When the issue closes, the people who raised it hear.

Be specific about what changed rather than sending a release-notes link and expecting them to find their item in it.

Say no as well as yes. A request that will not be built deserves a reply saying so, and in practice people take a clear no far better than silence.

Aggregate the reply for volume cases. Forty people reporting one thing get one honest note each, not a personal message that reads as a form letter.

Measure whether repeat reporting rises. Customers who hear back report more, and that increase is the return on this step — and the easiest part of the whole system to justify keeping.

Making it survive

Handle sources that go quiet. A source producing zero feedback for a week is usually a broken connection rather than happy customers.

Deduplicate across sources, since the same customer often says the same thing in two places.

Make ingestion idempotent, because every source will re-deliver at some point.

Preserve the original text untouched alongside whatever was extracted, permanently.

Report the outcome: pieces processed per source, category distribution, low-confidence rate, and the size of the catch-all.

Limits worth knowing about

Classification is uncertain, and a system that hides its uncertainty will be trusted until the first time somebody checks.

Sarcasm and understatement classify badly, and they are disproportionately common in feedback from frustrated long-term customers.

Multilingual feedback needs deliberate handling rather than being left to a default.

Feedback is not representative. It over-represents the very unhappy and the very engaged, and no amount of categorisation fixes that sampling problem.

Source rate limits apply, and a backfill across a year of support tickets will take longer than the first estimate.

What to build first

The weekly report that pairs the counts with three verbatim quotes per category and the size of the catch-all. It requires the taxonomy to exist but nothing else, and it immediately answers the question that matters: are these categories describing what customers are actually saying?

Two habits make the difference. Watch the catch-all as the primary metric rather than an afterthought, since a growing "other" is the earliest signal that something new is happening. And have somebody read a random sample every week whatever the counts say, because the failure mode of this whole system is that it replaces reading with counting and nobody notices until a category has been quietly wrong for two quarters.

Frequently asked questions

What makes a good feedback taxonomy?

Separate type (bug, request, confusion, praise) from subject (area of the product), keep the top level under a dozen, name categories by customer experience rather than internal components, and allow multiple labels per item.

Why should confusion be its own category?

Because it is neither a bug nor a feature request, it is very common, and it is usually the cheapest thing to fix — a documentation or onboarding change rather than engineering work. Folded into "bugs" it disappears.

Is the count of mentions a good priority signal?

On its own, no. Frequency and severity are uncorrelated: the most frequent feedback is often cosmetic and the most serious is often mentioned once. Weight by account value, segment, or repetition.

What is the most useful metric from this system?

The size of the catch-all category. A growing "other" is the earliest signal that customers are talking about something your taxonomy cannot see.

How do I stop automation replacing reading?

Attach verbatim quotes to every category in every report, and have somebody read a random weekly sample regardless of what the counts say. That sample is how you find out a category is wrong.

Should feature requests become tickets?

Not individually. Attach them as evidence to a theme, because a backlog of raw feedback items becomes unreadable within a month and then gets ignored wholesale.

What happens when I change the taxonomy?

Historic data stays under the old scheme, so either re-run the history or mark the boundary explicitly. Otherwise a step change from the taxonomy edit is indistinguishable from a real shift in customer behaviour.

Does categorised feedback represent my customers?

No. It over-represents the very unhappy and the very engaged, and categorisation does not fix that. Treat it as a source of themes to investigate rather than as a measurement of the customer base.

Should I tell customers when their feedback is acted on?

Yes, and almost nobody does. Record who reported what at ingestion, notify when the tracker closes the issue, and be specific about what changed rather than sending a release-notes link.

Is it worth replying to requests that will not be built?

Yes. People take a clear no considerably better than silence, and the teams that reply to declines see repeat reporting rise rather than fall.

Related reading

• Automating a weekly team digest
• Zendesk and CodeWords
• CodeWords integrations and templates