Automating sentiment analysis on reviews: the score is the weak part

Running sentiment analysis over your reviews produces a number that goes up and down, and almost nothing you can act on. The score tells you that customers are less happy than last month; it does not tell you about what, which is the only part that leads to a change.

The useful version extracts what each review is about and how the writer felt about that specific thing, then tracks those separately. It is more work than scoring, it is what makes the output actionable, and it changes the failures you have to design around.

What we'll cover

• What a single score costs you
• Aspects, which is the useful unit
• Where sentiment models fail predictably
• Building it with CodeWords
• The rating and the text disagreeing
• Comparing across sources
• Making the output actionable
• Validating the classification rather than assuming it
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

What a single score costs you

Mixed reviews average to neutral, which describes nothing. A review praising the product and condemning the delivery is not a neutral review.

It cannot be acted on. Nobody can fix "sentiment declined four points".

The rating already tells you the valence. On a platform with star ratings, a sentiment score adds little that the stars did not.

Movement is ambiguous. A decline could be one product, one region, one shipping partner, or a change in who is reviewing.

Small changes are noise, and teams routinely respond to a two-point shift that is within the normal variation.

It hides the important minority. A small number of reviews describing a serious problem is invisible in an average that is otherwise healthy.

Aspects, which is the useful unit

Extract what the review is about: the product itself, delivery, packaging, price, support, setup, documentation.

Score sentiment per aspect, so one review can be positive about two things and negative about a third.

Define the aspect list deliberately and keep it small enough to be reliable, with a catch-all you monitor.

Watch the catch-all's growth, which is how a new problem announces itself before you have a category for it.

Track aspects over time separately, since that is where a trend becomes a specific piece of work.

Attach examples to every aspect in any report, because the quote is what persuades somebody to act and the count never does.

Where sentiment models fail predictably

Sarcasm and understatement, which are common in reviews from articulate, frustrated customers — exactly the ones worth reading.

Negation over distance. "I expected it to be slow and it was not" trips a surprising number of approaches.

Comparatives. "Better than the last one" is positive about a change and says nothing about whether the product is good.

Conditional and hypothetical statements about what would have happened.

Domain vocabulary, where a word that is negative in general use is neutral in your category.

Non-native and translated text, which scores less accurately and therefore under-represents some of your customers in whatever you conclude.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so the review sources, the product catalogue, the tracker and your channels can meet in one pipeline.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which sources, the aspect list, and what should be reported or alerted.
3. Authorize the connections — the review platforms or their exports, and the destinations.
4. Describe the exceptions: a low-confidence classification, a language you do not handle, a review with no text.
5. Run it over a few hundred reviews you have read yourself and compare the aspects it found with what you saw.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

The rating and the text disagreeing

The most informative signal in review data and the one a sentiment score erases.

A high rating with critical text usually means a loyal customer flagging something — which is the most valuable feedback you receive and the least likely to be escalated.

A low rating with mild text often means an unmet expectation rather than a defect, and it is frequently addressable with a description change.

Surface the mismatches specifically, as their own queue, since they are rare enough to read individually.

Do not correct the rating. The disagreement is the information, not an error to resolve.

Check the distribution by product, since a product with many mismatched reviews is usually one where the listing promises something slightly different from what arrives.

Read them. This is the one category where the count adds nothing and reading twenty of them changes what you do.

Comparing across sources

Rating scales differ and normalising them is a judgement rather than arithmetic.

Populations differ. Marketplace reviewers, your own site's reviewers and app store reviewers are different people with different expectations.

Verified purchase matters, and where it is available it should be a dimension rather than a filter.

Prompting affects the distribution. Reviews collected by an automated request skew differently from spontaneous ones, and mixing them silently makes trends meaningless.

Compare within a source over time rather than between sources at a point in time, which is the comparison that is actually valid.

Say which source a number came from on every report, because these get quoted without their provenance almost immediately.

Making the output actionable

Route by aspect, not by sentiment. Delivery complaints to operations, setup complaints to documentation, defect reports to the product team.

Alert on a new aspect appearing rather than on the score moving, which is the earlier and more specific signal.

Quantify against volume. Eleven complaints about one thing out of four hundred reviews is a different situation from eleven out of thirty.

Link to the product or batch where you can, since a problem concentrated in one product is a specific piece of work rather than a general concern.

Report change, not level. "Packaging complaints tripled this month" prompts action; "packaging sentiment is 0.4" does not.

Include the quotes. Every report, every aspect, three examples.

Validating the classification rather than assuming it

Aspect extraction is confident about everything, including the things it has wrong, and nothing in the pipeline notices. The checking has to be deliberate.

Label a sample by hand, a few hundred reviews, and measure agreement per aspect rather than overall. The average hides the one aspect that is badly wrong.

Re-check after any change to the taxonomy or the model, since a change that improves one aspect frequently degrades another.

Watch the low-confidence rate as a running health measure — a rising rate means the reviews have changed even if nothing else has.

Check by language and by product, since accuracy varies by both and the overall figure conceals it.

Have somebody read a random sample weekly whatever the numbers say, which is how you discover an aspect is being systematically misread.

Publish the accuracy alongside the results, because a report presented without it will be treated as exact and then disbelieved entirely the first time somebody checks.

Making it survive

Keep the original text permanently, alongside anything derived from it.

Handle source outages by reporting them, since a source going quiet looks like a change in customer behaviour.

Deduplicate across sources, since syndicated reviews appear in several places.

Version the aspect taxonomy and record which version produced a classification, or historical trends are not comparable.

Report the outcome: reviews processed per source, aspect distribution, low-confidence rate, and the catch-all's size.

Limits worth knowing about

Reviews are not representative. They over-sample the very pleased and the very annoyed, and no amount of analysis fixes that.

Volume is uneven across products, so a small-volume product has no reliable signal at all.

Platform terms restrict how review content may be stored and reused, which varies by source.

Fake reviews exist in both directions and distort any aggregate, particularly for smaller products.

Sentiment is not satisfaction, and neither is a proxy for whether somebody will buy again.

What to build first

The mismatch queue: reviews where the rating and the text disagree, presented individually for somebody to read. It requires no taxonomy, it is a small enough set to read, and it contains the feedback that is most useful and least likely to reach anybody — the loyal customer quietly flagging a real problem inside a five-star review.

Two habits make the difference. Report change rather than level, since a moving number prompts action and a standing one does not. And attach three real quotes to every aspect in every report, because the count tells somebody the size of a problem and the quote is what makes them do something about it.

Frequently asked questions

Why is a single sentiment score not useful?

Because mixed reviews average to neutral, nobody can act on "sentiment declined four points", and on platforms with star ratings the valence was already known. It also hides a small number of reviews describing a serious problem.

What should be extracted instead?

Aspects — what the review is about, such as the product, delivery, packaging, price, support or setup — with sentiment scored separately for each, so one review can be positive about two things and negative about a third.

Where does sentiment analysis fail?

Sarcasm and understatement, negation over distance, comparatives, conditionals, domain-specific vocabulary, and non-native or translated text — which scores less accurately and under-represents those customers in your conclusions.

What does it mean when the rating and the text disagree?

A high rating with critical text is usually a loyal customer flagging something real, which is the most valuable and least escalated feedback you get. A low rating with mild text is often an unmet expectation rather than a defect.

Can I compare reviews across platforms?

Compare within a source over time rather than between sources at a point in time. Scales differ, populations differ, and prompted reviews skew differently from spontaneous ones.

How should the output be routed?

By aspect rather than sentiment — delivery complaints to operations, setup complaints to documentation, defect reports to product. Alert on a new aspect appearing rather than on the score moving.

Are reviews representative of my customers?

No. They over-sample the very pleased and the very annoyed. Treat them as a source of themes to investigate rather than as a measurement of the customer base.

What is the most useful thing in a review report?

Three real quotes per aspect. The count tells somebody how big a problem is; the quote is what makes them act on it.

How do I know the aspect extraction is accurate?

Hand-label a few hundred reviews and measure agreement per aspect rather than overall — the average hides the one aspect that is badly wrong. Re-check after every taxonomy or model change, and break the figures down by language and product.

Should accuracy be published with the results?

Yes. A report presented as exact will be treated as exact, and then disbelieved entirely the first time somebody checks a number themselves.

Should I act on a small monthly change in sentiment?

Usually not. Small movements are within normal variation, and responding to a two-point shift is how teams end up chasing noise. Report change per aspect against volume instead.

Related reading

• Automating product review requests
• Automating product feedback categorisation
• Automating social proof collection