Measuring what WhatsApp automation is worth, without fooling yourself

The standard business case for messaging automation multiplies three numbers that were each guessed, and presents the product as a finding. Messages per day, minutes per message, cost per hour. Every one of them is an estimate, the errors compound, and the result is quoted to the pound.

The arithmetic is not the hard part. The hard part is the question underneath it: what would have happened if you had done nothing. Most of the value claimed for automation is value that would have arrived anyway, and separating the two is the whole exercise.

What we'll cover

• The 98% figure, and why it does not help
• Three questions the arithmetic has to answer
• Measure the baseline before you build
• The counterfactual problem
• Building it with CodeWords
• What you can measure honestly
• What belongs in the denominator
• What to stop claiming
• Presenting it to somebody who will push back
• Making it survive
• Limits worth knowing about
• What to measure first
• Frequently asked questions

The 98% figure, and why it does not help

You will see it everywhere: a 98% open rate for WhatsApp against roughly 20% for email. It appears without a study, a sample size or a date, and it is reproduced by companies selling messaging software, each citing another.

Set aside whether it is true. The deeper problem is that an open is not a business outcome. A message being opened tells you nothing about whether the recipient acted, whether they would have acted through some other route, or whether they were annoyed. A channel with a very high open rate and a low action rate is worse than the reverse, and the headline figure cannot distinguish them.

The useful version of that claim is narrow and you can test it yourself: for your own customers, a message on this channel is acted on sooner than the same request by email. That is measurable in a fortnight, it is specific to your business, and it is the only form of the claim worth putting in front of somebody.

Three questions the arithmetic has to answer

Any figure you present should survive these.

Compared with what? Against doing nothing, against a cheaper alternative, or against the way you work now. Most cases quietly compare against a strawman where nobody replies to anything.

Would it have happened anyway? The customer who booked after an automated reply may well have rung the next morning. Counting them as recovered revenue is the single commonest error in this arithmetic.

What did it cost, fully? Including your time building it, the time reviewing it, the messages it sends, and the work it creates elsewhere.

A case that answers all three will produce a smaller number than the ones in the marketing material, and it will survive somebody sensible reading it.

Measure the baseline before you build

You cannot compute a difference without a before, and the before has to be collected while you are still doing nothing.

Count the inbound. Volume by day and hour, for at least two ordinary weeks. Most businesses guess this badly in both directions.

Time the handling. Not the reply itself, the whole interruption, including getting back to what you were doing.

Record what actually happens to enquiries. How many convert, how long they take, how many go silent. This is the number the case will rest on and almost nobody has it.

Note the out-of-hours split. Enquiries arriving when nobody is available, and what currently becomes of them.

Two weeks of this is the most valuable thing in the whole exercise, and it is worth delaying the build to get it. Once the automation is live the baseline is gone for good.

The counterfactual problem

Here is the case that gets presented: ten after-hours enquiries a week, converting at thirty per cent, worth two hundred each, therefore six hundred a week recovered.

Here is what is wrong with it. Some of those people would have messaged again in the morning. Some would have rung. Some were never going to buy and replied to the bot out of politeness. The honest number is the conversion rate of after-hours enquiries with automation minus the rate without it, and you only have the second if you measured the baseline.

Where you can, run it properly. Hold back a group, or alternate by week, and compare. Where you cannot, use the before-and-after honestly and state plainly that it is not a controlled comparison, because seasonality and everything else you changed that quarter are sitting inside the difference.

Faster response almost certainly helps. The size of the effect is smaller than the naive calculation says, and a case that admits this is more persuasive than one that does not.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder, including what you want recorded about every conversation so the measurement is possible at all.
3. Authorize the connections to wherever enquiries and outcomes are tracked, and your WhatsApp Business Platform account.
4. Describe the exceptions, and make sure each one is logged rather than silently handled, since the exception rate is one of the numbers you will want.
5. Run it against a held-back group for a month rather than switching everybody at once, which is the difference between a comparison and an anecdote.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What you can measure honestly

Four things, all of them concrete.

Time to first response, before and after. This one is unambiguous and it is genuinely the automation's doing.

Proportion of conversations closed without a person. Count a conversation as handled only if nobody touched it and the customer did not come back with the same question two days later, which is stricter than most reporting and closer to the truth.

Enquiries that reach a resolution at all. Before automation a share simply evaporate. That share is countable and the change in it is meaningful.

Staff interruptions per day. Harder to value in money and often the thing people actually wanted.

Notice that none of these require guessing an hourly rate. They are quantities you can observe, and they make a better case than a fabricated pound figure because nobody can dispute them.

What belongs in the denominator

Cases fail scrutiny because the cost side is thin. It should include:

Per-message charges, which move with volume and by country, and which changed on 1 October 2026 when service messages became chargeable and in-window utility replies stopped being free.

Model costs where the bot generates replies, which grow with conversation length as well as count.

Build time, at what your time is actually worth.

Review time, which is permanent rather than a launch cost, because somebody has to read a sample of conversations for as long as the thing runs.

The escalation queue. Whatever the bot does not handle arrives somewhere, and that is staffing proportional to volume.

The second version. The first one will need rebuilding once real conversations exist, and budgeting for it is more honest than being surprised by it.

What to stop claiming

Compounding improvement. "It gets better as it learns" is untrue of most deployments, which do not learn from anything unless somebody changes them deliberately.

Deflection rates borrowed from elsewhere. Whether your bot handles half or four fifths of your messages depends on your customers and your business, and no published figure predicts it.

Revenue from recovered leads, unless you measured the counterfactual. Without it you are counting sales you would have made.

Cost per interaction figures with no method behind them. If you cannot say what was included, leave it out.

Each of these makes a case weaker rather than stronger in front of anybody numerate, and they are the reason finance people distrust automation proposals generally.

Presenting it to somebody who will push back

Lead with the measured facts and the method. Time to first response fell from this to that, over this period, measured this way. Conversations closed without a person, counted strictly, is this proportion.

Then give the value estimate as a range with its assumptions stated, and say which assumption the result is most sensitive to. A case that volunteers its own weak point is trusted; one that presents a single confident number invites a search for the flaw, and there always is one.

Keep the costs on the same page as the benefits. Separating them is the tell that somebody is selling rather than analysing.

Making it survive

Keep measuring after launch, monthly, on the same definitions.

Recount the baseline annually, since volume and mix drift.

Log exceptions and escalations as first-class numbers rather than as failures to hide.

Report cost per resolved conversation, which moves in the right direction only if the thing is genuinely working.

Write down the assumptions next to the figures, so next year somebody can tell which have stopped being true.

Limits worth knowing about

Attribution in messaging is genuinely hard, and a clean answer usually means somebody stopped asking.

Hours saved are not money saved unless the time went somewhere useful or somebody was not hired.

Small samples say very little, and a fortnight of conversations at low volume will not separate a real effect from noise.

The counterfactual is unavailable retrospectively. Once it is live, the chance to measure properly has gone.

Automation can move a cost rather than remove it, most often into the escalation queue, where it is less visible and no cheaper.

What to measure first

Two weeks of baseline, before building anything: inbound volume by hour, the true handling time including the interruption, what becomes of enquiries, and the out-of-hours split. Nothing else, and do not start the build during it.

Two habits make the difference. Hold a group back when you launch, because a comparison is worth more than a larger rollout a month earlier. And put the costs on the same page as the benefits, since the case that survives contact with a finance director is the one that did not need to be caught hiding anything.

Frequently asked questions

Is the 98% open rate for WhatsApp real?

It circulates widely without a study, a sample size or a date, and it is reproduced mostly by companies selling messaging tools. An open is also not a business outcome, so even a true figure would not tell you what to do.

How do I work out the ROI of WhatsApp automation?

Measure a baseline for two weeks first, then compare against a held-back group. The arithmetic is easy; the baseline and the counterfactual are what make the answer mean anything.

What is the counterfactual problem?

Much of what automation appears to recover would have happened anyway. The customer who replied at 9pm might have rung in the morning. Counting them as recovered revenue overstates the effect, often substantially.

What can I measure honestly?

Time to first response, the proportion of conversations closed without a person, the share of enquiries that reach any resolution, and staff interruptions per day. None requires guessing an hourly rate.

What costs should be included?

Per-message charges, model costs, build time, permanent review time, the escalation queue, and the rebuild you will need once real conversations exist.

Do bots improve on their own over time?

Generally no. Most deployments change only when somebody deliberately changes them, so compounding improvement is a claim to avoid unless you are actually doing the work.

Can I use published deflection rates?

They will not predict yours. What proportion a bot handles depends on your customers, your business and how the messages are worded, and the spread between businesses is wide.

How long before the numbers mean anything?

Long enough to accumulate a sample that separates an effect from noise. At low volume that can be a couple of months, and reporting a fortnight of data as a result is how cases lose credibility.

Are hours saved the same as money saved?

Only if the time was redeployed or a hire was avoided. Otherwise it is capacity, which can be valuable and is a different claim.

What makes a business case survive scrutiny?

Stating the method, giving a range rather than a point, naming the assumption the answer is most sensitive to, and keeping the costs on the same page as the benefits.

Related reading

• What a WhatsApp chatbot costs
• What to measure on a conversation
• The cost curve as volume grows