AI chatbots for customer support: deflection is the wrong target

Support bots are almost always measured on deflection: the proportion of conversations that did not reach a person. It is the easiest number to produce and it counts two opposite outcomes as the same thing — a customer whose question was answered, and a customer who gave up.

Those have very different consequences, and a bot optimised for the combined figure will drift towards the second. Measuring resolution instead changes the design, because it forces the handover to be good rather than rare.

What we'll cover

• Deflection counts failures as successes
• What to measure instead
• Questions a bot should answer
• Building it with CodeWords
• Questions it should never take
• The handover
• Knowing when it does not know
• What it says on your behalf
• Helping agents rather than replacing them
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Deflection counts failures as successes

A contained conversation is either an answered question or an abandoned one.

Abandonment looks identical to success in the metric, and it is cheaper, so optimising the number pushes towards it.

The customer who gave up frequently contacts you through another channel, which moves the cost rather than removing it.

Or they do not contact you again at all, which is the outcome the metric rewards most and the business wants least.

Public complaints follow, since somebody who cannot reach support tells somebody else.

The number is still worth having, alongside the ones that make it interpretable.

What to measure instead

Resolution: did the customer's problem get solved, which requires asking rather than inferring.

Repeat contact within a window, which is the strongest available proxy for an unresolved issue.

Handover rate and what happens after, since a good handover is a success rather than a failure of the bot.

Time to resolution end to end, including any human stage, rather than time to first response.

Satisfaction asked after resolution, not after the bot conversation, which measures something different.

Abandonment explicitly, so the two halves of deflection are separable.

Questions a bot should answer

Factual questions with a single correct answer — opening hours, policy details, feature availability.

Account-specific lookups where the customer is authenticated: order status, balance, delivery date.

Routine self-service actions: reset something, update a detail, cancel within a clear policy.

Navigation, meaning getting somebody to the right place, which is a large share of contacts.

Anything the documentation answers, which the bot can do faster than a person finding the page.

High volume and low ambiguity is the pattern, and most support volume fits it.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which questions are handled, which route to a person, and what context travels with them.
3. Authorize the connections — the helpdesk, the account or order system, and the knowledge base.
4. Describe the exceptions: an authenticated action, a complaint, a vulnerable customer, a question outside scope.
5. Run it against last month's real conversations and check what it would have said.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Questions it should never take

Anything where the customer is upset, which should route immediately regardless of topic.

Complaints, which have process requirements in some sectors and relationship consequences in all of them.

Anything involving money moving — refunds, disputes, payments — beyond stating a policy.

Cancellation, where an automated attempt to retain reads badly and is restricted in several markets.

Anything about a vulnerable customer's circumstances, which needs a person.

Anything it has already failed at once, since repeating the same answer is the single most frustrating experience a bot produces.

The handover

It is the most important part and is usually treated as the failure case.

Transfer the whole conversation, so the customer never repeats themselves — which is the most common complaint about bot interactions.

Say what the bot tried and why it escalated, so the agent starts informed.

Be honest about the wait, since an unexplained silence after a fast bot is worse than an accurate estimate.

Offer the handover early rather than making the customer discover the escape route.

Never hide it. A bot that makes reaching a person difficult converts a small problem into a complaint about your company rather than about the question.

Knowing when it does not know

Not knowing is an acceptable answer and should be available rather than engineered out.

The alternative is confident wrongness, which on a support channel is a statement your company has made.

Set a confidence threshold and hand over below it.

Escalate on repetition, since a customer rephrasing twice is telling you it is not working.

Watch the questions it declined, which is the list of what to add and is the most useful output of the whole system.

Never invent policy. An answer about a refund window or a warranty that the bot constructed is a commitment somebody may hold you to.

What it says on your behalf

Its answers are your statements, and are treated as such by customers and in several jurisdictions by regulators.

Ground it in your actual documentation rather than allowing general knowledge to fill gaps.

Review what it says about anything regulated — claims, guarantees, safety, eligibility.

Keep a transcript, since a disputed answer needs a record.

Tell people they are talking to a bot, which is required in some jurisdictions and is good practice everywhere.

Update it when policy changes, because a bot stating last quarter's policy is a specific and avoidable liability.

Helping agents rather than replacing them

The application with the clearest return is not customer-facing at all. A bot answering customers is visible and contested; one helping an agent answer is neither.

Suggest the answer, with the source, for the agent to send or edit — which is faster than searching and keeps the judgement human.

Assemble the context before the agent opens the conversation: account, history, recent orders, prior contacts.

Surface the similar resolved case, since most contacts resemble one that was handled last month.

Draft the reply for editing, and record what agents change, which is the best available signal about where the suggestions are wrong.

Flag the policy constraint relevant to this case, so an agent does not have to remember every rule.

It is easier to adopt, because an agent-facing error is caught by the agent and a customer-facing one is not.

Making it survive

Fail to a person, always, rather than to an error or a loop.

Handle the knowledge base being unavailable by handing over rather than answering from nothing.

Monitor for the loop, where a conversation cycles without progressing, which is the most damaging failure pattern.

Keep the escape route visible at every point in the conversation.

Report the outcome: conversations, resolved, handed over, abandoned, and the questions it could not answer.

Limits worth knowing about

Coverage is bounded by your documentation, and a bot cannot answer what nobody wrote down.

Ambiguous questions are common, and customers describe symptoms rather than causes.

Tone matters and is hard, particularly with a frustrated customer.

Languages vary in quality, which means some customers get a materially worse service.

A bot cannot fix a product problem, and deflecting contacts about a genuine fault delays finding out about it.

What to build first

The handover, before any answering. A route to a person that carries the full conversation and is visible at every point. It makes everything else safe to add, because a bot whose failure mode is a fast, well-informed transfer to a human can attempt questions that one with no escape route cannot.

Two habits make the difference. Measure resolution and abandonment separately rather than reporting deflection, since the combined number rewards exactly the outcome you least want. And let it say it does not know, because confident wrongness on a support channel is a statement your company has made and may be held to.

Frequently asked questions

Why is deflection the wrong metric?

Because it counts an answered question and an abandoned conversation as the same outcome, and abandonment is cheaper — so optimising the number drifts towards customers giving up, which is the result the business least wants.

What should be measured instead?

Resolution, repeat contact within a window, handover rate and what happens afterwards, end-to-end time including the human stage, satisfaction asked after resolution rather than after the bot, and abandonment as its own number.

What should a support bot answer?

Factual questions with one correct answer, authenticated account lookups, routine self-service actions, navigation, and anything the documentation covers. High volume and low ambiguity is the pattern.

What should it never take?

Anything where the customer is upset, complaints, anything involving money moving beyond stating policy, cancellation, anything about a vulnerable customer, and anything it has already failed at once.

Why is the handover the most important part?

Because it is what makes everything else safe. Transfer the full conversation so the customer never repeats themselves, say what was tried, be honest about the wait, and offer it early rather than hiding the escape route.

Should the bot be allowed to say it does not know?

Yes, and engineering that out is a mistake. The alternative is confident wrongness, which on a support channel is a statement your company has made — particularly damaging when it invents a policy about refunds or warranties.

Do customers need to be told they are talking to a bot?

It is required in some jurisdictions and is good practice everywhere. Its answers are your statements, so a transcript should be kept for anything disputed.

What is the most useful output of a support bot?

The list of questions it could not answer. That is the documentation gap, stated precisely, and it is more valuable than the deflection figure.

Is an agent-facing assistant better than a customer-facing bot?

It is easier to adopt and safer, because an error is caught by the agent rather than reaching a customer. Suggested answers with sources, assembled context and similar resolved cases speed agents up without moving the judgement.

How do I tell whether the suggestions are good?

Record what agents change before sending. The edit pattern is the most honest available signal about where the suggestions are wrong, and it costs nothing to collect.

Related reading

• Automating product feedback categorisation
• Automating knowledge base updates
• Connecting Intercom to HubSpot