Testing a WhatsApp bot before customers are the ones finding the bugs

Test a WhatsApp bot with people who did not build it, using real customer phrasings, and run every case several times. The builder is the worst tester, because the builder knows the expected words and avoids weak paths. A bot built on a language model also never gives the same answer twice, so one passing test proves little.

Key takeaways

• The person who built a bot cannot test it fairly, because they know the vocabulary, follow the intended path and avoid the weak spots.
• A language-model bot gives different replies to the same message, so run each case several times and check the shape of the answer, not the exact wording.
• A phrase bank of real customer messages, typos included, is the most useful test set.
• Test four failure modes first: safety escalations, the opt-out, anything that costs money or moves an appointment, and the failure message.
• Re-run the whole test set after every change to the instructions, because a fix for one case can break another.

What we'll cover

• Why you cannot test your own bot
• Non-determinism changes what a test means
• Build the phrase bank first
• The four things to test before anything else
• Building it with CodeWords
• Testing the paths nobody completes
• Templates ship to everybody at once
• Testing in a language you do not speak
• Regression, which is the ongoing cost
• Making it survive
• Limits worth knowing about
• What to test first
• FAQs

Why you cannot test your own bot

You cannot test your own bot because of three biases, and trying harder does not remove them. Every test conversation you run succeeds, and none resembles what a customer sends.

You know the vocabulary. You wrote "what is the model number", so you type a model number. A customer types "it's a Bosch, the silver one".

You follow the path. You answer the question that was asked, in order, one thing at a time. Real people answer three questions at once, change their mind, and go quiet for a day.

You avoid what you suspect. Nobody deliberately walks into the part of their own system they know is thin.

Use your own conversations to check that the bot runs, and other people's to find out whether it works. Somebody who has never seen the flow, given a realistic situation and no instructions, finds more in fifteen minutes than you find in a day.

Non-determinism changes what a test means

Non-determinism means the same message can produce different replies on different runs, so "I tested that" is a weaker statement than it sounds. Testing here is closer to sampling than to a pass-or-fail check.

Run each case several times. A path that works four times out of five is not working, and only repetition shows the fifth.

Test the shape, not the string. Exact-wording assertions fail constantly and teach people to ignore failures. Check whether the right information appeared, the right field was captured, and the right action was or was not taken.

Watch for confident wrongness. The failure that matters is a plausible answer that is untrue: an invented price, a misread date, a policy the business does not have. These look like successes unless somebody who knows the business reads them.

Have a person who knows the real answers read a sample of output regularly, for as long as the bot runs.

Build the phrase bank first

A phrase bank is a list of things real customers have written, and it is the most useful testing asset. It costs nothing to assemble.

Take a month of your existing messages, emails and voicemail transcripts, and pull out the opening lines. Do not tidy them. Keep the typos, the missing punctuation, the two-word messages, the four-paragraph ones, and the ones that start with "hi are you open".

That list is your test set. It is real, and it contains phrasings nobody on your side would invent. Add to it every week from live conversations, particularly from anything the bot handled badly.

If you have no history, copy the shape of the messages from wherever customers reach you today. The first fortnight of live conversations then becomes the test set you did not have.

The four things to test before anything else

Test four failure modes before any feature, in order of what they cost. An afternoon covers all four, and they cover most of what can genuinely harm you.

The safety and distress escalations. Test each one with several real phrasings, including the vague ones people actually use. Check that the alert arrives at a person who is available. A single failure here is unacceptable, not merely disappointing.

The opt-out. Ask the bot to stop in three different wordings, then wait and see what still arrives, including from other automations. Almost nobody tests this, and it is the failure that generates complaints.

Anything that costs money or moves an appointment. Confirm that nothing consequential happens without an explicit confirmation, and try to trick the bot into acting early.

The failure message. Send the bot something it cannot handle and read the reply as though you were already annoyed. If it apologises and offers nothing, fix it before launch.

Building it with CodeWords

CodeWords connects to 3,000+ integrations (CodeWords pricing page, 2026), and each connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder, including what must never happen without confirmation and what has to escalate.
3. Authorize the connections to your existing systems and your WhatsApp Business Platform account, using test records rather than live ones while you check behaviour.
4. Describe the exceptions, then deliberately try to break each one, since exception paths are never exercised by accident.
5. Run your phrase bank through it before any customer sees it, and repeat the whole set whenever you change the instructions.

You describe the outcome; Cody builds it, connects it and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows (CodeWords pricing page, 2026). Details are on the pricing page.

Testing the paths nobody completes

Test the incomplete paths, because the clean run-through is the only case that reliably works and the least common thing that happens. Each path below is ordinary customer behaviour, and no designed test plan includes them.

Abandon halfway and come back three days later. Check whether the bot resumes sensibly or asserts something that has since changed.

Answer out of order, giving it three things before it asks for any of them.

Change an earlier answer after the flow has moved on, especially one that something depends on.

Send the wrong thing. Try a photograph where a number was asked for, a voice note mid-flow, or an emoji.

Say something unrelated partway through, then return to the task.

Send two messages in quick succession before the bot replies to the first. This is common and breaks naive implementations.

Templates ship to everybody at once

Templates need separate testing because a mistake reaches everybody on the next send, with no gradual discovery. A conversational reply that reads badly affects one person. An approved template with a mistake, or a placeholder that renders wrongly, affects the whole list.

Test templates with real data rather than sample values. Check the longest plausible name, an empty optional field, a price with no decimals, and a date at a month boundary. Check the rendered length in every language you send in, since the same label overflows differently.

Send one template to yourself before any batch, every time. It takes thirty seconds.

Testing in a language you do not speak

You cannot evaluate a language you do not speak, so get a speaker to read real conversations rather than sample sentences. Pretending otherwise is how a business ships a bot that is fluent and rude.

Ask the speaker about register, not accuracy. A translator asked whether a sentence is correct will say yes about a sentence that addresses a stranger far too informally.

Test the failure message and the decline in each language, since those are the highest-traffic messages and the least reviewed. Repeat the exercise after any change to the instructions, because tone drifts in ways that do not show in the language you speak.

Regression, which is the ongoing cost

Regression testing means re-running the whole test set after every change, because changing one instruction changes behaviour everywhere. That property makes these systems quick to build and awkward to maintain.

A fix for one complaint can break something unrelated, and nobody notices for a fortnight. Re-run the full phrase bank after any change, not just the case you were fixing. Run the four failure-mode checks monthly as a standing list, whether or not anything changed, since the world changes underneath a bot that has not.

Keep a short, dated log of what was changed. When behaviour goes odd, the first question is what changed recently, and in most small teams nobody can answer it.

Making it survive

Five habits keep a bot reliable after launch.

Keep and grow the phrase bank, weekly, from live conversations.

Re-run everything after any instruction change, not only the case you fixed.

Have somebody who knows the real answers read a sample of output regularly.

Test the opt-out and the escalations monthly, whether or not anything changed.

Log changes with dates, so regressions can be traced.

Limits worth knowing about

Testing a language-model bot has five limits.

You cannot prove correctness on a non-deterministic system, only sample it.

Your own testing is biased towards the paths that work.

Exact-match assertions fail constantly and train people to ignore failures.

A passing test says nothing about tone, which needs a person to read it.

Live conversations are the real test set, which is an argument for a small pilot rather than a long pre-launch.

What to test first

Test the four failure modes first: the safety and distress escalations, the opt-out, anything that costs money, and the failure message. They take an afternoon and cover nearly everything that can do real harm.

Two habits make the difference. Build the phrase bank from what customers have actually written, because your own phrasing is the one thing guaranteed to work. And re-run the whole set after every change to the instructions, since the characteristic failure of these systems is a fix for one thing quietly breaking another.

FAQs: WhatsApp Chatbot

Are WhatsApp bots illegal?

No, WhatsApp bots are not illegal when they run on the WhatsApp Business Platform and follow Meta's rules. From 15 January 2026, Meta bans general-purpose AI chatbots there, while bots that handle a business task stay allowed (respond.io, 2026).

Is there a free chat bot for WhatsApp?

Yes, there are free WhatsApp chatbot plans, but they come with tight limits. CodeWords Free includes a WhatsApp workflow capped at 5 DMs a day and 1 group (CodeWords pricing, 2026). Meta may still charge for some messages [VERIFY: Meta rate card for your country].

How much is a WhatsApp chatbot?

A WhatsApp chatbot costs a builder fee plus Meta's per-message fees.

• Builder: CodeWords Pro $39/mo includes a WhatsApp agent (pricing, 2026).
• Larger teams: CodeWords Business $100/mo.
• Meta fees: vary by country and message type [VERIFY: rate card].
• Custom build: developer time plus hosting.

Can I send bulk messages using the WhatsApp Business API?

Yes, you can send bulk messages using the WhatsApp Business API, but only approved templates to people who opted in. Daily sending limits grow with your quality rating [VERIFY: current Meta messaging limit tiers].

Related reading

• The edge cases that actually matter
• The smallest version worth launching
• What a WhatsApp chatbot costs