# Testing a WhatsApp bot before customers are the ones finding the bugs You cannot test this yourself, the output is not deterministic, and the cases that matter are the ones you would never think to type. The person who built a bot is the worst possible tester of it. You know which words it expects, you phrase things the way the flow was written, and you unconsciously avoid the paths you suspect are weak. Every one of your test conversations succeeds, and none of them resembles what a customer sends. There is a second problem underneath that one. A bot built on a language model does not give the same answer twice, so a test that passed once has not established that it passes. Testing here is closer to sampling than to the pass-or-fail checks most people have in mind, and designing for that changes what is worth doing. ## What we'll cover - [Why you cannot test your own bot](#why-you-cannot-test-your-own-bot) - [Non-determinism changes what a test means](#non-determinism-changes-what-a-test-means) - [Build the phrase bank first](#build-the-phrase-bank-first) - [The four things to test before anything else](#the-four-things-to-test-before-anything-else) - [Building it with CodeWords](#building-it-with-codewords) - [Testing the paths nobody completes](#testing-the-paths-nobody-completes) - [Templates ship to everybody at once](#templates-ship-to-everybody-at-once) - [Testing in a language you do not speak](#testing-in-a-language-you-do-not-speak) - [Regression, which is the ongoing cost](#regression-which-is-the-ongoing-cost) - [Making it survive](#making-it-survive) - [Limits worth knowing about](#limits-worth-knowing-about) - [What to test first](#what-to-test-first) - [Frequently asked questions](#frequently-asked-questions) ## Why you cannot test your own bot Three biases, and they are not fixable by trying harder. **You know the vocabulary.** You wrote "what is the model number", so you type a model number. A customer types "it's a Bosch, the silver one". **You follow the path.** You answer the question that was asked, in order, one thing at a time. Real people answer three questions at once, change their mind, and go quiet for a day. **You avoid what you suspect.** Nobody deliberately walks into the part of their own system they know is thin. The practical response is to stop treating your own conversations as evidence. Use them to check it runs; use other people's to find out whether it works. Somebody who has never seen the flow, given a realistic situation and no instructions, will find more in fifteen minutes than you will in a day. ## Non-determinism changes what a test means The same message can produce different replies on different runs, which makes "I tested that" a weaker statement than it sounds. Three consequences worth building around. **Run each case several times.** A path that works four times out of five is not working, and you only see the fifth by repeating. **Test the shape, not the string.** Asserting exact wording will fail constantly and teach you to ignore failures. What matters is whether the right information appeared, the right field was captured, and the right action was or was not taken. **Watch for confident wrongness.** The failure that matters is a plausible answer that is untrue: an invented price, a date misread, a policy the business does not have. Those look like successes unless somebody who knows the business reads them. That last point is the argument for having a person who knows the real answers read a sample of output, regularly, for as long as the bot runs. ## Build the phrase bank first The most useful testing asset is a list of things real customers have actually written, and it costs nothing to assemble. Take a month of your existing messages, emails, voicemail transcripts, whatever you have. Pull out the opening lines. Do not tidy them. Keep the typos, the missing punctuation, the two-word messages, the four-paragraph ones, the ones that start with "hi are you open". That list is your test set. It has the enormous advantage of being real, and it will contain phrasings nobody on your side would have invented. Add to it every week from live conversations, particularly from anything the bot handled badly. Where you have no history, borrow the shape from wherever customers currently reach you, and accept that the first fortnight of live conversations is itself the test set you did not have. ## The four things to test before anything else Not features. Failure modes, in order of what they cost. **The safety and distress escalations.** Test each one with several real phrasings, including the vague ones people actually use. Check the alert arrives, at a person, who is available. This is the only category where a single failure is unacceptable rather than disappointing. **The opt-out.** Ask it to stop, in three different wordings, then wait and see what still arrives, including from other automations. Almost nobody tests this and it is the failure that generates complaints. **Anything that costs money or moves an appointment.** Confirm that nothing consequential happens without an explicit confirmation, and try to trick it into acting early. **The failure message.** Send it something it cannot handle and read what comes back as though you were already annoyed. If it apologises and offers nothing, fix it before launch rather than after. Those four take an afternoon and they cover most of what can genuinely harm you. ## Building it with CodeWords CodeWords connects to more than 3,000 integrations, and the connection is made once and reused. 1. Open [CodeWords](https://codewords.agemo.ai) and start a new automation. 2. Describe what should happen in plain language to Cody, the automation builder, including what must never happen without confirmation and what has to escalate. 3. Authorize the connections to your existing systems and your WhatsApp Business Platform account, using test records rather than live ones while you are still checking behaviour. 4. Describe the exceptions, then deliberately try to break each one, since the exception paths are the ones that are never exercised by accident. 5. Run your phrase bank through it before any customer sees it, and repeat the whole set whenever you change the instructions. You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the [pricing page](/pricing). ## Testing the paths nobody completes The clean run through works. It is the only case that reliably does, and it is the least common thing that happens. **Abandon halfway** and come back three days later. Does it resume sensibly, or assert something that has since changed? **Answer out of order**, giving it three things before it asks for any of them. **Change an earlier answer** after the flow has moved on, especially one something depends on. **Send the wrong thing.** A photograph where a number was asked for, a voice note mid-flow, an emoji. **Say something entirely unrelated** partway through and then return to the task. **Send two messages in quick succession** before it has replied to the first, which is extremely common and breaks naive implementations. Each of these is ordinary customer behaviour and each is a path that no designed test plan includes. ## Templates ship to everybody at once Worth separating, because the failure mode is different. A conversational reply that reads badly affects one person. An approved template with a mistake in it, or a placeholder that renders wrongly, goes to everybody on the next send. There is no gradual discovery. So test templates with real data rather than sample values. Check the longest plausible name, an empty optional field, a price with no decimals, a date at a month boundary. Check the rendered length in every language you send in, since the same label overflows differently. And send one to yourself before any batch, every time, which takes thirty seconds and has saved more embarrassment than any other habit here. ## Testing in a language you do not speak You cannot evaluate this, and pretending otherwise is how businesses ship a bot that is fluent and rude. Get a speaker to read real conversations rather than sample sentences, and ask them specifically about register rather than about accuracy. A translator asked whether something is correct will say yes about a sentence that addresses a stranger far too informally. Test the failure message and the decline in each language, since those are the highest-traffic messages and the least reviewed. And repeat the exercise after any change to the instructions, because tone drifts in ways that do not show up in the language you do speak. ## Regression, which is the ongoing cost Changing one instruction changes behaviour everywhere, which is the property that makes these systems quick to build and awkward to maintain. A fix for one complaint routinely breaks something unrelated and nobody notices for a fortnight. So keep the phrase bank and re-run the whole thing after any change to the instructions, not just the case you were fixing. Keep the four failure-mode checks as a standing list and run them monthly regardless of whether anything changed, since the world changes underneath a bot that has not. Keep a short log of what was changed and when. When behaviour goes odd, the question is always what changed recently, and in most small teams nobody can answer it. ## Making it survive **Keep and grow the phrase bank**, weekly, from live conversations. **Re-run everything after any instruction change**, not only the case you fixed. **Have somebody who knows the real answers** read a sample of output regularly. **Test the opt-out and the escalations monthly**, whether or not anything changed. **Log changes with dates**, so regressions can be traced. ## Limits worth knowing about **You cannot prove correctness** on a non-deterministic system, only sample it. **Your own testing is systematically biased** towards the paths that work. **Exact-match assertions fail constantly** and train people to ignore failures. **A passing test says nothing about tone**, which needs a person to read it. **Live conversations are the real test set**, which is an argument for a small pilot rather than a long pre-launch. ## What to test first The four failure modes: the safety and distress escalations, the opt-out, anything that costs money, and the failure message. An afternoon, before anything else, and they cover nearly everything that can do real harm. Two habits make the difference. Build the phrase bank out of what customers have actually written rather than what you would write, because your own phrasing is the one thing guaranteed to work. And re-run the whole set after every change to the instructions, since the characteristic failure of these systems is a fix for one thing quietly breaking another. ## Frequently asked questions ### Why can I not test my own bot? You know the vocabulary it expects, you follow the path in order, and you avoid the parts you suspect are weak. Somebody who has never seen the flow will find more in fifteen minutes than you will in a day. ### What does non-determinism mean for testing? The same message can produce different replies, so a case that passed once is not proven. Run each several times, assert the shape rather than exact wording, and have somebody who knows the real answers check for confident wrongness. ### What is a phrase bank? A list of opening lines real customers have written, kept untidy, with the typos and two-word messages intact. It is the most useful testing asset available and it costs nothing to assemble from existing messages. ### What should be tested before launch? The safety and distress escalations, the opt-out in several wordings, anything that costs money or moves an appointment, and the failure message. An afternoon covers all four. ### Why test the opt-out? Because almost nobody does, and being messaged after asking to stop is the failure that reliably turns irritation into a complaint. Ask it to stop three ways and see what still arrives from other automations. ### Which conversation paths get missed? The incomplete ones. Abandoning and resuming days later, answering out of order, changing an earlier answer, sending the wrong media, going off topic, and sending two messages before it has replied to the first. ### Why are templates different? They go to everybody at once, so a mistake has no gradual discovery. Test with real data, check the longest plausible values and empty optional fields, verify rendered length in every language, and send one to yourself before any batch. ### How do I test a language I do not speak? Have a speaker read real conversations and ask specifically about register rather than accuracy. Check the failure and decline messages, which carry the most traffic and get the least review. ### What happens when I change the instructions? Behaviour changes everywhere, not only where you intended. Re-run the whole phrase bank after any change, and keep a dated log so an odd behaviour can be traced to what changed. ### When is testing finished? It is not. Keep the four failure-mode checks as a monthly list, grow the phrase bank from live conversations, and have a person read a sample of output for as long as the bot runs. ## Related reading - [The edge cases that actually matter](/blog/how-to-handle-edge-cases-in-whatsapp-chatbot-flows) - [The smallest version worth launching](/blog/how-to-build-a-whatsapp-bot-in-5-simple-steps) - [What a WhatsApp chatbot costs](/blog/how-much-does-it-cost-to-build-a-whatsapp-chatbot)