Article
Chatbot Testing: Test Cases, Checklist, and Automation
Learn how to test chatbot conversations, responses, integrations, and handoffs. Use practical test cases and compare automation and evaluation tools.

Chatbot testing looks at what a bot says and what happens after the conversation. A bot can tell a customer their delivery address has changed, even though the order still shows the old one. It might also save an address after the customer cancels the change.
Testing gets harder when a bot writes its own replies. Two answers may use different words and still mean the same thing, so an exact text match is often too strict. An order number needs a strict match because one wrong digit can point to another order.
This guide follows a customer changing the delivery address on an order. The tests cover the conversation and the saved result, even when the bot changes its wording. Later sections compare testing tools and show where Functionize Studio fits.
What Is Chatbot Testing?
Chatbot testing checks how a bot responds to a customer's request and what happens when that request leads to an action. A chatbot that only answers store-hour questions needs different checks from one that can change an order. In both cases, the test starts with a request and a result the business expects.
Suppose a customer wants to change a delivery address. The bot has to recognize the request, find the customer's order, and then wait for approval before making the change. A confirmation message is useless if the address is saved to the wrong order. The test has to check the order itself.
Not every problem will be in the conversation. A chat window can get stuck with keyboard focus when a new message comes in, making it very difficult for the user to type a reply. A screen reader may fail to announce the new message. If the order service stops responding, the bot must not claim the address was updated. The team can run the tests manually, or automate them depending on how often they need to run.
How Chatbot Design Changes What You Test
A fixed-flow bot simply follows the rules the developers write. An AI chatbot might generate a different response to the same question, or search the company's own documents before replying.
A delivery bot can understand “change my address” and still send the order service the wrong details. Another may update the right order, then promise delivery tomorrow against the store's policy.
A browser test could catch the wrong address by checking the saved record, but catching the false promise takes a reply check.
Chatbot Testing Example: Changing a Delivery Address
Let’s learn more about Chatbot Testing with an illustrative example: A signed-in customer wants to change the address on an order that has not shipped. The bot has to ask for confirmation before saving it. Order A123 belongs to the test customer and still hasn't shipped. The customer has an old address saved to their account, which becomes important halfway through the conversation. A change of mind tests more than a straight request because the bot must stop using the first address. Here's how the conversation plays out:
- Customer: “Change the delivery address for order A123.”
- Bot: Finds A123 and asks for the new address.
- Customer: Enters a made-up address used only for testing.
- Bot: Shows the proposed change and asks for confirmation.
- Customer: “Actually, use my previous address.”
- Bot: Asks which previous address the customer means if more than one could apply.
- Customer: Selects an address and confirms the change.
- Application: Saves the confirmed address to A123.
The fifth step changes what the bot should save. It can no longer rely on the earlier address, and “my previous address” may refer to more than one saved address. If the bot cannot identify the intended one under the store's rules, it needs to ask. The order must stay untouched until the customer gives a clear answer and approves the change.
A passing conversation is only the beginning. Under this policy, the bot should refuse to change a shipped order, while a request for someone else's order must fail at the permission check. Cancellation is different again: the customer has the right order but no longer wants anything changed.
For accepted requests, the test also checks that the bot didn't touch any other order. For rejected requests, the result is no change, not just a refusal in the transcript. An order record showing the wrong address still fails, even if the bot's final message sounds correct.
Chatbot Testing Checklist: What to Verify
A chatbot can follow a script and still miss a fundamental shift in the customer's request. It may answer the question correctly but fail when a connected service is down. The checks below examine those two issues in isolation, using observable outcomes rather than an overall "accuracy" score.
Conversation Understanding and Context
Real customers don't go with a script. They'll misspell words, cut themselves off mid-sentence, and refer back to something they said several messages earlier. In the address test, the bot should be able to treat “send it to my old place” as a potential correction rather than as a completely new order. If the reference is unclear, it's better to ask one question than guess.
Responses and Knowledge
Check the bot's phrasing against the store rules. The address change message should identify the correct order and show the address that is awaiting approval. If the bot says the change will take effect immediately, it must comply with store policy and must not state any shipping costs that someone hasn't approved.
Actions and Connected Systems
An action-taking bot needs permission to change a record, not just the ability to find it. The order service must reject another customer's order number even if the bot sends a request to change it. Approval also belongs in the test: a saved address before confirmation is a defect, regardless of the reply shown afterward.
Interface and Accessibility
Someone may complete the whole conversation without a mouse. If a new bot reply takes keyboard focus away from the input box, the next message becomes awkward to send. Screen-reader users need to hear new replies without having the entire conversation read again. Small phone screens also need attention, especially when a confirmation message is long.
Human Handoff and Recovery
Some conversations get to a point where the bot simply can't do the job. A customer may request an agent after the address service has failed. The handoff needs to route to the right support queue, and the agent needs enough context from the earlier conversation to continue. Outside of support hours, the bot must use the business' actual callback rules rather than invent a callback time.
How to Create Chatbot Test Cases and Conversation Datasets
A failed chatbot test is very difficult to debug if you can't reproduce the conversation. Imagine the scenario where A123 was shipped in one run and unshipped in the next. The same words could have very different outcomes in the two cases even if the bot itself is perfectly healthy. Good test case design involves recording the test's initial state and the application version used.
Customer support chats can show how people ask for help in language the test team didn't expect. After removing personal details, the team can turn those patterns into cases run against made-up accounts.
Previous failures should be in the dataset as well. If a bug once let a customer skip confirmation, then re-run that scenario after every change related to it. Data-driven testing allows you run the same address-change scenario against different customer phrases.
To test an attempt to change someone else’s order, you need a second test account. Keep test cases separate from examples used to tune the bot, or later scores may look better than they really are.
For a multi-turn test, the entire conversation is the case. Starting at “Actually, use my previous address” leaves out the address proposed just before it. Without that earlier turn, the bot has a different problem to solve.
How to Evaluate Chatbot Responses When Wording Changes
A chatbot may say “I've found order A123” one time and “Order A123 is here” another. Both are perfectly acceptable. A test looking for one specific sentence would fail the second response. The order number is a different matter: it must match exactly. A response referring to A124 would be incorrect, no matter how natural the rest of the response sounds.
The team should write pass conditions before reading the replies. For the address test, the bot must request approval before announcing any change. A few examples of acceptable answers can guide the grader, but a person should still review a sample of the results.
An LLM evaluator might approve a reply that wrongly promises next-day shipping. When that happens, the scoring rule needs to change rather than letting the mistake count as a pass. The team also needs to record each grader version, since a new scoring rule can change results even when the bot hasn't changed.
Repeated runs are worth looking at, particularly for requests which could result in a saved change. A bot that gets the address right 9 times and gets it wrong once is still a risk. Multiple runs under the same initial conditions can expose behavior a single run would miss. For A123, the confirmation wording may differ, but the permission check and the final saved address must be correct every time.
Security and Failure Scenarios for Chatbot Testing
A chatbot that can view orders is handling customer data, even if the chat itself appears harmless. Tests should try to open an order that belongs to a different customer and check that a new session doesn't expose the data from the previous account. Simply refusing the request in the response isn't sufficient if data has already been returned to the screen in the background.
Prompt injection poses another risk. A customer may type instructions to override the bot's rules, and an article the bot reads can contain similar instructions. OWASP calls these direct and indirect prompt injection.
A controlled test can hide a harmless instruction in a help article, then check that the bot treats the text as source material rather than an instruction to act. The important result is the bot's behavior, not how confidently it answers.
The service the bot calls needs its own access checks. A chatbot shouldn't be able to edit any order just because it can call an update service. In the address example, the order service would need to verify the customer's identity before accepting any change.
Failures can occur without an attack. A save request may time out after the user confirms, leaving the bot uncertain whether the update actually occurred. The next reply must reflect the system's actual outcome rather than a guess. Passing these tests demonstrates how the application behaves under the circumstances being tested.
How to Automate Chatbot Testing and Regression
With regression testing, you can check chatbot features affected by a change without running every test.
- Set the rules: Write the permitted customer action and the result the application must produce.
- Choose the cases: Include the normal conversation and the mistakes the bot must reject.
- Assign the checks: Decide which outcomes need application tests and which replies need separate scoring.
- Prepare the starting state: Use a fresh conversation with controlled customer records.
- Run the conversation: Replay every turn and inspect the application's action.
- Review the replies: Apply the scoring rules to generated answers, including refusals.
- Save failure details: Keep enough of the run to reproduce the problem.
- Compare releases: Check the new results against the previous version before approving changes.
The whole suite does not need to run after every edit. Changing a button in the chat widget mostly calls for browser checks. Updating the bot's instructions can change its replies without changing the button at all. The table shows which tests are worth running for each kind of change.
Live monitoring can catch problems missed in pre-release tests, but it finds them after someone may have been affected. LangSmith describes both offline testing and review of live conversations. The two serve different purposes, so a passing production score should not replace the cases a team runs before release.
A normal conversation test records the time until the first reply appears. A load test simulates many customers using the bot at the same time. You need to agree in advance what the traffic is for that load test, and the report should show the slowest responses, not just the average.
Chatbot Testing Tools and How to Choose an Approach
Testing a chat window is not the same job as judging a generated answer. Some products run customer actions through a browser, while others compare chatbot replies with written scoring rules. A bot that searches company documents may also need a separate check on the material it finds.
A website chatbot may never touch customer records. For that bot, the team could spend most of its time checking replies and the chat window. An order-support bot needs proof that it respects account permissions, even when the conversation sounds normal. If customers use several channels, the test must run through each supported channel.
The address change case is a fair comparison: each candidate gets the same request, and the team looks at what each one can prove about A123. A tool that reports a passed conversation without actually checking the stored address has missed half the job. The setup work and time spent investigating failures should factor into the cost comparison.
Metrics That Show Chatbot Testing Is Working
A report claiming “92% chatbot accuracy” tells a team little on its own. That number might reflect good replies, while the application still fails to complete some customer tasks. The two results need separate measures, and each measure needs a clear set of cases.
For the address change bot, one rogue update is worth flagging even if every other response passes the language check. The average masks the failure with a high overall score. You can look at customer engagement separately, of course, but a long conversation doesn't mean the bot handled the customer's request correctly.
Testing Chatbot Application Workflows With Functionize Studio
Functionize Studio tests the app side of the chatbot (the chat window and the records behind it). Studio doesn't do any model benchmarking or score hallucinations and retrieval quality. Specialist evaluation tools cover those, and teams can use those alongside Studio.
The demo starts with an address-change test in plain English. It names the signed-in customer and the address that A123 should end up with. Studio builds the steps from that description, and the tester reviews them before the first run. The run then opens the chat and plays each turn (including the correction).
The test only passes in two circumstances. Firstly, nothing is saved until the customer confirms. Second, once the customer confirms, the order record shows the correct address. Studio can verify that record directly,because its tests can call APIs and query databases and databases themselves rather than just the screen.
A second run tests cancellation. The customer says "never mind" at the confirmation stage. The test only passes if the original address remains in the record. If a bug saves the address, the run should fail, and the transcript should be displayed alongside with the record.
Studio runs the clicks and the checks, and people make the decisions. The team decides what the bot must do and what it must never do. You set the standard, and you make the release call.
The Agent Tasks page shows what else Studio can check, including APIs, email, and SMS. The Free plan is enough to build the address-change test on your own app. Sign-up is self-serve, so there's no need to talk to sales first.
How to Evaluate a Chatbot Testing Tool
A pilot should cover an approved change and a canceled request. The team should also attempt to change another customer's order and simulate a failed save. Before running the cases, reviewers need to agree on which results block a release.
An unauthorized update is a serious failure even if the bot politely refuses afterward. An awkward but correct reply may call for human review instead. Each run should save enough evidence for reviewers to decide without repeating the conversation.
Teams testing private applications can review Functionize's Enterprise options before setting up their pilot. The subscription price is only part of a pilot's cost. Every model-based evaluation may have a usage charge, while difficult failures take time to investigate. Recording that work during the trial gives the team a better picture of what regular testing will cost.
Frequently Asked Questions
How Do You Test a Chatbot?
Start with a customer request and the outcome the bot is expected to deliver. Run the conversation as a customer would and then check what actions (if any) have been taken as a result of the chat. If the bot is composing its own responses, then assess those against the written rules rather than against a single sentence.
What Should Chatbot Test Cases Include?
Each case needs the customer's goal and the complete conversation in order. It also needs the starting application state and the actions the bot must not take. Save the result and the software version so someone can reproduce a failure.
How Do You Test Multi-Turn Conversations?
Replay the conversation from the start rather than testing a later message on its own. Include a correction like “Actually, use my previous address” after the customer has already provided another address. The final action should follow the customer's most recent confirmed choice.
Should Chatbot Tests Use Exact Response Matching?
Exact matching works for fixed values (e.g., an order number). It's usually too rigid for replies where wording can vary without changing the meaning. Use written scoring rules for open-ended answers, but do maintain strict checks on the stored data.
How Is Application Testing Different From LLM Evaluation?
Application testing checks what the software actually did (e.g., did it save the correct address to A123). LLM evaluation looks at the response and its claims. You need both for an order support assistant, as a good message won't fix an incorrect change.
Can Chatbot Testing Be Fully Automated?
Many repeatable cases can run without someone watching each step. Human reviewers still need to handle unclear answers and judge serious failures that automated checks cannot settle. Exploratory sessions also find new situations that the existing test cases never covered.

