Article

Chatbot Testing: Test Cases, Checklist, and Automation

November 16, 2026

Learn how to test chatbot conversations, responses, integrations, and handoffs. Use practical test cases and compare automation and evaluation tools.

Learn how to test chatbot conversations, responses, integrations, and handoffs. Use practical test cases and compare automation and evaluation tools.

Chatbot testing looks at what a bot says and what happens after the conversation. A bot can tell a customer their delivery address has changed, even though the order still shows the old one. It might also save an address after the customer cancels the change.

Testing gets harder when a bot writes its own replies. Two answers may use different words and still mean the same thing, so an exact text match is often too strict. An order number needs a strict match because one wrong digit can point to another order.

This guide follows a customer changing the delivery address on an order. The tests cover the conversation and the saved result, even when the bot changes its wording. Later sections compare testing tools and show where Functionize Studio fits.

What Is Chatbot Testing?

Chatbot testing checks how a bot responds to a customer's request and what happens when that request leads to an action. A chatbot that only answers store-hour questions needs different checks from one that can change an order. In both cases, the test starts with a request and a result the business expects.

Suppose a customer wants to change a delivery address. The bot has to recognize the request, find the customer's order, and then wait for approval before making the change. A confirmation message is useless if the address is saved to the wrong order. The test has to check the order itself.

Not every problem will be in the conversation. A chat window can get stuck with keyboard focus when a new message comes in, making it very difficult for the user to type a reply. A screen reader may fail to announce the new message. If the order service stops responding, the bot must not claim the address was updated. The team can run the tests manually, or automate them depending on how often they need to run.

How Chatbot Design Changes What You Test

A fixed-flow bot simply follows the rules the developers write. An AI chatbot might generate a different response to the same question, or search the company's own documents before replying. 

Chatbot component Main thing to check
Rule-based flow Correct branches and behavior when a request cannot continue
Intent-based system Meaning of the request and details taken from the conversation
Generated replies Factual claims and acceptable answers when wording changes
Retrieval-backed answers The information retrieved and support for the final answer
Action-taking bot Permission to act and what was saved after approval

A delivery bot can understand “change my address” and still send the order service the wrong details. Another may update the right order, then promise delivery tomorrow against the store's policy. 

A browser test could catch the wrong address by checking the saved record, but catching the false promise takes a reply check. 

Chatbot Testing Example: Changing a Delivery Address

Let’s learn more about Chatbot Testing with an illustrative example: A signed-in customer wants to change the address on an order that has not shipped. The bot has to ask for confirmation before saving it. Order A123 belongs to the test customer and still hasn't shipped. The customer has an old address saved to their account, which becomes important halfway through the conversation. A change of mind tests more than a straight request because the bot must stop using the first address. Here's how the conversation plays out:

  1. Customer: “Change the delivery address for order A123.”
  2. Bot: Finds A123 and asks for the new address.
  3. Customer: Enters a made-up address used only for testing.
  4. Bot: Shows the proposed change and asks for confirmation.
  5. Customer: “Actually, use my previous address.”
  6. Bot: Asks which previous address the customer means if more than one could apply.
  7. Customer: Selects an address and confirms the change.
  8. Application: Saves the confirmed address to A123.

The fifth step changes what the bot should save. It can no longer rely on the earlier address, and “my previous address” may refer to more than one saved address. If the bot cannot identify the intended one under the store's rules, it needs to ask. The order must stay untouched until the customer gives a clear answer and approves the change.

A passing conversation is only the beginning. Under this policy, the bot should refuse to change a shipped order, while a request for someone else's order must fail at the permission check. Cancellation is different again: the customer has the right order but no longer wants anything changed.

Situation Expected behavior Evidence of the result
Customer owns an unshipped order Change saved after confirmation A123 holds the confirmed address
Customer cancels No change saved Original address remains in A123
Order has shipped Change refused under the policy Order record remains unchanged
Customer requests another person's order Access denied without revealing private details Permission result and bot reply
Previous address is unclear Bot asks which address is intended No change before the answer
Address service fails Bot does not claim success Error response and recorded order state
Customer asks for support Handoff follows the store's rules Ticket or routing record

For accepted requests, the test also checks that the bot didn't touch any other order. For rejected requests, the result is no change, not just a refusal in the transcript. An order record showing the wrong address still fails, even if the bot's final message sounds correct.

Chatbot Testing Checklist: What to Verify

A chatbot can follow a script and still miss a fundamental shift in the customer's request. It may answer the question correctly but fail when a connected service is down. The checks below examine those two issues in isolation, using observable outcomes rather than an overall "accuracy" score.

Conversation Understanding and Context

Real customers don't go with a script. They'll misspell words, cut themselves off mid-sentence, and refer back to something they said several messages earlier. In the address test, the bot should be able to treat “send it to my old place” as a potential correction rather than as a completely new order. If the reference is unclear, it's better to ask one question than guess.

Test area Example What should happen
Different wording “Update where my parcel goes” Address-change conversation starts
Other supported languages Same request in a supported language Correct flow in that language
Typos “chnage adress A123” Bot finds the request or asks a clear question
Mid-chat correction “Actually, use my previous address” Final action follows the correction
Topic change Customer asks about billing halfway through Address task can resume with its earlier context
Unclear reference “Use that one” when two addresses exist Bot asks which address the customer means
Request outside the bot's scope “What's the weather in Leeds?” Bot states its limit without inventing an answer
New session Customer returns after finishing the task Earlier chat details aren't carried into the new task

Responses and Knowledge

Check the bot's phrasing against the store rules. The address change message should identify the correct order and show the address that is awaiting approval. If the bot says the change will take effect immediately, it must comply with store policy and must not state any shipping costs that someone hasn't approved.

Test area Example What to check
Required facts Summary before confirmation Correct order number and proposed address
Policy statement “Changes take effect immediately” Statement matches the approved policy
Missing answer Question absent from support content Bot says it cannot confirm the answer
Conflicting sources Two articles give different cutoff times Current rule used or conflict made clear
Unsupported promise “I'll refund your delivery fee too” No promise outside the store's rules
Fallback Request the bot cannot handle Customer gets a useful next step

Actions and Connected Systems

An action-taking bot needs permission to change a record, not just the ability to find it. The order service must reject another customer's order number even if the bot sends a request to change it. Approval also belongs in the test: a saved address before confirmation is a defect, regardless of the reply shown afterward.

Test area Example What to check
Correct order Save an address for A123 No other order changed
Correct data Customer confirms the second address Stored fields match that address
Access rules Customer supplies someone else's order number Backend refuses the change
Early save Bot tries to update before confirmation No write takes place
Cancellation “Never mind” at the last step Original address remains
Repeated confirmation Customer sends “yes” twice No duplicate action under the agreed retry rule
Service timeout Save request does not finish Bot reports the problem; order state is checked

Interface and Accessibility

Someone may complete the whole conversation without a mouse. If a new bot reply takes keyboard focus away from the input box, the next message becomes awkward to send. Screen-reader users need to hear new replies without having the entire conversation read again. Small phone screens also need attention, especially when a confirmation message is long.

Test area Example What to check
Keyboard access Complete the address change without a mouse Required controls remain reachable
Keyboard focus Bot sends a new reply Focus stays where the user can continue
Message display Long reply followed by another New content can be found and read
Screen reader Bot sends a message New reply is announced appropriately
Screen size Open the widget at supported widths Input and confirmation controls stay usable

Human Handoff and Recovery

Some conversations get to a point where the bot simply can't do the job. A customer may request an agent after the address service has failed. The handoff needs to route to the right support queue, and the agent needs enough context from the earlier conversation to continue. Outside of support hours, the bot must use the business' actual callback rules rather than invent a callback time.

Test area Example What to check
Request for a person “I want to speak to an agent” Ticket or queue entry created
Conversation transfer Handoff after address discussion Agent receives the relevant transcript
Support unavailable Handoff requested after hours Message matches real support availability
Closed queue No agent can take the request Customer gets a clear next step

How to Create Chatbot Test Cases and Conversation Datasets

A failed chatbot test is very difficult to debug if you can't reproduce the conversation. Imagine the scenario where A123 was shipped in one run and unshipped in the next. The same words could have very different outcomes in the two cases even if the bot itself is perfectly healthy. Good test case design involves recording the test's initial state and the application version used.

Field What to record
Case ID Stable name or number
Customer goal Task the conversation is meant to finish
Starting conditions Test account, permission level, and order state
Conversation User turns in their original order
Required behavior Result the bot must produce
Prohibited behavior Action or claim that must never happen
Evaluation method Fixed assertion, written scoring rule, or human review
Evidence Transcript and actual application result
Version details App build, model settings, and knowledge source version
Severity Business impact of a failed case

Customer support chats can show how people ask for help in language the test team didn't expect. After removing personal details, the team can turn those patterns into cases run against made-up accounts. 

Previous failures should be in the dataset as well. If a bug once let a customer skip confirmation, then re-run that scenario after every change related to it. Data-driven testing allows you run the same address-change scenario against different customer phrases.

To test an attempt to change someone else’s order, you need a second test account. Keep test cases separate from examples used to tune the bot, or later scores may look better than they really are.

For a multi-turn test, the entire conversation is the case. Starting at “Actually, use my previous address” leaves out the address proposed just before it. Without that earlier turn, the bot has a different problem to solve.

How to Evaluate Chatbot Responses When Wording Changes

A chatbot may say “I've found order A123” one time and “Order A123 is here” another. Both are perfectly acceptable. A test looking for one specific sentence would fail the second response. The order number is a different matter: it must match exactly. A response referring to A124 would be incorrect, no matter how natural the rest of the response sounds.

Method Useful for Main limit
Exact or structured check Order IDs and saved field values Too strict for changing prose
Content check Required facts or claims the bot must avoid A keyword can appear in the wrong context
Meaning-based comparison Replies that say the same thing in different words Similar meaning does not prove facts are correct
Written scoring rules Helpfulness and factual support in a full reply Review criteria can be applied inconsistently
Human review Unclear responses or high-risk cases Takes time and needs agreed review standards

The team should write pass conditions before reading the replies. For the address test, the bot must request approval before announcing any change. A few examples of acceptable answers can guide the grader, but a person should still review a sample of the results. 

An LLM evaluator might approve a reply that wrongly promises next-day shipping. When that happens, the scoring rule needs to change rather than letting the mistake count as a pass. The team also needs to record each grader version, since a new scoring rule can change results even when the bot hasn't changed.

Repeated runs are worth looking at, particularly for requests which could result in a saved change. A bot that gets the address right 9 times and gets it wrong once is still a risk. Multiple runs under the same initial conditions can expose behavior a single run would miss. For A123, the confirmation wording may differ, but the permission check and the final saved address must be correct every time.

Security and Failure Scenarios for Chatbot Testing

A chatbot that can view orders is handling customer data, even if the chat itself appears harmless. Tests should try to open an order that belongs to a different customer and check that a new session doesn't expose the data from the previous account. Simply refusing the request in the response isn't sufficient if data has already been returned to the screen in the background.

Prompt injection poses another risk. A customer may type instructions to override the bot's rules, and an article the bot reads can contain similar instructions. OWASP calls these direct and indirect prompt injection. 

A controlled test can hide a harmless instruction in a help article, then check that the bot treats the text as source material rather than an instruction to act. The important result is the bot's behavior, not how confidently it answers.

The service the bot calls needs its own access checks. A chatbot shouldn't be able to edit any order just because it can call an update service. In the address example, the order service would need to verify the customer's identity before accepting any change.

Failures can occur without an attack. A save request may time out after the user confirms, leaving the bot uncertain whether the update actually occurred. The next reply must reflect the system's actual outcome rather than a guess. Passing these tests demonstrates how the application behaves under the circumstances being tested.

How to Automate Chatbot Testing and Regression

With regression testing, you can check chatbot features affected by a change without running every test. 

  1. Set the rules: Write the permitted customer action and the result the application must produce.
  2. Choose the cases: Include the normal conversation and the mistakes the bot must reject.
  3. Assign the checks: Decide which outcomes need application tests and which replies need separate scoring.
  4. Prepare the starting state: Use a fresh conversation with controlled customer records.
  5. Run the conversation: Replay every turn and inspect the application's action.
  6. Review the replies: Apply the scoring rules to generated answers, including refusals.
  7. Save failure details: Keep enough of the run to reproduce the problem.
  8. Compare releases: Check the new results against the previous version before approving changes.

The whole suite does not need to run after every edit. Changing a button in the chat widget mostly calls for browser checks. Updating the bot's instructions can change its replies without changing the button at all. The table shows which tests are worth running for each kind of change.

Change Checks to run
Small application change Related UI and service tests
Prompt or model change Affected conversation cases and reply scoring
Knowledge article update Retrieval results and claims in the final answer
Before release Important customer tasks and permission cases
Live operation Controlled monitoring of failures seen by customers

Live monitoring can catch problems missed in pre-release tests, but it finds them after someone may have been affected. LangSmith describes both offline testing and review of live conversations. The two serve different purposes, so a passing production score should not replace the cases a team runs before release.

A normal conversation test records the time until the first reply appears. A load test simulates many customers using the bot at the same time. You need to agree in advance what the traffic is for that load test, and the report should show the slowest responses, not just the average.

Chatbot Testing Tools and How to Choose an Approach

Testing a chat window is not the same job as judging a generated answer. Some products run customer actions through a browser, while others compare chatbot replies with written scoring rules. A bot that searches company documents may also need a separate check on the material it finds.

Tool or approach Main role Good fit What it cannot prove alone
Selenium, Playwright, or Cypress Browser testing Chat widgets and customer journeys Quality of open-ended answers
Cyara Botium Conversation testing Chat flows on supported channels Fit of every channel and connector without checking
Promptfoo LLM reply evaluation Repeatable answer checks and adversarial cases Correctness of a saved order change
LangSmith Reply evaluation and tracing Offline tests and reviewing live conversations Complete application behavior without other checks
Functionize Studio Application testing UI checks and business actions connected to the chat Dedicated model benchmarking or hallucination scoring

A website chatbot may never touch customer records. For that bot, the team could spend most of its time checking replies and the chat window. An order-support bot needs proof that it respects account permissions, even when the conversation sounds normal. If customers use several channels, the test must run through each supported channel. 

The address change case is a fair comparison: each candidate gets the same request, and the team looks at what each one can prove about A123. A tool that reports a passed conversation without actually checking the stored address has missed half the job. The setup work and time spent investigating failures should factor into the cost comparison.

Metrics That Show Chatbot Testing Is Working

A report claiming “92% chatbot accuracy” tells a team little on its own. That number might reflect good replies, while the application still fails to complete some customer tasks. The two results need separate measures, and each measure needs a clear set of cases.

Metric What it measures
Verified task completion Portion of test cases that finish with the required application result
Reply pass rate Portion of scored replies that meet the agreed rules
Unauthorized-action failures Number and seriousness of actions that broke permission rules
Handoff success Portion of eligible conversations routed correctly
Response time Time between agreed start and end points, including slower runs
Consistency across runs How often repeated cases reach the same required result
Escaped defects Production failures linked to cases the suite missed

For the address change bot, one rogue update is worth flagging even if every other response passes the language check. The average masks the failure with a high overall score. You can look at customer engagement separately, of course, but a long conversation doesn't mean the bot handled the customer's request correctly.

Testing Chatbot Application Workflows With Functionize Studio

Functionize Studio tests the app side of the chatbot (the chat window and the records behind it). Studio doesn't do any model benchmarking or score hallucinations and retrieval quality. Specialist evaluation tools cover those, and teams can use those alongside Studio.

The demo starts with an address-change test in plain English. It names the signed-in customer and the address that A123 should end up with. Studio builds the steps from that description, and the tester reviews them before the first run. The run then opens the chat and plays each turn (including the correction).

The test only passes in two circumstances. Firstly, nothing is saved until the customer confirms. Second, once the customer confirms, the order record shows the correct address. Studio can verify that record directly,because its tests can call APIs and query databases and databases themselves rather than just the screen.

A second run tests cancellation. The customer says "never mind" at the confirmation stage. The test only passes if the original address remains in the record. If a bug saves the address, the run should fail, and the transcript should be displayed alongside with the record.

Studio runs the clicks and the checks, and people make the decisions. The team decides what the bot must do and what it must never do. You set the standard, and you make the release call.

The Agent Tasks page shows what else Studio can check, including APIs, email, and SMS. The Free plan is enough to build the address-change test on your own app. Sign-up is self-serve, so there's no need to talk to sales first.

How to Evaluate a Chatbot Testing Tool

A pilot should cover an approved change and a canceled request. The team should also attempt to change another customer's order and simulate a failed save. Before running the cases, reviewers need to agree on which results block a release. 

An unauthorized update is a serious failure even if the bot politely refuses afterward. An awkward but correct reply may call for human review instead. Each run should save enough evidence for reviewers to decide without repeating the conversation.

Area What to check during the pilot
Chat channels Supported website widgets and other customer channels
Test sessions How conversations start again with clean data
Customer permissions Proof that the backend rejects forbidden actions
Failure evidence Saved transcript beside the application result
Reply scoring Who maintains the rules used to grade answers
Version history App version and model settings tied to each result
Private app access How tests reach nonpublic systems
Release process How failures reach the team approving a release
Running cost Limits on runs and charges for automated reply reviews
Human review How uncertain results reach a person

Teams testing private applications can review Functionize's Enterprise options before setting up their pilot. The subscription price is only part of a pilot's cost. Every model-based evaluation may have a usage charge, while difficult failures take time to investigate. Recording that work during the trial gives the team a better picture of what regular testing will cost.

Frequently Asked Questions

How Do You Test a Chatbot?

Start with a customer request and the outcome the bot is expected to deliver. Run the conversation as a customer would and then check what actions (if any) have been taken as a result of the chat. If the bot is composing its own responses, then assess those against the written rules rather than against a single sentence.

What Should Chatbot Test Cases Include?

Each case needs the customer's goal and the complete conversation in order. It also needs the starting application state and the actions the bot must not take. Save the result and the software version so someone can reproduce a failure.

How Do You Test Multi-Turn Conversations?

Replay the conversation from the start rather than testing a later message on its own. Include a correction like “Actually, use my previous address” after the customer has already provided another address. The final action should follow the customer's most recent confirmed choice.

Should Chatbot Tests Use Exact Response Matching?

Exact matching works for fixed values (e.g., an order number). It's usually too rigid for replies where wording can vary without changing the meaning. Use written scoring rules for open-ended answers, but do maintain strict checks on the stored data.

How Is Application Testing Different From LLM Evaluation?

Application testing checks what the software actually did (e.g., did it save the correct address to A123). LLM evaluation looks at the response and its claims. You need both for an order support assistant, as a good message won't fix an incorrect change.

Can Chatbot Testing Be Fully Automated?

Many repeatable cases can run without someone watching each step. Human reviewers still need to handle unclear answers and judge serious failures that automated checks cannot settle. Exploratory sessions also find new situations that the existing test cases never covered.

About the author

author photo: Tamas Cser

Tamas Cser

Founder & CEO

Tamas Cser is the founder, CEO, and Chief Evangelist at Functionize, the leading provider of AI-powered test automation. With over 15 years in the software industry, he launched Functionize after experiencing the painstaking bottlenecks with software testing at his previous consulting company. Tamas is a former child violin prodigy turned AI-powered software testing guru. He grew up under a communist regime in Hungary, and after studying the violin at the University for Music and Performing Arts in Vienna, toured the world playing violin. He was bitten by the tech bug and decided to shift his talents to coding, eventually starting a consulting company before Functionize. Tamas and his family live in the San Francisco Bay Area.

Author linkedin profile