AI Unit Testing: Examples, Tools, and Test Generation

AI can generate unit tests in seconds, saving developers the time it takes to write them manually. However, a test that passes doesn't always mean the code works correctly. Sometimes, AI generates tests based on existing code without knowing what the code is supposed to do. In this article, you'll learn how AI unit testing works and how to generate tests using AI tools. We'll use a Python example to see how AI-generated tests can identify bugs. You'll also learn how to review these tests and choose the right tools for your project.

What Is AI Unit Testing?

AI unit testing is the use of artificial intelligence to generate tests for individual parts of an application. These tests check whether a function or class works as expected without running the entire application. For example, imagine you have a function that calculates the total price of an order. You can write a unit test to check whether the function returns the correct amount for different inputs. Instead of writing every test manually, you can ask an AI tool to generate them.

AI can examine the function and suggest inputs that need to be tested. It can also write the test code using frameworks such as pytest, JUnit, or xUnit. However, the developer still needs to confirm that the expected results are correct.

Another thing to understand is that AI doesn't have to run every time a generated test is executed. Once the test has been created, the existing testing framework handles its execution. AI mainly helps with writing and improving the tests. Conventional unit tests already run automatically once they’re written. The difference with AI-assisted unit testing is how the tests are created, not how they run.

The term “AI unit testing” is sometimes used differently, to mean testing software that contains AI, such as checking a machine learning model’s accuracy, bias, or responses. That’s a separate subject. This article focuses on using AI to help write unit tests for application code.

How Does AI Unit Test Generation Work?

AI unit test generation starts with understanding the code that needs to be tested. A tool can examine a function to identify its inputs and possible outputs. It then uses that information to suggest test cases.

However, source code alone may not provide enough information. The code could contain a mistake, and AI might generate tests that accept the mistake as correct behavior. This is why requirements are important when generating unit tests. They tell the tool what the function is supposed to do rather than what it currently does.

Here are the main parts of the AI unit test generation process:

Tools such as GitHub Copilot can generate tests inside a coding environment. Other tools, such as Diffblue Cover, focus on generating unit tests for Java applications.

Some tools can also execute the generated tests and suggest fixes when they fail. However, those fixes need to be reviewed because the tool might change a correct assertion instead of fixing the problem in the application code.

AI Unit Testing Example: Catching an Incorrect Boundary Condition

Let's use a simple example to understand how AI-generated unit tests work. Imagine you're developing an appointment-booking application. The application allows customers to cancel appointments, but they may have to pay a cancellation fee depending on when they cancel.

The cancellation policy works like this:

• Customers who cancel at least 24 hours before an appointment pay no fee.
• Customers who cancel less than 24 hours before an appointment pay 2,000 cents.
• Negative hours are invalid and must return an error.

For this example, we'll use whole hours because the booking service handles time calculations and rounding.

Here are the expected results for different cancellation times:

Now that we know how the function is supposed to work, let's look at the code.

Step 1: Create a Function With a Bug

We'll start by creating a Python function called cancellation_fee_cents.

The function accepts the number of hours before an appointment and returns the cancellation fee. It also raises an error if the number of hours is negative.

Here's the code for fees.py:

FREE_CANCELLATION_HOURS = 24
LATE_CANCELLATION_FEE_CENTS = 2000

def cancellation_fee_cents(hours_before: int) -> int:
    if hours_before < 0:
        raise ValueError("hours_before cannot be negative")

    if hours_before > FREE_CANCELLATION_HOURS:
        return 0

    return LATE_CANCELLATION_FEE_CENTS

At first glance, the function might look correct. It returns zero when customers cancel early and charges a fee when they cancel late. For this example, we’ve added this bug on purpose so we can check whether the generated tests catch it.

However, there's a mistake in the second if statement:

if hours_before > FREE_CANCELLATION_HOURS:
    return 0

The function only allows free cancellation when the number of hours is greater than 24. According to our policy, customers should also be allowed to cancel for free at exactly 24 hours.

So, if a customer cancels 24 hours before an appointment, the function incorrectly returns 2,000 instead of zero.

An AI tool that only examines the function might assume this behavior is correct. We need tests based on the cancellation policy to identify the mistake.

Step 2: Generate Unit Tests From the Requirements

Now we'll create unit tests using pytest. Instead of writing a separate function for every cancellation time, we can use pytest parameterization This allows us to test multiple inputs using the same test function.

Here's what the test file, test_fees.py, looks like:

import pytest
from fees import cancellation_fee_cents

@pytest.mark.parametrize(
    ("hours_before", "expected_cents"),
    [
        (48, 0),
        (25, 0),
        (24, 0),
        (23, 2000),
        (0, 2000),
    ],
)
def test_cancellation_fee_follows_policy(
    hours_before, expected_cents
):
    assert cancellation_fee_cents(hours_before) == expected_cents

In the code above, @pytest.mark.parametrize provides five different inputs to the test function. Each input has an expected result based on the cancellation policy.

The assert statement compares the result returned by the function with the expected amount. If the two values are different, the test fails. We also need to check what happens when someone provides a negative number.

def test_negative_hours_are_rejected():
    with pytest.raises(ValueError):
        cancellation_fee_cents(-1)

Here, pytest. raises (ValueError) checks whether the function rejects negative hours. raises(ValueError) checks whether the function rejects negative hours. If the function doesn't raise the expected error, the test fails.

These examples use black-box testing because the expected results come from the cancellation policy. They also use data-driven testing to check the same function with different inputs.

Step 3: Run the Tests

Now that we have the test file, we can run it using pytest. Open the terminal and enter the following command:

python -m pytest -q

With the faulty function, one test will fail while the other five will pass. Here's an illustrative version of the output:

..F...

FAILED test_fees.py::test_cancellation_fee_follows_policy[24-0]
assert 2000 == 0

1 failed, 5 passed

The failed test tells us that the function returned 2,000 when the expected result was zero. This happens when the customer cancels exactly 24 hours before the appointment.

The other tests pass because the function handles their inputs correctly. Without the test for exactly 24 hours, we might not have noticed the mistake.

Step 4: Fix the Bug

To fix the problem, we need to change the comparison operator in the function. Here's the corrected statement:

if hours_before >= FREE_CANCELLATION_HOURS:
    return 0

We've replaced > with >=, which means greater than or equal to. The function will now return zero when a customer cancels 24 hours or more before the appointment. If we run the same tests again, all six will pass.

Notice that we didn't change the expected results in our test file. We corrected the function because its behavior didn't match the cancellation policy. An AI tool might suggest changing the expected result from zero to 2,000. That would make the test pass, but the original bug would still exist.

How to Prompt AI to Generate Unit Tests

When generating unit tests with AI, the information you provide affects the tests you receive.

For example, asking AI to "write tests for this function" may produce tests based entirely on the existing code. If the function contains a bug, the generated tests might not identify it. A better approach is to include the requirements and explain which inputs need to be checked.

Here's an example prompt for our cancellation-fee function:

This prompt tells the AI tool what the function is supposed to do. It also gives clear instructions about what the tool can change. For more complex functions, you may need to provide additional information. This could include the Python version, existing test files, or details about external services.

For example, if the function makes an API request, the tool needs to know whether that request should be mocked during testing. You can learn more about selecting useful test inputs in this guide on test case design.

How to Review AI-Generated Unit Tests

AI-generated unit tests need to be reviewed before they become part of a project.

A test can execute successfully without checking anything important. For example, a test might only confirm that a function returns a value. It won't tell you whether the value is correct.

Here are some things to check when reviewing generated tests:

One common problem involves mocking. Mocking means replacing a dependency with a controlled version during testing. For example, if a function sends an email after creating an account, you can mock the email service to avoid sending real messages.

Python provides the unittest.mock module for this purpose. However, mocking the function you're trying to test defeats the purpose of the test. The test would only confirm that the mock returned its configured value.

Test names are also worth reviewing. For example, test_fee_at_24_hours_is_zero tells us exactly what behavior the test checks. A name such as test_fee_3 doesn't provide the same information.

Clear test names make it easier to understand failures when a project contains hundreds of tests.

How to Measure the Quality of AI-Generated Unit Tests

One way to evaluate unit tests is through code coverage. Code coverage measures how much of the application's code is executed during testing. For example, 80% line coverage means that tests executed 80% of the measured code lines.

However, high coverage doesn't automatically mean that the tests can identify bugs.

Remember our cancellation-fee example? A test could execute the faulty function and expect a 2,000-cent fee at exactly 24 hours. That test would pass, even though the function violates the cancellation policy.

Here are some common ways to evaluate unit test quality:

What Is Mutation Testing?

Mutation testing is a technique used to check whether existing tests can detect changes in the code. A mutation testing tool makes small changes to the application code and runs the tests again. If a test fails because of the change, the mutation is considered caught or "killed."

For example, imagine we've fixed the cancellation-fee function by changing > to >=.

A mutation testing tool could change the operator back to >. Our test for exactly 24 hours would fail, showing that the test can detect the mistake. Tools such as MutMut for Python and PIT for Java support mutation testing.

Not every mutation can be detected because some changes don't affect how the code behaves. Mutation testing can also take longer on large projects, so teams may run it on selected parts of their code.

AI Unit Testing Tools Compared

There are several tools that can help developers generate unit tests. Some work inside code editors, while others focus on specific programming languages. For example, GitHub Copilot can help generate tests for different programming languages. Diffblue Cover and Parasoft Jtest focus mainly on Java testing.

Here are some tools and their main uses:

Before choosing a tool, check whether it supports your programming language and testing framework. A tool designed for Java may not help you generate pytest tests for a Python project.

It's also worth considering how much work is needed after the tests are generated. A tool that produces many tests quickly may still require considerable time to review and maintain them.

Another difference is the type of testing each tool performs. Functionize Studio, for example, tests a running application rather than generating unit test files for a repository. We'll look at that difference later in the article.

Common Problems With AI-Generated Unit Tests

AI-generated unit tests can contain mistakes just like other generated code. Some problems are easy to identify because the tests fail immediately. Others are more difficult to notice because the tests pass without checking the correct behavior.

Here are some common problems and ways to address them:

There is also a situation where developers may want to test existing behavior without deciding whether that behavior is correct. These are known as characterization tests. They are often used when developers need to understand older code before changing it.

For example, a developer may generate tests for an existing payment system before replacing part of its code. The tests help identify whether the changes affect current behavior. However, they don't automatically prove that the old behavior meets the business requirements.

How to Add AI-Generated Unit Tests to CI/CD

Continuous integration and continuous delivery (CI/CD) allow teams to run automated checks when code changes. AI-generated unit tests can become part of this process just like manually written tests. Once developers review and approve the tests, they can add them to the project's existing test suite.

A practical approach is to generate tests in a separate branch. This allows developers to review the new test files before merging them into the main codebase. If the AI tool also changes production code, those changes need to be reviewed separately. The team can then run the tests using the project's usual test command.

For example, a Python project using pytest might run:

python -m pytest

The AI model doesn't need to generate the tests again every time the pipeline runs. The saved test files are executed by pytest. Teams can also use AI to generate new tests in a separate workflow and open a pull request for review.

Tests need to behave consistently across different environments. For example, a test that depends on the current time may work on one computer but fail on another. Controlling external dependencies helps keep the results consistent.

What Passing Unit Tests Cannot Check

Unit tests help developers confirm that individual functions behave correctly. However, they cannot guarantee that the entire application works as expected.

Let's return to the cancellation-fee example. We've corrected the function, and all six tests pass. The function now returns the correct fee for every input in our test table. But what happens if another part of the application supplies the wrong input?

Imagine a customer cancels an appointment exactly 24 hours before it starts. The booking service has a time-zone bug and calculates the remaining time as 23 hours. Our cancellation-fee function receives 23 and returns a 2,000-cent fee. The function works correctly for the input it received, but the customer is still charged incorrectly.

This is where other types of testing become important.

Another problem could occur when the application saves the cancellation fee.

For example, the booking page might display the correct amount, but the backend could save a fee calculated before the function was corrected. A unit test for the calculation wouldn't identify that problem.

An application-level regression test can check whether the complete cancellation process still works after code changes.

Checking the Cancellation Flow With Functionize Studio

Functionize Studio is an AI-powered testing platform that lets testers create and run web application tests using natural-language instructions. Unlike a unit test generator, Studio doesn't create pytest or JUnit files for individual functions. It tests how the running application behaves when someone uses it.

For example, we can use Studio to check whether the appointment-booking application charges the correct cancellation fee.

Testing a Free Cancellation

Imagine we have an appointment-booking website where customers can view and cancel their bookings. We want to check whether someone cancelling exactly 24 hours before an appointment is charged a fee.

A tester could give Studio the following instruction:

Studio can use the instructions to create and execute a browser test. The test would need to confirm that the cancellation page displays the correct fee. If the application provides access to saved cancellation records through an API or data connection, the team could also check the amount recorded by the backend.

For this example, we're describing a proposed test rather than results from an actual Studio run.

Finding an Error That Unit Tests Miss

Now imagine that the booking service contains the time-zone bug discussed earlier. The cancellation-fee function is still correct, so all its unit tests pass. However, the booking service calculates the wrong number of hours before the appointment.

When a customer tries to cancel at the 24-hour limit, the application displays a 2,000-cent fee. An application test can identify this problem by comparing the displayed fee with the expected result.

The tester can then investigate whether the issue comes from the time calculation or another part of the booking process. This is one reason application testing remains useful even when a project has good unit test coverage.

How Functionize Studio Works

Functionize Studio uses a prompt-based approach to application testing. Testers describe what they want to check, and Studio creates and runs the test in a browser. It can also help investigate failures and maintain tests when the application's interface changes.

The approach is based on a simple idea: You direct. Agents execute. For example, a tester can describe a login process without manually writing a browser automation script. Studio then performs the required actions and checks the result against the instructions.

However, the tester still needs to decide what the correct result is. The platform executes the checks, while the team remains responsible for deciding whether the application is ready for release.

You can learn more about its use in development workflows on the Functionize developer page. The Agent Tasks page also explains the different testing tasks Studio can handle.

Functionize offers a free plan for trying application tests.

How to Evaluate an AI Unit Testing Tool

Before choosing an AI unit testing tool, it's useful to test it with code from your own project. A simple function may be easy for almost any generator to test. More complex code can reveal problems with dependencies or project setup.

For example, you could choose three different parts of your application for evaluation:

• A simple function with clear input and output rules.
• A function that depends on an external service.
• An older function that has little existing test coverage.

You can also include a known bug to find out whether the generated tests detect it. After running the tool, review how many useful tests it produced. Check how much time developers spent correcting the output and whether the tests found the known problem.

The tool's access to your project also matters. Some tools need permission to read source files or execute commands. Teams need to understand those requirements before using them with sensitive repositories.

Application-testing tools need a different evaluation. For example, Functionize Studio can be assessed by checking whether it completes a browser flow and identifies an incorrect result. A unit test generator would instead be assessed based on the quality of the tests it creates for the codebase.

Frequently Asked Questions About AI Unit Testing

Can AI Generate Unit Tests?

Yes, AI coding assistants and dedicated test generators can create unit tests for supported programming languages. They can use existing code and project requirements to suggest test cases. Developers still need to check whether the generated tests expect the correct behavior.

Are AI-Generated Unit Tests Reliable?

AI-generated unit tests can be reliable when their assertions are correct and their dependencies are properly controlled. However, a passing test doesn't automatically prove that the code works as intended. The expected results need to match the application's requirements.

Do AI-Generated Unit Tests Need a Special Testing Framework?

No, AI-generated tests can run using existing frameworks such as pytest, JUnit, and xUnit. For example, Python tests generated by AI can be executed using pytest. The AI tool isn't required every time those tests run.

Does 100% Code Coverage Mean the Tests Are Good?

No, code coverage only shows which parts of the code were executed during testing. A test may execute every line of a function while checking an incorrect result. Test quality also depends on the assertions and cases included.

Can AI-Generated Unit Tests Replace Code Review?

No, developers still need to review generated assertions and any suggested changes to production code. AI may misunderstand a requirement or create a test that accepts incorrect behavior. Code review helps identify these problems before the tests become part of the project.

How Are Unit Tests Different From Application Tests?

Unit tests check individual functions or classes without running the entire application. Application tests check how the software behaves when someone uses it. For example, a unit test can verify a cancellation-fee calculation, while an application test checks whether the customer sees the correct fee during cancellation.