Automated regression testing: what it is and how AI helps

Automated regression testing checks that your software still works as expected after you push new code. This is a central requirement for CI/CD and automating it dramatically speeds up your release velocity. However, getting it right is hard, unless you turn to AI-powered testing tools.

What is regression testing?

Regression testing involves checking that new code hasn’t broken your release. You do this by running sets of tests that are designed to test every element of the software. This includes testing whether old bugs have recurred as well as testing all the known happy and sad paths in your system.

Early on in the development of a project, few bugs will be known, and the focus of testing is on finding bugs. But as your software nears release, you hope that most bugs have been found. So, the focus switches to regression testing. The aim now is stability and reliability.

Why is it essential?

Software is complex. Indeed, it is often described as the most complex man-made object in existence. On a typical project, you may have 3 or 4 teams of developers working in parallel. Initially, they focus on creating functionality and getting different modules to work together. But as your codebase matures, there will be more emphasis on refactorization. That is, rewriting the core code to make it more performant and efficient.

One of the results of this is that existing bugs may recur. This is often because a bug fix was poorly documented and a developer accidentally removes an essential line of code. But it can also be because only a partial fix was made. In such cases, the bug recurs but often in a subtly different manner. Other times, the fix may have actually been more of a band-aid that solved a bug’s effect, not the cause. So, a new piece of code may call the underlying buggy code and trigger a seemingly new bug. Finally, some bugs are latent. They exist in the code, but only get triggered when some new code is added.  

Regressions can be a real pain because often they involve code that was written many months earlier. As a result, the developers will take longer to fix these bugs. This is one of the strongest arguments to move to CI/CD, with your entire application being checked whenever new code is released. That way, any bug should be found when the code is still fresh in the developer’s mind.

How do you do regression testing?

The naive approach is to just rerun all your existing tests every time. However, this isn’t necessarily the best approach. Firstly, you probably have a lot of tests, and so it may be infeasible to run them all in the time that is available. Secondly, as your code evolves, many tests actually become obsolete because the underlying codebase has changed so much. Thirdly, newer tests are often supersets of existing tests. So, it is unnecessary to run the older test. Finally, if only part of the codebase has changed, it may only be necessary to run some of the tests. In short, a good regression test suite will evolve over time, and will often change for each release candidate.

What are the challenges?

One of the biggest challenges for any QA manager is deciding exactly which tests need to be run each time. In order to determine this, she will have to liaise closely with her colleagues in the dev team. Particularly, the project managers and lead developers. They are the ones that know what code has changed recently. This means they have insights into which tests are needed.

However, as you approach release, you need to ensure all necessary regression tests are run. This is a problem, since there are likely to be a huge number of tests needed. On a large project, you may easily have thousands of tests in your regression suite. The issue is, running this many tests takes a long time and acts as a delay for release.

How automated regression testing helps

This is where automated testing comes in. Automated regression testing involves getting a computer to run as many of your tests as possible. This speeds things up significantly. For a start, a computer can run tests 24/7 without breaks. Even the most dedicated test engineer can’t achieve that! For another thing, it frees up your manual test team to concentrate on the more complex test cases.

Automated regression testing allows you to steadily process the entire regression suite between releases. This means you are achieving better overall test coverage. It also means that in the run-up to a release, you don’t need to complete as many tests. In turn, this is a key enabler for CI/CD.

How to automate regression testing

Test automation involves getting a computer to interact with your application as a user would. Typically, this means interacting with the UI. However, some applications may also be tested via a CLI or even an API. But UI testing is by far the most common, so that’s what we will focus on here. In order to test a UI, you need two things. Firstly, an automated way of selecting and interacting with the elements in the UI (e.g. buttons, text fields, etc.). Secondly, the ability to run through the steps that make up your test plan.

Classically, this means using Selenium, which has been around since 2004 and is often viewed as the de facto standard for UI testing. Selenium interacts with your application via its Web Driver. You tell this what to do using a script which can be written in almost any language you like. The script tells the Web Driver what to select on the screen and what to do with it. For instance, find the login button and click on it. You then get the Web Driver to check that the action achieved the expected outcome (opening the login page).  

What are the problems with this?

There are three key problems with the Selenium approach for automated regression testing.

Test creation

Creating a test script is a slow process. It takes many hours, even for a skilled test automation engineer. This is because you need to meticulously test and debug each step, like a mini software project. On average, even a simple script takes around a day to create and adapt for cross-browser testing.

Test execution

Selenium isn’t optimized for running tests in parallel. Your only options are Selenium Grid or cloud-based Selenium services. But both of these only scales to a few tens of instances running at once. So, you are always limited in how many tests you can get through each day. As a result, QA managers often have to make difficult decisions, such as only completing smoke tests for some browsers, rather than full regression testing.

Test maintenance

The last and greatest problem for Selenium is the constant need for test maintenance. This is because it relies on static selectors that change each time the UI is updated. In turn, this means you need to debug or even rewrite most of your scripts after every release. Clearly, this is not efficient, and it means you lose many of the benefits of automated regression testing. In fact, test engineers often spend over half their time on this.

Why Functionize’s approach is better

Here at Functionize, we are focused on using AI to remove the barriers for test automation. Our AI-powered test solution makes it easier to create tests, allows you to run thousands of tests at once, and dramatically reduces test maintenance.

We achieve this by combining multiple forms of AI. This includes deep learning, computer vision, machine learning and even natural language processing. Tests are created in Architect, our intelligent test recorder. This allows anyone to rapidly create robust tests. These tests automatically work on different browsers without debugging, even mobile browsers. You even get advanced capabilities, such as test data management, test variables, and the ability to test end-to-end flows. 

The tests run in our Test Cloud, allowing you to run thousands of tests in parallel. This in turn powers our self healing capabilities, thanks to the huge volume of data it collects during every test run. As a result, our tests are 5x as reliable as Selenium test scripts. This means your team can focus on what matters the most—ensuring your software is as reliable as possible.  

AI-powered automated regression testing

The upshot of all this is that Functionize offers the perfect solution for automated regression testing. Thanks to Architect, your team can automate many more tests than with Selenium. The tests are far more reliable, so you won’t sink under the burden of growing test debt. You can run more tests faster, meaning you slash the time to sign off each build. Finally, Functionize integrates seamlessly with all your existing tools, like Jira and TestRail. If you want to find out more, simply book a demo with one of our team. Alternatively, check out our video to learn more about what we do.