Automated regression testing allows you to verify that known bugs haven’t recurred in your software. It is an essential requirement for CI/CD and cuts the time needed for releases. Here, we see why it is so hard to get it right and show how Functionize helps.
What is regression testing?
Regression testing involves checking that known bugs have not recurred in your program. Bugs that come back are known as regressions, hence the term regression testing. Early on in the development of a project, few bugs will be known, and so the focus of all testing is on finding new bugs. But as your software nears release, you hope that most bugs have been found. So, the focus switches to regression testing.
Why is it essential?
Software is complex. Indeed, it is often described as the most complex man-made object in existence. On a typical project, you may have 3 or 4 teams of developers working in parallel. Initially, they focus on creating functionality and getting different modules to work together. But as your codebase matures, there will be more emphasis on refactorization. That is, rewriting the core code to make it more performant and efficient.
One of the results of this is that existing known bugs may recur. This is often because a bug fix was poorly documented and a developer accidentally removes an essential line of code. But it can also be because only a partial fix was made. In such cases, the bug recurs but often in a subtly different manner. The final type of regression happens when a fix was made to the calling code, not the code with the bug. Now, if a new piece of code calls the underlying buggy code, it will also trigger the regression. Regressions can be a real pain because often they involve code that was written many months earlier. As a result, the developers will take longer to fix these bugs.
How do you do regression testing?
The naive answer is to just rerun all your existing tests every time. However, this isn’t necessarily the best approach. Firstly, you probably have a lot of tests, and so it may simply not be feasible to run them all in the time that is available. Secondly, as your code evolves, many tests actually become obsolete because the underlying codebase has changed so much. Thirdly, newer tests are often supersets of existing tests. So, it is unnecessary to run the older test. Finally, if only part of the codebase has changed, it may only be necessary to run some of the tests. In other words, a regression test suite will evolve over time, and will often change for each release candidate.
What are the challenges?
One of the biggest challenges for any QA manager is deciding exactly which tests need to be run each time. In order to determine this, she will have to liaise closely with her colleagues in the dev team. Particularly the project managers and lead developers. They are the ones that know what code has changed recently. This means they often have insights into which tests are needed.
However, as you approach release, you need to ensure all relevant regression tests are run. This is a problem since there are likely to be a huge number of tests needed. On a large project, this will typically be many hundreds of tests. The issue is, running this many tests takes a long time and acts as a delay for release.
How automated regression testing helps
This is where automated testing comes in. Automated regression testing involves getting a computer to run as many of your tests as possible. This speeds things up significantly. For a start, a computer can run tests 24/7 without breaks. Even the most dedicated test engineer can’t achieve that! For another thing, it frees up your manual test team to concentrate on the more complex test cases.
Automated regression testing promises the ability to steadily process the entire test suite between releases. This means you are achieving better overall test coverage. It also means that in the run-up to a release, you don’t need to complete as many tests. In turn, it acts as an enabler for CI/CD.
How to automate regression testing
Test automation involves getting a computer to interact with your application as a user would. Typically, this means interacting with the UI. However, some applications may also be tested via a CLI or even an API. But UI testing is by far the most common, so that’s what we will focus on here. In order to test a UI, you need two things. Firstly, an automated way of selecting and interacting with the elements in the UI (e.g. buttons, text fields, etc.). Secondly, you need to be able to run through the series of steps that make up your test plan.
The classic UI testing methodology is to use Selenium scripts. Selenium has been evolving for the last 15 years and is often viewed as the de facto standard for UI testing. Selenium consists of several elements as shown below. The test script is translated into Selenese (the domain-specific language for Selenium). This is passed to the Selenium server which coordinates the tests and results. The server then uses Selenium Webdriver to interact with the real browser where the UI under test is running.
What are the problems with this?
There are three key problems with the Selenium approach for automated regression testing. Let’s look at each one in turn.
Creating a test script is a slow process. It takes many hours, even a skilled test automation engineer. This is because you need to meticulously test and debug each step, almost like a mini software project. Our researches suggest on average a simple script takes around 8 hours to create and adapt for cross-browser testing.
Selenium was created 15 years ago. As a result, it isn’t really optimized for running tests in parallel. Your only options are Selenium Grid or cloud-based Selenium services, such as SauceLabs. Selenium Grid is pretty clunky, and only scales to a few tens of instances running at once. And cloud-based services charge thousands of dollars per month without giving real scale.
The final bugbear for Selenium is the constant need for test maintenance. This is because it relies on element selectors that change each time the UI is updated. In turn, this means you will need to debug or even rewrite most of your scripts after every change. Clearly, this is not efficient, and it means you lose many of the benefits of automated regression testing.
Continuous integration and delivery is a popular approach for deploying web applications. The problem is, your codebase is never stable. As a result, you will be constantly having to perform test maintenance as well as creating tests for new features. The upshot is, Selenium-based automated regression testing acts to reduce your delivery velocity.
Why Functionize’s approach is better
Here at Functionize, we have concentrated our efforts on reducing the friction for test automation. We believe anyone should be able to create, execute, and analyze a test. And test maintenance should simply not exist. To achieve this, we have developed an intelligent test agent. This allows you to write test plans in plain English, which are then automatically converted to tests. Tests are executed at scale in our Test Cloud with no need for complex setup. Then the results are displayed to you graphically, with failures or unexpected elements highlighted in a series of screenshots.
We achieve this by combining multiple forms of AI using advanced techniques like boosting. Our Adaptive Language Processing™ engine takes your test plans, parses them, and uses the result to model how your UI works. It then creates tests that work cross-browser and cross-platform with no need for additional work. The tests run in our serverless Test Cloud, allowing you to run potentially thousands of tests in parallel. Our new runtime allows you to create complex tests that use dynamic data controlled by simple reusable apps. Best of all, our Adaptive Event Analysis™ engine slashes routine test maintenance by over 90%. It also streamlines the process of tracking down and fixing more complex test failures.
Applying this to automated regression testing
The upshot of all this is that Functionize offers the perfect solution for automated regression testing. ALP™ makes 100% test coverage feasible for any application. In turn, this means the regression suite can test all known bugs automatically and autonomously. Our serverless approach makes testing at scale extremely simple. In turn, this means you can run the entire test suite in hours not days. Finally, analyzing the results takes minutes, allowing you to quickly locate any regressions and get them fixed. The upshot of all this is that you are now able to run automated regression testing 24/7 with minimal input from your test team.