Machine learning in software testing

Test creation

Automated testing requires you to create and run tests. Traditionally, this meant using a framework such as Selenium. Selenium replicates the actions of a real user interacting with the UI. It does this using a combination of element selectors and actions. Selectors allow it to identify the correct element on the screen. Then it can perform actions such as clicking, hovering, entering text, or just verifying that an element exists. You control all this by creating a detailed test script.

The problem with test scripts

Test scripts are remarkably hard to get right. Each script is a mini software development project. Creating a new script is iterative and slow, requiring frequent rounds of testing and debugging. Even a simple script can take hours to create. When you add in the requirement for cross-browser testing this grows to days. This is one reason why few companies manage to automate more than a fraction of their regression tests.

How AI improves the process

AI allows you to create tests by simply clicking your way through your test case step by step. Creating tests like this takes minutes rather than hours, since you’re essentially performing the test manually while the underlying system automates it. In the background, the system is building ML models of how your site works, recording huge amounts of data as it goes. It can learn what button is being pressed and why, using techniques like NLP to understand what the button does. Creating a model is important—it means the test will work on any browser or device. It will even cope with dynamic content and 3rd party widgets, such as PayPal “Buy now” buttons or HubSpot forms.

Test analysis

Test scripts are generally simple pass-fail affairs. The test works by checking whether the outcome of an action is what you expected. You have to tell it precisely what to look for in order for this to work. Superficially, this makes test analysis really easy—either the test passed or it failed. But the reality is much more complex.

Issues with interpreting test results

Ideally, your test should always be reliable. A test should only fail if there’s a bug or defect. Conversely, a test should only pass if everything is working as expected. However, there’s two problems with traditional test automation.

It only tests things that you explicitly tell it to.

This leaves large amounts of your UI untested, unless you spend significant effort creating your tests. As a result, you may find there’s a glaring mistake that none of your tests are picking up.

Many test failures are false positives

That means they aren’t really failures at all. This is because of how test scripts select elements on the page. These selectors are brittle and can change every time your site is updated or redesigned. 

Taken together, these mean you can’t be certain whether your tests really passed or actually failed.

How AI improves test analysis

AI can improve test analysis in three key ways. Firstly, you can leverage computer vision and machine learning to increase the amount of the UI you test. This visual testing approach allows you to compare how your site looks now against previous test runs. If anything unexpectedly changes, it can be flagged as a potential failure. Secondly, tests are more reliable. This is because false positives are far less likely because tests are robust to changes in the UI. Thirdly, the ML model constantly learns how your site should perform. For instance, it learns how long each page load should take. It uses this to create an intelligent wait before the first interaction with the page. So, if a page suddenly takes longer to load, it knows to report this as a test fail.

Test maintenance

Test maintenance is the bane of every test engineer’s life. Every time the UI changes or the site logic is updated all the tests break. This means you waste hours of time fixing and debugging your tests. This is euphemistically called test maintenance. But that hides the fact that it shouldn’t be necessary in the first place.

Why test maintenance exists

There’s one simple reason why your test scripts require maintenance—selectors. As we said above, selectors are used to tell Selenium which elements to interact with. The problem is, these selectors tend to change whenever your site changes. This happens regardless of how carefully you choose your selectors in the first place. And if the selector changes, either your script fails immediately, or it proceeds with the wrong element. Either way, the result is you get a test failure that needn’t have happened.

How AI removes the need for routine maintenance

AI can slash the need for test maintenance. This is because it relies on a complete machine learning model of your site. If an element changes or moves, it just works out what happened and still selects the correct one. This is known as Self Healing. For example, Functionize tests can cope if you move your “sign up” button to a different place on the page and change it to “register”. AI also allows you to implement intelligent test editing, for instance updating a test step directly from a screenshot by making use of all the ML data collected each test run. We call this feature Smart Screenshots and it transforms test editing and debugging.

Of course, sometimes a change may be more significant. Maybe your developers got rid of a button completely. Or changed every item in a drop-down menu. Not a problem if you are using an AI powered software testing platform. It will realise there’s a problem and come up with suggestions of what might have changed. These SmartFix suggestions are based on data gathered across millions of different tests. You simply need to click on the correct one and your test will be updated.