Top 10 tips for modern web app testing
There was a time when applications were created natively for every platform. Desktop apps were delivered monolithically. Mobile apps were native to each phone platform. Web apps were used for things like eCommerce. Nowadays, we see a very different landscape. Many apps are designed to work cross-platform. There is often no difference between a desktop office app and the web-based version. Sometimes the browser can even replace the OS completely.
This change puts a new emphasis on web UI testing. In this eBook, we will look at the top 10 scenarios for modern web app testing. We will explain what makes them hard, and show how an AI-powered test automation platform allows you to address all 10 scenarios. FInally, we will preview how AI testing will evolve over the coming months and years.
What are web apps?
Web application is a catch-all term for any application where the UI is accessed via a web browser. In many cases, there are also native versions of these apps that run directly in the OS. Increasingly, these native apps have been created by using an approach known as progressive web apps. This allows the app to perform the same on whichever platform it runs. For the purposes of this eBook, we are going to focus on apps that are accessed directly via the browser. However, much of what we will talk about also relates to native apps and progressive web apps.
How do you test web apps?
Modern web apps are often at least as complex as the monolithic desktop apps of old. Indeed, in some ways, they can be even more complex. That’s because they often rely on integrating services from third parties. For instance, it is common for eCommerce sites to use a third party to provide their payment system. Or for business apps to integrate with GSuite to provide access to Google Drive, etc. So, how do you go about testing these large, complex apps? Where possible, you want to use test automation for this.
Test automation is the process of interacting with the web app via its browser-based UI. The computer locates and selects elements in the UI, and then interacts with them. This interaction includes any action a user could do, including clicking, hovering, dragging, and entering text. Once the action is complete, the computer then checks to see if the UI behaved as expected. This sort of testing closely replicates how a human would undertake manual testing. And like the human, all the computer sees is what is actually on the UI.
How does traditional test automation work?
All of this might sound quite convoluted, and it is. Usually, the test script has to be created by hand, step-by-step. The test engineer has to keep checking that the script is behaving as expected when it interacts with the UI. Often, she will need to debug the script multiple times before it behaves reliably. Effectively, this process is replicating how you actually develop the application that is being tested. It means each test script can take hours or even days to create. The problem is, despite all this time and effort, there are lots of scenarios where the test script is going to struggle.
10 scenarios where traditional test automation struggles
Traditional test automation frameworks based on Selenium really struggle with some common test scenarios. Yet these scenarios are often the most vital ones for any modern web application. What follows is our top 10 list of modern web application test scenarios. For simplicity, we have divided them into 3 groups.
Testing on different platforms
Almost by definition, a modern web application has to work equally well on every browser and platform.
- Cross-browser testing. Chrome still dominates the browser market with around ⅔ market share. However, the overall market is now more fragmented than ever before. Safari, Edge, Firefox, and Opera all account for 3-10% of the market. But there are at least another dozen browsers that make up the remaining 5% of the market. Thus, as a minimum any modern web application needs to be tested on five different browsers. Then bear in mind that each of these browsers receives two or three updates per year. The upshot is you end up needing to test on potentially 50 different browsers.
That is a huge challenge for traditional test scripts. Every browser will handle selectors a little differently. Each browser will render the UI differently. Thus, your poor test engineer ends up having to test and debug her test script for every single browser. That dramatically increases the time needed to create a new test. And we didn’t even touch on the problems caused by responsive web design!
- Mobile testing. Of course, nowadays most web applications are accessed via a wide range of devices, not just a laptop or desktop. Almost everyone uses a smartphone. Many people have tablets. And some people will even access web applications through other devices, such as smart TVs. This means that as well as testing cross-browser, you also need to test cross-platform.
Generally, there are two approaches to cross-platform testing. The first is to do it manually, usually by contracting it to a specialist test company. That company will employ dozens of people furnished with hundreds of different devices. They will work through all your tests manually, often recording what they see on the screen so you can check the results. The other approach is to use device emulation to allow you to run your automated tests as if they were on the appropriate device. This requires you to be able to run your tests in the cloud. And typically, that means passing this task over to a company that has the suitable infrastructure set up.
Testing throughout the product lifecycle
As applications get more complex it becomes more essential to test them as early and as often as possible. This ensures that any bugs are found early, when they are still easy to track down and fix. This testing also needs to extend into production. A web application is a living thing, served by a backend that may run slow, go offline, or otherwise damage the user experience.
- Shift-left testing. This term harkens back to the traditional placement of software testing in the waterfall development model. Here, testing was the last thing to happen before release. Shift-left testing refers to testing that happens earlier in the development process. It has been shifted left in the development timeline. This sort of testing is ever more important. The earlier you can find a bug, the faster a developer can fix it. The worst thing is if the developer only learns about the bug weeks or months after writing the code. Then he has to try and remember exactly how the code works and work out what has gone wrong.
Shift-left testing requires you to be able to run your tests as soon as a new build has been created. The problem is, Selenium tests are really fragile. They break almost any time you make a change to the UI or site logic. As a result, the test team spends a good portion of their lives “maintaining” tests. Basically, that’s their euphemism for fixing bugs in their test scripts. That means they won’t be able to do a complete set of tests every time a new build is ready. If they tried, they’d never get round to completing any testing at all. Often, the test team spends more of their time fixing scripts than actually testing your application.
- Shift-right testing. As you might guess, shift-right means moving testing beyond release and into production. That makes a whole lot of sense for modern web applications. After all, unlike a desktop or mobile app, web apps are heavily reliant on the backend. It’s the backend that sends the UI to the browser, handles all API calls, etc. So, you need to make sure your backend is working as expected.
The traditional way to do shift-right testing is using performance monitoring to assess how well the backend is performing. You can then couple that with various test approaches, such as canary testing or dark launching. What you can’t do is test how the actual end user will see your application. For that, you need to run your automated tests but point them at your production system. That is perfectly doable, but it can easily tie up your (limited) test infrastructure, so it’s hard to do it at any meaningful scale. This means many test teams rely on manual testing in production. So, there’s a heavy dependence on performance monitoring tools or, worse yet, vocal customers to detect issues.
Many test scenarios depend on repeating the same basic test with multiple different data. Collectively, this is referred to as data-driven testing. There are lots of ways you can leverage this in your QA. Here are the three most useful scenarios.
- Geographic testing. Many applications use intelligence in their backend to serve the correct content depending on the client’s geographic location. More often than not this is defined using IP address lookup. A great example of this is needing to check which local laws apply. For instance, a visitor entering your site from the EU is protected by the GDPR (General Data Protection Regulation). That means they may need to agree to different terms and conditions regarding the processing of their personal data. You will also need to comply with the ePrivacy directive. This requires a cookie banner to allow the user to customize which cookies and trackers they will allow. In both cases, getting it wrong is bad. Sadly, doing this sort of testing in a traditional test framework is difficult. All IP addresses are internal, so you end up having to “fake” the location. Then, the test is run repeatedly with different apparent geographic locations set. Often, the test team has to request special functionality be added to the application to make this easier.
- Localization testing. Localization and the related internationalization allow you to present your application in the correct language and with appropriate settings. For example, ensuring the correct units are displayed for weights and measures. Changing the currency and how numbers are shown for product prices. It’s really important to get these things right. The best way to test this is to run the same basic test, but set the system up each time such that it defaults to different settings. You then check that the correct output is displayed. There are a number of approaches you might use. For instance, you could have a different test user for each region you are targeting. One of the things many applications do is associate localization settings with the user profile. Then, you just need to rerun the test multiple times, changing the test user each time.
The real challenge here comes from the need to manage potentially significant volumes of test data. That’s where TDM (test data management) comes to the fore. This allows you to connect complex data sources to your tests. So, you might have all your test conditions and outcomes stored in a large CSV file. Your test framework loads that file and each row defines a specific test. For more complex data, you might need to use an SQL database to drive your tests.
- Testing of complex logic. The third category of data-driven testing is becoming more and more significant as companies make web applications their primary interface. This in turn means there is a huge amount of logic running underneath the web application. To give an example, consider how an online automobile insurance quotation form works. Typically, you start by entering the details of your vehicle. This may be done using a series of drill-down menus, or it could be connected to a registration lookup system, such as the one run by DVLA in the UK. Next, you are asked for your personal details such as age and occupation. This may be a dumb form where you select from a list of jobs, or increasingly it might be driven by an intelligent bot. Finally, you enter some details about the sort of insurance you want, the excess you will pay, etc. The system is then able to give you a quotation.
But just think how hard all this is to test. Almost every entry you make in the form results in a new set of options being presented. Worse still, some choices will reveal or hide elements on the screen. So, you need to have a script that is able to cope with that. But you still need to test every possible combination. Often, this proves too complicated to automate and you end up with entirely manual testing. Alternatively, you need extremely advanced test scripts connected to the underlying database to check all the possible outcomes.
Most automated testing relies on rather simplistic verifications. Typically these fall into two groups: either you test for the presence of something on the screen, or you check that you have advanced to a new screen/site. However, real life is often much more complex. For instance, in registration flows, you might need to set up two factor authentication (2FA). Or you might want to check that the whole UI is correct. These complex verifications are much harder to implement in test automation.
The more modern approach to testing this is to rely on visual testing. That is, get the test script to check if the UI looks the same as it should. That usually means leveraging some form of computer vision. In the simplest case, the computer does a pixel-by-pixel comparison between the page being tested and the original example output. However, this is a nightmare when you are dealing with dynamic content, multiple browsers and responsive sites. A good example is embedded adverts being served by a third party which change every time and so need to be ignored. Or ignore a 1 pixel offset triggered by a browser calculating the CSS slightly differently.
- Attachment verifications. As already mentioned a few times, modern web applications are complex. Often, they output real documents. For instance, a ticket booking site might generate a PDF with the resulting ticket. Or an online bank might generate a downloadable Excel file to show your monthly statement. Effectively, these cannot be tested by a traditional test framework because they are totally invisible to the Web Driver. As a result, you end up having to test them manually, or use a hack such as using a script to check if a suitably-named file has been downloaded. For many companies that generate documents, such as tickets or statements, the output file carries significant value for customers. Often, this is the most important part of the whole customer experience. “Hacking” your way through file testing like this just isn’t going to cut it.
- 2FA testing. The final advanced scenario is actually one of the fundamental ones for testing modern login flows. Namely, testing two factor authentication. This security approach is becoming ever more prevalent. It involves either texting or emailing a security code to a user’s device which they have to enter in addition to their username and password. This follows the principle of “something you know, something you have, and something you are”.
The problem is, how do you test this flow automatically? Consider what that requires. Firstly, when you create the test user, you need to associate it with a real email address or working cell phone number. Then, each time the test user logs in, your test framework needs to somehow get ahold of the verification code. That means setting up some super complicated script that accesses the email or text, extracts the code, and shares it as a test variable. That’s a complicated flow to repeat over and over again.
How does AI testing help with these scenarios?
Artificial intelligence is rapidly becoming the new normal in the world of software services. Support requests are handled and filtered by AI bots. An AI helps decide whether you are a good credit risk. You even have AIs deciding what products to promote to you when you visit your favourite online retailer. AI, and particularly machine learning (ML), can also transform test automation. It can solve all the issues with the scenarios above and thus gives you a far more capable test setup.
Creating tests that work on any platform
Smart automated tests should work on any and every platform without the need for rewriting and debugging. This can be done by using machine learning to determine which elements to interact with on the UI. Functionize tests collect huge volumes of data each time they run. When you first create the test, this data is used to build a model of the whole site. The model creates a fingerprint for every element in the UI. This combines information about the CSS attributes, any API calls associated with the element, what the element actually displays, and even how it relates to other elements. When you run the test on a different platform or browser, this means it’s easy for the platform to find the same element again. Even better, if your UI is updated, the system still finds the correct element.
Powering shift-left and shift-right testing
One of the benefits of AI-powered testing is that it greatly simplifies test creation. Functionize Architect is a smart test recorder that allows you to build new tests simply by interacting with the UI like a normal user. But under the hood, the platform is actually building its detailed ML models of your system. This approach is great for shift left testing. It means you can generate tests for new features with almost no effort. Those tests will even cope with features that are still evolving, thanks to the intelligent selectors.
AI can also help with shift-right testing. The Functionize Test Cloud offers you almost unlimited VMs to run tests from. That makes it really easy to test against your production system at scale. Moreover, it allows you to run tests from multiple locations with real IP addresses. That gives you realistic testing of aspects like load balancing and responsiveness.
Intelligent data-driven testing
AI-powered testing really starts to come into its own for complex data-driven test scenarios. Firstly, ML needs as much data as possible. So, the system is built from the ground up to ingest and process data. Adding data sources for your specific test scenarios is just a matter of pointing the test platform at the right data source and off you go. Secondly, a by-product of the Functionize Test Cloud is that you can do much more realistic geographic and localization testing, simply by selecting the location to run your tests. Finally, AI makes it easy for the system to cope with complex site logic. That’s because the test isn’t trying to find elements defined by some specific selector. Instead, it is intelligently locating the element that matches the fingerprint it expects. That means it can cope with flows where every choice impacts which options are displayed in subsequent steps.
Of course, the above all require you to create and attach test data. But Architect also offers functions to generate its own test data and to capture the resulting data in test variables. A great example is testing a login flow. You can ask Architect to create a random email address for you and store this in a variable “registeredEmail”. You can then ask it to create a random password, providing a suitable template to ensure that it meets any minimum complexity requirements. This is also stored in a variable for later use. Then, when you test the login flow, you can ask Architect to input the data from these variables in the correct box.
Advanced verifications are a given with AI-powered testing. That’s because the system isn’t trying to just verify the presence or absence of an element. Instead, it uses AI to check things. Take visual verifications as an example. Here, the system is able to compare screenshots from the current test run against previous successful runs and the original run. Rather than just comparing pixels, Functionize allows you to specify the percentage of variation you allow. Additionally, the system looks at the underlying computed CSS values and sees how these vary between runs. When something seems to have changed more than expected, it highlights this on the screenshot and gives you a warning.
Another benefit of Architect is being able to test 2FA logins. You simply input the email address or phone number to send the message, and you can verify that it’s received so that you can progress the login flow. Last but not least, you can also perform verifications against Word documents, Excel files, or PDFs to check for specific elements or to check for data consistency. That allows you to create the rich test scenarios that modern web applications demand.
What’s next for AI-powered testing?
AI-powered testing is still relatively new and the technology is developing rapidly. We are expecting several new advances in the near future, which will further revolutionize test automation. Here’s three we’re particularly looking forward to.
Test gap analysis
One of the huge challenges with modern web applications is knowing exactly what you need to test. In the old days it was easy. You tried to test the entire application exhaustively. But as applications grow more complex, the number of test cases grows exponentially alongside. As a result, your test team is going to focus their attention on the test cases that seem most to reflect real usage of the application. The problem is, they can only define this list using their knowledge, experience, and in cooperation with the product team. In future, AI will allow you to record real app usage and automatically extract all the user flows from that. This will let you quickly see if there are any gaps in your testing.
User-generated test scenarios
Building on the above, an intelligent test platform should soon be able to actually define its own tests based on the identified user journeys. In effect, the test scenarios are being generated by the actual users of the application. That will be transformative for a lot of complex applications. Clearly, the test team will still need to create the core tests needed to release the app. But once released, they can remain confident that the testing will continue to keep up with how users are interacting with the app.
The ultimate AI-powered system will be capable of autonomous testing. This is the logical end-game for the approach above. Here, the platform will be presented with a new application. It will analyze it and compare it with its existing knowledge. From this, it will be able to create the test scenarios, build the tests, and run them. If the tests pass, it will give the green light to the CD system. If they fail, it will alert the CI system. Finally, once the app is launched, it will monitor how users are interacting with it and dynamically update the test cases as needed. All this without any direct human intervention. Sounds like SciFi? Well, our belief is we will see systems like this well before the end of the decade!