October 7, 2022

Automated testing

Automated testing has transformed modern software development. Arguably, is a major enabler for the explosion of apps over the past decade. Before test automation, all tests had to be done manually. This was slow, error-prone, and expensive. As a result, it often led to QA teams being seen as a blocker to releasing software
ebook
How to Future-proof Your Career with AI-Powered Testing
Download Now
Table of contents

What is automated testing?

1

Automated testing refers to any approach that makes it possible to run your tests without human intervention. Traditional testing has been done manually. A human follows a set of steps to check whether things are behaving as expected. By contrast, an automated test is created once and then can run any time you need it.

For a long time, developers have automated their unit testing. That is, the tests that check whether a given function is working properly. Then automated testing frameworks like Selenium were developed. These allow modules or entire applications to be tested automatically.

These frameworks allow a test script to interact with your UI, replicating the actions of a user. For instance, they allow you to find a specific button and click it. Or locate a text entry box and fill it out correctly. They also allow you to verify that the test has completed correctly.

As a result, automation testing reduces the time and effort needed for extensive and repetitive testing tasks.

What is automated testing

Why should I use automated testing?

Put simply, automating your testing allows you to do more testing, faster, and more efficiently. More specifically, it brings these benefits:

2
Improved operation scale of your team: with automated testing

Improved Scale: Automation testing transforms the scale at which your test team operates. This is because computers can run tests 24 hours a day, 7 days a week. Even your keenest quality engineers can only manage 60 hours a week! The result is, you get to run many more tests with the same resources. This is important since test engineers are a relatively scarce resource.

Speed of delivery: with automated testing

Speed of delivery: You are probably under constant pressure to release new features. However, releasing a buggy application can wipe out all those gains in minutes. Automated testing speeds up your regression testing significantly. In turn, this slashes the time between integrating a new feature and releasing it. At the end of the day, the faster you get new features out, the more competitive you will be.

Streamlined releases: with automated testing

Streamlined releases: The traditional approach to software development sees all testing done after the product is developed. However, with automation testing, you can continually re-test your application during development. Running smoke tests each time new code is pushed and testing stable features immediately streamline your entire release process, making it more efficient.

why should one use automated testing
Manual Testing
Automated Testing
Speed
Slow, requires manual effort and execution of test cases
Faster, allows for simultaneous execution of test cases using automation tools
Reliability
More prone to human error
Less prone to human error, as test cases are executed using automation tools and can be repeated consistently
Maintenance
Requires more effort to maintain large test suites
Requires more effort to set up initially, but requires minimal effort to maintain once automation scripts are created
Reusability
Tests can be reused, but require manual execution, limiting the scalability of testing
Tests can be reused multiple times with minimal effort, enabling more comprehensive and scalable testing
Scope
Limited in scope, requires manual effort and time, making it difficult to achieve full test coverage
Can cover a wider scope of testing, including regression testing, making it easier to achieve full test coverage
Cost
Lower upfront cost as it does not require specialized tools or technology, but can become more expensive over time due to increased labor costs
Higher upfront cost as it requires specialized tools and technology, but can become more cost-effective over time due to increased efficiency
Skillset
Requires a tester with manual testing skills, including an understanding of the application being tested and the ability to identify and report issues
Requires a tester with automation testing skills, including an understanding of programming languages and automation tools, as well as the ability to write and maintain automated test scripts

What kinds of tests should be automated?

There are a few factors to consider when deciding whether a test can and should be automated. Here are the most important ones:

3
Repeatable test should be automated first

Repeatable. The test must be one that can (and will) be repeated regularly. For instance, there is no point automating a test for a feature that is about to be deprecated.

Determinant test should be automated

Determinant. There has to be a clear right and wrong outcome for the test. In other words, it must be easy for a computer to decide whether the test failed or not.

Manual repetitive tests, labor intensive should be automated

Repetitive or tedious. As a rule, humans are very poor at repetitive tasks. Our minds wander or we get distracted. Automation testing is particularly advantageous for any test that involves repeatedly doing the same action, as it ensures consistent and accurate execution.

Business-Critical tests - to be automated

Business-Critical. If a test is absolutely critical, you should try your best to automate it and schedule it to run regularly. That way, you can be certain this test is always being carried out.

Manual testing

What kinds of tests are best done manually?

On the other side, there are a few types of tests that are really hard to automate. These are best done manually:

constantly changing tests

Constantly changing. Tests where the correct outcome changes frequently can’t be automated. Equally, tests where the outcome isn’t always clear.

One-off or ad-hoc tests

One-off or ad-hoc tests. Sometimes you need to test to check a particular condition, or to search for a reported bug. These ad-hoc tests aren’t suitable for automation. However, if you do find steps to recreate the bug, you may then want to automate the test.

Features that are still evolving

Features that are still evolving. As a new feature is developed, you need to develop your tests in parallel. It usually isn’t worth automating the test while the feature is still evolving constantly.

Types of Automation Testing

Automation testing can be particularly beneficial for repetitive tasks, ensuring consistent and reliable results. There are a wide range of types of tests that you can (and should) automate. The following list should help you decide.

4

Unit Testing

These tests are created and run by developers to ensure each function in your application is working correctly. They are easy to automate and should be run automatically whenever new code is pushed to your master branch.

Integration testing

Integration tests check whether modules/subunits of your application are working. These can often be tested automatically. You also need to ensure that you test both expected and unexpected inputs.

System testing

Once you have a complete application, you need to test the complete system. There are several types of system testing you can automate.

  • Functional testing. This is where you verify that your system functions in the expected manner. For instance, when a user logs in, does the system correctly load their account details.

  • Regression Testing. These are used to verify that new code hasn’t broken your existing application. They also allow you to check whether any old bugs have reappeared.

  • Smoke Testing. A smoke test is a quick set of tests that verify the core functionality of your application. It’s great to automate these. That way, you can instantly test whether any new code or change to your backend has major problems.

Pyramid of basic types of automated tests: Unit, Integration, System and Acceptance

User Acceptance testing

The “final” level of testing for any application is acceptance testing. This checks that the application does what the user wants it to. Some forms of acceptance testing can’t be automated, but others can be.

  • Performance tests. These make sure that your backend will be able to perform under the expected load from your users. This includes stress tests, load tests, and responsiveness tests. Each of these checks a different aspect of your backend.

  • A/B testing. This allows you to choose which version of a feature or UI element your users prefer or whether a new feature is popular. It can be automated relatively simply by using feature flags to turn features on and off. This is then coupled with instrumentation and analysis to understand how your users engage with the application.

Acceptance testing, UAT beer

End to End Testing

End-to-end testing evaluates a software product's entire workflow from the user's perspective, verifying the integration of all components. Automating these tests can be challenging due to their complexity, but modern, capable test automation tools can facilitate this process. An ideal test automation platform for efficient end-to-end testing should be robust against fragile tests, offer automatic test maintenance, enable straightforward test creation, and provide detailed, specific data.

End to end testing is still the king

Key automation testing concepts

Data-driven testing, TDM, DDT

Data-driven testing. Here, you want to repeat the same test multiple times with different data. For instance, you might want to test whether your shopping site correctly calculates shipping costs. You create one test, then run it with lots of different test accounts with different shipping addresses and account preferences. This is a great approach for internationalization testing.

black box testing

Black box testing vs white box testing. Black box testing means running a test without knowledge of how something has been implemented. This is important when you are doing system testing. By contrast, white box testing means you know exactly what is happening inside your system. This is better for unit and integration testing, where you want to ensure every bit of code is being tested.

UI vs API testing

UI testing vs API testing. As you would guess, UI testing involves interacting with the user interface to perform your tests. Most regression and smoke tests fall into this category. By contrast, API testing is explicitly calling API functions to ensure they return the correct results. This is also sometimes used during performance testing.

TDM and data-driven testing

How to automate your tests

There are four steps needed to create any automated test.

1Choose a suitable tool or framework for running the tests. This will depend on the type of tests you are automating. We discuss some options for system-level testing below.

2Precisely define your test case. This means writing down every single step and the required outcome. It is important not to make any assumptions and not to miss any steps that a manual tester might do without thinking about it. For instance, accepting a cookie banner popup.

3Convert your test case into a test that can run in your chosen framework. This can often involve writing a custom script. You need to check that your test actually works correctly and that it works in every case you need to test. For instance, if your app needs to run on different browsers, you need to make sure your test can work on every browser.

4Run the test and evaluate the outcome. This step can sometimes be hard. Often, test failures don’t show up immediately and it can take some detective work to find out what really went wrong. Also, many test failures are “false positives” triggered by some minor change in the application. In these cases you will need to update your test and run it again.

test automation flow diagram

Automating tests is a painstaking task. And you need to be certain not to cut any corners. Effectively, treat test automation as if it were a form of software development! Define the scope, design the tests, implement them, and then test and debug as needed.

5

How to choose a test automation tool

There are a number of criteria you need to take into account when choosing the perfect test automation tool. The following will allow you to make an informed choice.

6
Ease of use and test creation

Ease of use and test creation. Some tools have an incredibly steep learning curve. For instance, Selenium requires you to learn a new scripting language in order to create your test scripts. Other tools may cause problems when running tests, for instance requiring you to setup and maintain a large test infrastructure.

Support for multiple platforms/browsers

Support for multiple platforms/browsers. Cross-browser and cross-platform testing are vital for any systems or acceptance testing. You need to check how easy it is to run your tests on the different browsers and platforms your application supports. Often, this will need you to rewrite the test for each browser.

Ease of analysis

Ease of analysis. Not all tools are equal when it comes to test analysis. Some provide you with nice dashboards showing the results of your tests, which tests failed, how often they have run, etc. Others might require you to generate reports to access this information. There are also real differences in the sort of metrics that get collected in parallel. For instance, does a page suddenly take much longer to load than before?

Flexibility in Automated testing

Flexibility. The best frameworks and tools let you perform different types of testing. So, you may be able to run smoke tests, regression tests, API tests, and performance tests in one tool. However, beware the “jack of all trades, master of none” trap. No single framework is going to support every single type of testing we listed above. And if they claim to, look for where they are cutting corners.

Power features

Power features. The best test automation tools will give you a number of power-user features. These include things like data-driven testing, the ability to define custom metrics or test criteria, or being able to create orchestrations which join together separate tests.

How to choose a test automation tool

Of course, there are other criteria you should also take into account:

Manual testers are enabled to create scalable, durable test in minutes

Cost of the tool. Some tools are free, others you have to pay to use. But the true cost of your decision will depend on the factors above. For instance, a tool may be free, but if it takes weeks to develop each test, it is actually costing you in real terms.

Manual testers are enabled to create scalable, durable test in minutes

Fragility of the tests. Many test tools create tests that are brittle. Any minor change to your UI or app logic instantly breaks every test. This leads to wasting a huge amount of time on test maintenance.

Availability of support is critical for SaaS world

Availability of support. Clearly, most free tools are only supported by the community. By contrast, commercial tools should give you access to customer support, training materials, tutorials, etc. This could be critical if your tests are at all complex.

Test automation best practices

Once you have chosen your tool, you need to apply test automation in the correct way. The following best practices apply to system level testing. However, many of them can also be applied to other types of testing.

7

1Plan your testing carefully. Make sure your test cases are clearly defined and well written. Test cases should be self-contained and easy to understand.

2Test as early and as often as possible during development. The earlier you identify a bug, the easier it is to fix. Conversely, bugs found late on require much more effort to fix.

3Plan the order in which tests run. Often, you can use one test to create the state you need for a second test. For instance, creating a user before you then test the account page.

4If possible, use tools that can schedule testing automatically, either when new code is pushed or according to a schedule. This makes sure your code is always well tested.

5Set up alerts so you are told as soon as a test fails. Then decide if you should abort the current test run or let it complete. For instance, if it is a serious bug it is probably best to abort and fix the bug.

6Remember that you need to constantly reassess your test plans as your app develops and changes. There is no point testing legacy features that are no longer part of the applications.

Unit testing automation, self-healing

The best choice for automated testing

8

Here are a couple of choices of tools for different types of testing.

Unit Tests

Here, you need to understand that not all unit tests are equal. It is really easy to end up creating tests that always pass. Our advice is to couple a traditional unit test framework, like JUnit with a framework for performing mutation testing, such as PIT. Mutation testing will purposely change your function to allow you to check if your unit test picks up every type of problem. This means you can be confident your tests are far more robust.

Pitest - PIT - Unit testing
JUnit logo - unit testing

Functional UI testing

Here there are two choices, depending on whether you want a free tool or a commercial solution.

For a free solution, the classic approach is to use Selenium, which has been around since 2004. Selenium aims to replicate the actions of a real user interacting with the UI. It also allows you to write tests in many different languages. However, the tests are hard to create, need constant maintenance, and rely on having your own test infrastructure.

For a paid tool, you can’t go far wrong with Functionize. Here at Functionize, we have spent years developing smart test solutions. Our platform specifically addressed the shortfalls of Selenium:

  • Tests are easier to create, even for non-experts

  • Maintenance is reduced by our robust intelligent element selection

  • Tests always work on any browser or platform without needing rewriting

  • All the tests either run in the Functionize Test Cloud or in your own cloud instance

Overall, this delivers the power and flexibility of Selenium testing with none of the pain.

Functionize smart test automation lets anyone create end-to-end tests in minutes. Tests are powered by AI/ML to run on any browser and self heal as your site changes.

API testing

Again, with API testing, you need to choose between a free tool or one you pay for.

There are several free tools to choose from. But one of the cleanest is Postman, which also has a ton of other features for API development. This makes API testing really easy, regardless of your API architecture. It copes seamlessly with REST, SOAP, and GraphQL calls. It can also integrate with your CI/CD pipeline to ensure all your tests are run whenever new code is pushed.

Postman logo - API testing

For a paid solution, why not check out what Functionize can offer you. Just as with UI tests, we make it really easy to create and run API tests. One of the most powerful features we offer is the ability to store API responses in variables that you can then call in your tests. This is invaluable when, for instance, you need to test with an API key that needs to change each time.

Explore Product

About the author

author photo: Tamas Cser

Tamas Cser

FOUNDER & CTO

Tamas Cser is the founder, CTO, and Chief Evangelist at Functionize, the leading provider of AI-powered test automation. With over 15 years in the software industry, he launched Functionize after experiencing the painstaking bottlenecks with software testing at his previous consulting company. Tamas is a former child violin prodigy turned AI-powered software testing guru. He grew up under a communist regime in Hungary, and after studying the violin at the University for Music and Performing Arts in Vienna, toured the world playing violin. He was bitten by the tech bug and decided to shift his talents to coding, eventually starting a consulting company before Functionize. Tamas and his family live in the San Francisco Bay Area.

Author linkedin profile