When developing any complex application, it’s essential to know that the system functions properly as a whole. Unit testing and integration testing can tell you that every part of the system works correctly, but that doesn’t guarantee that they all the parts fit together properly. Imagine if you are creating something mechanical such as a car drive-train. You can test that the engine, gearbox, and differential all work perfectly. But when you come to assemble them if you find that the prop shaft was built with metric couplings and the differential only accepts shafts with imperial couplings your car simply won’t work. This is where end-to-end testing comes in.
End-to-end testing is very different from most other forms of testing. For a start, end-to-end tests should concentrate on testing specific end-user functionality in a system. For instance, if you run an online gaming site, you might need to check that a user can register correctly. Or that they can deposit and withdraw funds correctly from their account. This means that you can’t start doing end-to-end testing until your system is actually functional. That isn’t saying your system has to be complete, but you do really need to have reached the MVP (minimum viable product) stage of development.
Techopedia defines end-to-end testing as:
a methodology used to test whether the flow of an application is performing as designed from start to finish. The purpose of carrying out end-to-end tests is to identify system dependencies and to ensure that the right information is passed between various system components and systems.
Breaking that down a bit, there are two key aspects to e2e testing. The first is to ensure that all system dependencies have been correctly met. This means that you have all the necessary moving parts in your system to achieve the required functionality. So, taking the online gaming example above, this would mean checking that you have all the components needed to create and manage user accounts, the APIs to handle logins, payments, etc. The second part is to test that the correct information is flowing between all these pieces of the system. This second aspect is the heart of end-to-end testing and is best done by using realistic test examples.
What end-to-end testing isn’t
It’s important to be clear that end-to-end testing is not the same as UI testing. While they may share elements in common, UI testing can fundamentally be treated as a form of unit or integration testing with each element of the UI being able to be tested separately. By contrast, e2e testing requires the entire flow to be tested. Also, UI testing is specifically concerned with whether the UI looks and behaves as expected, while end-to-end testing is concerned with whether the underlying system is working properly
However, often there will be significant overlap between UI and end-to-end testing. For instance, when testing a login flow for an app, you might simultaneously test that the login fields show the correct keyboard (e.g. email keyboard for the email field, hidden text for the password field) and that the login actually functions correctly. In other words, you can append UI tests to your end-to-end tests but only if you are very careful and are clear that you are doing this.
End-to-end tests are also really poor for testing specific functionality. This is because they break many of the cardinal rules of testing. They are usually quite slow to run because they require interaction with the whole system. They are certainly slow to get started, since they require the entire system to be built first, making it hard to follow the adage “fail fast”. They are also unable to isolate a bug – generally, all you will know is that something didn’t work. This means that if a bug is found you then need to invest further time and effort into isolating it. Finally, end-to-end tests are often quite unreliable since they can be affected by all the real-world issues that affect your final system including connectivity issues.
Planning your e2e tests
By their very nature, e2e tests are expensive to create and run, but end-to-end tests are extremely good at identifying the issues that will impact on real users. Consequently, it’s important to get the right balance. In the Google testing blog, they suggest that you should only spend around 10% of your testing time on e2e tests (versus perhaps 70% on unit testing and 20% on integration testing). Given this, it is important to make sure you select the most useful end-to-end tests to perform. This will need careful thought and planning.
The first step in planning your end-to-end tests is to identify the critical user flows that need to be tested. One approach here is to use the product team’s user stories to try and identify the actual key user flows. Work out where there is overlap between these. Then look at whether flows can be combined together to test more elements of the system with minimal extra steps. For instance, if you decide you need to test the login flow, you might also add in a step to test the forgotten password flow.
This will give you a set of possible test flows. However, there’s a good chance that there will still be too many flows to test them all efficiently. At this point, you need to look at how these flows interact with the underlying system. This will need you to look at the code and to talk to the development team(s) (or at least to the team leads). Your aim here is to come up with the set of flows that achieve the greatest coverage in terms of testing the overall system while minimizing the actual number of test steps needed.
Having identified the necessary set of tests, you then need to think about what you will need to set up in order to be able to run the tests. Often you will need to create test users which will need to be carefully designed to make sure they are suitable. You will also need to test your system in a realistic fashion. So, if your system includes payment handling, then really you need to test this with real-world payments, or at the least by using the test flows from your payments provider (not just connecting up a fake payments API that will always give the expected response). If you rely on orders being passed to a warehouse, you need tests that verify the order is correctly sent. If you run an e-commerce site, you need to make sure you have suitable (known) test products in the system – there’s no point having a test that requires purchasing a product if that product turns out either to not exist, or to be out of stock!
Automating end-to-end tests
As discussed above, e2e tests differ from UI tests. However, often you can use similar test automation tools to make them easier to conduct. This is especially true where your system relies on a user-facing front-end to interact with the backend. Before intelligent testing solutions like Functionize, tools such as Selenium IDE were effective in this use case because they enabled you to record your (planned) test cases much more quickly. Furthermore, because end-to-end testing is concerned with the underlying system and not the UI, it’s not necessary to test across every possible combination of browser and OS. So instead, you can use something like the Functionize Test Cloud to parallelize your end-to-end tests and speed them up significantly. Alternatively, you can use tools such as CasperJS, Nightwatch, Protractor or Testcafe to create your test scripts.
Many of you will be using continuous integration, continuous delivery (CI/CD) for your development. Integrating end-to-end testing here needs careful thought. If you are using an automation framework then you will have the choice between running the end-to-end tests after every new feature is pushed or running them on a periodical basis (e.g. overnight, or over the weekend). The choice you make will depend to some extent on the frequency (and scale) of code changes as well as the resources you have available.
As we have seen, end-to-end testing is a critical part of any testing strategy. For most projects, you should only expend about 10% of your testing budget on end-to-end testing. Consequently, you need to be careful to make sure you are using your resources wisely. Test automation can significantly improve the efficiency of end-to-end testing, but you need to be sure you are using it to test what actually needs to be tested.
The first time you run the end-to-end tests for your project can be both scary and rewarding. Scary because you have to hope everything does work, rewarding because this is usually the first time you see all the pieces coming together properly. For more mature projects, your end-to-end tests may form the basis for regression testing of the user-flows. Either way, there can be something very satisfying about seeing a complete system working as it should do.