DevOps engineers have a hard life. Your task can be summed up as “make it all work”. That’s tough enough, but increasingly, you also have to sort out the myriad issues with automated testing. In this blog, we will see how a proper cloud testing strategy solves all those issues and gives you back your time.
Agile teams seldom get a chance to stop and catch a breath. This is especially true for DevOps engineers. Many people have tried to give definitions of the role of a DevOps engineer, but I like to sum it up as “make it all work.” Essentially, they are the grease that allows high-velocity CI/CD releases to happen smoothly while ensuring the end user still receives the best UX.
As you well know, it’s hard enough being a DevOps engineer at the best of times. But increasingly, you are expected to help sort out the complex issues faced by the automated testing team. These issues range from maintaining their test infrastructure to automating the deployment of new images. And chances are, you will find yourself being asked to help out with other scripting tasks. Many of these problems come about because test automation is still stuck over a decade in the past. And in our world, that counts as prehistoric. However, it doesn’t have to be like this. In the following blog, we will show you how a proper cloud testing strategy can solve these problems.
The problem with automated testing
Selenium revolutionized the world of testing when it was released over 15 years ago. Finally, it was possible to automate the repetitive UI regression testing that is essential prior to releasing any product. Selenium works by using a script to interact with the UI, replicating the actions of a user. The original version only worked on a limited range of browsers and as a standalone instance. Selenium Webdriver and related projects led to it working on the majority of browsers. Later, Selenium Grid allowed multiple instances to be managed from a single control node. The trouble is, Selenium wasn’t really designed to scale in this way. As a result, there are several problems that arise.
Testing at scale
Comprehensive regression testing of a modern web application might require 100s of different tests to be run on each of 20-30 browser/device combinations. Each of these tests takes several minutes to run. Clearly, running these one at a time is simply infeasible unless you want each release to take months. Selenium Grid (which is now part of the core product) aims to solve this. With Selenium Grid, a group of servers can be coordinated from a single control node. Each server can host several browser versions, and the control node tries to distribute the tests as needed. With careful setup and planning, a set of 20 servers should complete a full set of tests in under 1/20th the time taken for 1 server.
So, where’s the catch? Well, there are various problems. Let’s explore the three worst ones. Firstly, Selenium Grid isn’t really optimized for running at large scale. 10 servers are completely feasible, 100 starts to become hard, more than that and you are into the realms of the impossible. The only solution is to constantly repurpose servers during your testing. Secondly, you almost certainly will need to reconfigure servers dynamically in order to ensure none are standing idle. Not all tests take the same time to complete. This is especially true when a test fails. And as regular readers of our blogs know, test failures (and test maintenance) are the norm, not the exception! Thirdly, all these servers need to be kept constantly updated with the latest browser releases, security patches, etc. This is essential from both a security viewpoint and a test quality viewpoint.
Why is this a pain in the butt for DevOps?
Test automation engineers have to combine two skills. They need to be good at conducting thorough testing and they have to be able to write efficient test scripts. Finding people that are skilled in two mismatched disciplines is hard enough. Finding ones that combine these skills with DevOps and sysadmin skills is effectively impossible.
As a result, the DevOps team is usually called in to support the testing team. We often hear reports of testing teams needing dedicated support from DevOps engineers. And as you well know, DevOps engineers are a scarce commodity (even scarcer than developers in test!) The real problem is, as a good DevOps engineer, it is in your interest to ensure all testing has been done as efficiently and effectively as possible. You don’t want testing to become a roadblock to efficient releases, especially if you are running CI/CD. But even more, you don’t want to release software with bugs. So, the solution is to move to a cloud testing strategy.
Moving to the cloud
The first step in your cloud testing strategy is to move from dedicated hardware to using virtual infrastructure. There are a couple of ways to do this. You can set up your own test cloud using a public IaaS provider. Effectively, you are creating a whole new backend, just for managing your testing. So, this will save some DevOps time by removing the need for machine maintenance. However, you will still end up doing additional tasks to automate the setup, ensure security, etc.
Alternatively, you can use a managed service such as BrowserStack or Ranorex. There are a number of such companies that offer Selenium automation in the cloud. Often, they market themselves as cross-browser testing services. Companies such as these save even more DevOps time, but they cost thousands of bucks a month just for a few dedicated test instances.
Autonomous testing as a solution
Of course, as always there is another way to do things. Autonomous testing solutions like Functionize have revolutionized testing. We have taken the cloud testing strategy to its logical limits. As we put it in our Manifesto “Testing must be cloud-first”.
This means we don’t just move your test infrastructure to the cloud. Our solution also leverages the almost-infinite resources of the cloud to provide an intelligent test agent (ITA) that runs all your testing for you. Our ITA will help the test team at every stage in the process.
Our system leverages the power of natural language processing to allow test engineers to write tests in plain English. They can take a test plan produced by the test management system and Adaptive Language Processing™ converts it to a fully-functional test. Even better, they can take a user journey from the product team and do the same thing.
Test debugging and maintenance
Adaptive Event Analysis™ is the brains of our intelligent test agent. AEA™ combines multiple forms of artificial intelligence. This includes different machine learning algorithms, computer vision, NLP and our own autonomous template recognition. The resulting system is able to learn exactly how the system under test works. As a result, any test will automatically work on every browser and device. Furthermore, if the dev team make changes to the style or layout, AEA™ simply works out what changed and updates all the tests.
Running tests is all very well, but what about analyzing the results? This is where our visual testing strategy comes to the fore. Every time you run a test, our system will capture screenshots before, during and after each test step. It then uses intelligent visual analysis to highlight problems.
Employing A Test Cloud Strategy
All of the above rely on the Functionize Test Cloud, which is hosted on the Google Compute Engine. Our test cloud allows you to run thousands of tests in parallel. Each test runs in its own virtual server, meaning the load on your backend is more realistic. The test cloud can support any browser/device you need to test, including accurate emulation of mobile devices. It also stores the history of all your tests, and our detailed analytics allows you to compare the relative performance of every test step. This can often help identify backend issues before they become a problem.
So, how is this helping DevOps?
Put simply, by moving your company’s testing to Functionize, you no longer need to spend your time providing DevOps support to the test team. The Functionize Test Cloud means you no longer need to spend any time maintaining test infrastructure. Our load testing solution is uniquely accurate because it makes use of realistic user sessions from multiple discrete locations. Better still, Functionize tests can be easily incorporated into any CI/CD toolchain. This makes it easier to ensure smooth releases.
Test automation is essential nowadays. But the classic approach to test automation has a habit of eating DevOps resources. As we have seen, the solution is to adopt a cloud testing strategy. This can be as simple as moving your infrastructure to a public IaaS provider. Or you can use one of the many cloud-based Selenium services. However, as you saw, Functionize provides the ultimate way in which a cloud testing strategy can save DevOps time.