As every developer and manager knows, the cloud transformed how software was delivered. But over a decade after the beta launch of Amazon’s Elastic Cloud Compute (EC2) service, automated testing often still uses dedicated hardware. Read on to learn how Functionize is bringing the benefits of the cloud to testing, slashing test infrastructure costs and hugely accelerating software delivery.
Selenium transformed the world of testing about 15 years ago. This free software finally enabled the dream of many testers – truly automated testing of UIs. While Selenium itself is free, to use it effectively costs money. On top of the costs involved in creating working Selenium scripts, you need infrastructure, support software, sysadmin time, and cross-browser test tools. In the following blog, we will look at the true costs of using this free software, explore why cloud-based autonomous testing is better, and explain how the Functionize Test Cloud saves you money.
The traditional approach to test automation relied on free software, such as Selenium. Managers might have been misled by the “free” aspect, but this approach was far from free to implement. So, let’s look at where some of the real costs came from.
Overall then, while the software might be free, there are some eye-watering costs associated with using it!
Anyone who has looked into buying a server will know they are not cheap. In large part, this is because servers use extremely high-grade components. After all, servers have to be able to run 24/7, often at full load for much of the time. As a comparison, an 8 core Intel Xeon E5-2687W v2 Ivy Bridge-EP running at 3.4 GHz comes in at about $2,100. By contrast, a higher specification desktop CPU, such as an Intel Core i7 with 8 cores running at 4.9GHz, costs $400.
But the CPU is just the tip of the iceberg. Servers need server-grade RAM which, you guessed it, is expensive! Really, you should be looking to store as much test data as possible. Storage is relatively cheap, but you’re going to need a lot of it, probably installed as a RAID array for reliability. Even mundane things like the case are generally way more expensive than your home computer. And if you are being at all serious about things, you will probably need to invest in a proper rack and the associated cooling, power distribution, and KVM. All in, you will be looking at upwards of $50k just on hardware (and that isn’t even going overboard on specifications). So imagine what a complete set of test infrastructure costs.
Selenium itself may be free. But what about all the other software and processes you need to develop to create a robust automated testing platform? Developing and maintaining these systems is a significant extra hidden cost. This is over and above the costs involved in just creating test scripts. If you are going to scale at all, you need to be able to manage your tests, have processes for logging and reporting test passes and fails, and a database to store past test results. One of the most valuable things for a test engineer is being able to answer the question: “Is this the first time this test failed like this?”
To create a robust test setup, you need to employ developers to support your test team. You also get into the catch 22 of having to do QA testing on your testing software. Alternatively, you can pay for off-the-shelf software that achieves what is needed. Either way, you are having to pay to be able to use your free software. Then comes the management costs of running all this software stack. You already needed to manage sysadmins for the hardware and developers in test to create the scripts. But you also need to manage the whole process of sharing developer resources.
Any modern web application needs to function equally well in any browser and on any device. Typically, you need to be able to run on round 10-15 different combinations. And if you are being exhaustive, there are many more. Cross browser testing like this has two big hidden costs. Firstly, all Selenium scripts need to be modified for the specific browser/device combination. Some may even need to be rewritten. Secondly, the costs of setting up and managing all the different combinations are high. Especially since many browsers release several major updates per year.
One obvious way to save some money when using Selenium is to move all your testing infrastructure into the cloud. This brings some key benefits.
However, there are still issues relating to cross-platform testing, since most cloud operators don’t support that. This is one reason so many people find themselves turning to 3rd parties like SauceLabs and BrowserStack. SauceLabs offer plans with 2 concurrent test sessions for $399 a month, rising to an eye-watering $4,799 for 24 concurrent sessions. While with BrowserStack unlimited plans start at $249 for a single user and rise to $6,229 for 25 concurrent sessions.
Here at Functionize, we have embraced the cloud revolution with open arms. But we have added a bit of AI magic to create a system that has revolutionized software testing. Our system is based on an intelligent test agent to power truly autonomous testing. The system builds an incredibly detailed model of your application and UI using a combination of AI techniques. These include machine learning, image recognition, and natural language processing. Tests are written in plain English, making test creation accessible to many more members of your team. Once they have been modeled by our Adaptive Event Analysis™ engine, tests automatically work on any browser or device. When you update your application all your existing tests simply self-heal. Without the power of the cloud, our intelligent test agent couldn’t function.
At the heart of Functionize lies our Test Cloud, which is hosted on Google Cloud Compute (GCP). Without the cloud, we wouldn’t have access to the compute power needed to run Natural Language Processing or our robust ML Engine (the two technologies at the heart of Functionize). But using GCP brings several other key benefits for our customers.
Finally, because you can complete your tests so quickly, the Functionize Test Cloud enables proper CI/CD. You can run a complete test suite in just a couple of hours. And of course, you aren't having to pay any test infrastructure costs!
With Functionize, deploy 100 new tests in 6 days vs 74 days with Selenium
There can be no doubt that the future of automated testing is in the cloud. Not only will this save you test infrastructure costs, but it will also help you to create a more robust test setup. However, we go further and say the future of testing is the intelligent application of AI. This can only be done when you have access to the true power of the cloud.