A brief guide for first-time users
Functionize delivers the first autonomous, AI cloud-based testing platform. This guide will help you get started. We’ll show you how to create tests, how to run tests across all major browsers including mobile devices (beta) and how to view the results of the tests. We’ll also explain how Functionize ensures that tests are self-maintaining and explain how things like 1-click updates work. Finally, we’ll briefly touch on API and load testing with Functionize.
NLP test creation
Right from the start, Functionize makes life easy for you. Our NLP (Natural Language Processing) engine means you can simply create your test in plain English, upload it to our system and your test will be created for you. The system understands commands based on keywords (“SCROLL down the page and VERIFY that the buy now button is shown”) as well as natural language (“The top menu bar should be the full width of the page”).
Example NLP Test Creation
After the test has been generated, you have the chance to approve it or tweak it step by step.
Test approvals process
The Functionize platform makes it extremely easy to execute tests. You can run individual tests or create orchestrations which run suites of tests, and you can test across multiple browsers and hardware architectures. We partner with Google to provide the Functionize Test Cloud which allows you to test your site from any geographic location and with simulated bandwidth restrictions – essential if you expect your users to come from other countries or to use bandwidth-limited connections.
The NLP-generated script being run
As the tests run you can see their progress on your dashboard. Once the test has completed you can quickly see if any problems were found. The system also takes a screenshot before, during and after each and every step in the test, which you can scroll through. This will highlight any problems it found to save you having to search for ages, hence helping with fault diagnosis. The visual comparison is done using template recognition, an AI-powered “fuzzy comparison” which won’t trigger a failure just because of a simple CSS style change or rendering artifact.
One of the biggest time sinks with automated testing is test maintenance. Traditional Selenium-based systems are dumb and 1-dimensional, being limited to a single selector at a time. The slightest change to a style sheet or a simple change to the page layout breaks the script. Our tests are completely self-healing. Each action is fingerprinted by the system, which uses machine learning to establish what the action is actually doing. So if you restyle your “Buy Now” button as “Add to Cart” the system will cope since it can tell the underlying functionality is the same. Likewise, if you move the button from the bottom left of the page to the bottom right, the system still knows it’s the same button.
Root Cause Analysis
Some test problems are harder to find though. These are typically cases where an action completes without error, you move on several steps in the test, and then the problem appears. Analyzing such failures is one of the hardest things in test maintenance. For instance, imagine your test places an item in the cart, then at a later stage checks if the correct item is in the cart. Imagine that your page has 2 Add to Cart buttons. One for the product being viewed and one for a special offer at the right of the page. If you redesign the page so that the main button changes to Add Now, the system may accidentally select the special offer instead. Now the test will happily proceed, but the failure will become apparent when it finds the wrong item in the cart.
Functionize’s ML Engine includes a Root Cause Analysis feature which is able to deal with this situation. By learning from previous tests, RCA can track-back, find the most likely problems, test the solutions and then present you with the best solutions ranked by probability.
If you click show suggestion, it will take you to a screenshot from the test (see below)
As you see, this page has 2 buttons to add an item which the system has highlighted. Simply by clicking the correct one, the system will repair the test. Now any tests that rely on this step will also be repaired.
The Functionize CLI
We know that many of you are more at home using the command line than a GUI! Which is why we have developed the Functionize CLI. We’ve built it on top of Java to allow portability across OSes. Simply go to your account, download the CLI zip file to your computer, extract it and you’re good to go. The file will already have your API credentials embedded so there’s no need for any setup. It also includes inline help so if you pass the wrong variables it will tell you. Here are a couple of cool things it can do.
Testing a local server
Often you want to test a feature before you push it to staging. One of the coolest features in our CLI is the ability to test against a local server. Spin up a local instance of your site with the changes you want. Put it on a suitable port and then tell the CLI to use that port for testing. Simply use the start-tunnel command and the CLI will create a secure tunnel:
functionize start-tunnel 5005
Now when you run a test, the Functionize system will run the tests against your local server over this secure tunnel.
Viewing test output
One of the most popular features in Functionize is the ability to view all the screenshots taken during a test. The Functionize CLI supports this using the view-screenshots command.
The Functionize CLI will launch a browser and take you to a custom URL showing the results of your CLI test. The URL is secure, but it can be shared with other members of your team if they are logged in. This feature is ideal for creating custom test integrations.
Integrating Functionize as part of your CI pipeline
Functionize works seamlessly as part of a continuous integration/continuous deployment (CI/CD) pipeline. For instance, here at Functionize, we use Spinnaker as part of our own pipeline. Spinnaker is ideal because it is open-source and works on IaaS clouds (clouds that run virtual machines like GCP, OpenStack, AWS, VMware, etc.) and container orchestration platforms (clouds that run containers like Docker Swarm, Kubernetes, etc.).
Spinnaker makes it easy to deploy new builds to staging. It can trigger a full set of Functionize tests (e.g. smoke tests, regression tests, API tests, etc.), and if the tests pass, the new code can be automatically deployed using the Promote to Production pipeline. Our customers have also used other CI tools to achieve the same including Travis CI, Heroku and Jenkins. If you want more advice, talk to one of our Solutions Architects.
Other test tools
The platform also provides tools for testing other aspects of your system, not just the web app frontend.
Our load testing runs on Google Cloud and is designed to perform far more realistic load testing than the traditional method of making multiple API calls. This is because it uses realistic user sessions generated by the Functionize test system from any location around the world. As a result, the tests accurately predict how real users will see the performance change as load increases. There are two modes of testing – testing how the system performs under a specified load and steadily increasing the load to trigger a failure. In the former case, the results will show you details, such as how quickly different page elements loaded, which elements were outliers and the relative performance of each page. Results are available in real-time and give you a detailed breakdown of the page performance.
The load tests results dashboard gives you detailed statistics
You can compare the relative performance of each page to see what is adding delay
You can see how long each step in your test took to complete
You can quickly spot outliers
Included in the Functionize platform is an API testing solution that uses an AI-powered recorder to capture the API test steps. Unlike traditional API testing, our approach tests APIs visually, reducing time and simplifying the process of creating API tests. Our platform also allows you to take responses such as API keys and store them using local variables. They can then be accessed later in the test just by calling the variable. This makes the tests far easier to create, and means an end to repeated copy-pasting of such details. The system also builds in extensive validations and uses machine learning to track how the API response should look. This allows it to be resilient to things like changes in response order.
As you add steps you specify the type of call, response type and output format
You can store the key as a parameter called apikey. Now when you run your tests, the API key will be automatically copied in the previous step and then it can be referred to in subsequent steps by putting apikey.
You can now copy the access token from the response