Global-class, enterprise SaaS providers face huge challenges in ensuring their business-critical apps remain available and perform as advertised. Recently, the team at Functionize had an opportunity to examine exactly how conventional test automation frameworks were failing to meet the needs of one large SaaS CRM provider for automation, insight, and scale. We were able to propose a Functionize solution that solved a big problem for this business -- helping them save time, allocate developer resources more efficiently, and keep customers happy -- all while continuing to evolve their app rapidly to stay ahead of competition.
THE PROBLEM: SCALING TEST AUTOMATION ACROSS A COMPLEX SITE
Modern SaaS apps tend to be extremely modular, offering an ever-broadening array of capabilities to many tiers of customers via a host of browsers and devices. Views, reporting, and other details are highly customizable, creating many uncharted paths through applications. SaaS code is constantly in flux, with back-ends, web front-ends, and mobile clients continuously improved, updated, and changes pushed to production daily or hourly.
Tens of millions of user sessions per month. Billions of potential paths through applications. Multiply through by dozens, or hundreds of browser, OS, and device combinations and it’s no surprise that script-driven, QA-engineer-maintained, premise-hosted software testing, based on tools such as Selenium and Robot, can’t keep up.
CONVENTIONAL QA HITS THE WALL
This SaaS CRM vendor needed to verify that sophisticated user segmentation and personalization features were working correctly in production. But this meant dealing with an explosion of possible user types, paths, pricing categories, access domains, and client/device combinations -- a huge test surface, much too large to cover with Selenium or Robot coding. Even with a burgeoning QA codebase of over 300 tests many of their test cases were repeatedly at risk due to frequent application updates.
But that challenge was just the beginning. Verifying production releases meant testing frequently (up to thousands of full-suite executions per day) across multiple datacenters -- a big task even with scaled-up DevOps test tooling. And all this work would be less useful if they couldn’t triage this flood of test results, identifying and prioritizing important bugfixes, and minimizing real user impacts.
ENTER FUNCTIONIZE
This CRM provider turned to Functionize’s AI-Powered Testing in the Cloud. Functionize offered a completely new approach to software testing -- one that met their requirements for speed and scale, verified app availability and correct performance in real-time. It also helped prioritize interventions, ensuring consistently-optimal user experience.
With the help of Functionize support, three of the provider’s dev/QA teams were quickly up-to-speed and productive with Functionize because test creation and execution do not require Selenium coding. They swiftly converted a large subset of existing functional tests, trusting Functionize to automatically parse web page semantics, emulate browser/platform and client quirks, and use machine vision to judge the visible (as opposed to logical) correctness of web page displays and UI behaviors.
They then swiftly instrumented the whole production application, adding code enabling Functionize’s Real User QA™, which builds test-suites automatically based on actual user behavior. Suddenly, millions of daily SaaS users were silently recruited to prune the combinatorial test-surface, helping Functionize build out a meaningful library of test sequences focused on what users were doing, reflecting their segmentation and different needs.
Testers were then able to quickly schedule a large volume of real-time tests against the production application, for automatic execution by Functionize bots. Functionize machine learning was again pressed into service to help triage resulting alerts -- with Functionize learning which issues required the most immediate attention, and notifying appropriate personnel to intervene. For further insight, ad-hoc tests can be run in minutes, with analytics immediately available.
RESULTS
Over the first 90 days, this CRM provider was able to fulfill requirements for real-time production testing, and help ensure continued high levels of customer satisfaction with their fast-evolving product. They’ve slashed time spent creating and executing tests by 75%, freeing developers to innovate on product not writing test scripts. Thanks to Functionize, they have improved insight into segmentation strategy, providing paths to increased revenue.