This article can serve as a simple guide you can read and share with your team about the advantages and best practices for reliable smoke testing. We also have a free offer at the end to a limited number of qualified teams each month, so be sure to read all the way through and register for the free offering.
Smoke tests are tests that development teams create in order to quickly verify that basic functionality in a software application runs properly or, if the tests fail, reject the deployment until the basic errors are addressed.
High-quality smoke tests share some common characteristics and areas of coverage:
Additional, high-level functionality to include in your test suite will depend on your application and the needs of your users
A smoke test suite combines a number of smoke tests into a small suite of tests so that when the application is ready to deliver to staging or pre-production environments, developers will know in a matter of minutes whether the latest version of their application is ready for further QA and review. Smoke tests are also used at times in emergencies or when teams want to release a hot patch to their production applications. This last practice is not recommended and there are better ways, outlined in the last section of this article under "How to Get Started with Smoke Testing."
There are many benefits to smoke testing but the most prevalent include:
A common first step for companies who want to build a suite of smoke tests is to first start with manual QA. This is relatively easy to start with and, particularly in younger companies, people who aren't engineers or QA testers will frequently get pulled into performing smoke tests.
As teams mature, they may choose to hire a QA manager or a manual tester. This is a good first step, even you plan to automate in the future because we will show you how to turn those manual QA testers into automation engineers as quickly as possible. There are two major benefits to starting with a manual process:
The main point of a smoke test suite is to let teams know whether or not they should continue testing their entire application. It's also important to understand what level of testing should be included.
There are many types of QA testing but to keep things simple, let's look at Unit Testing vs. Functional UI testing.
Unit Testing is a best practice among agile software development teams and is something that we won't cover in great detail in this article. Unit tests are automated tests that happen at the source code level in a project and should be executed even before the smoke suite is run.
The best fit for smoke testing is at the Functional UI layer. Functional testing is all about testing the user interface (UI) in order to uncover new bugs (exploratory functional testing) or ensure that previously known bugs have not found their way back into the application (functional regression testing).
You want a smoke suite to be an easily manageable number of tests that can be executed quickly, regardless of whether the smoke suite has been automated. The recommended number of test cases is generally 20 on the low end to around 50 on the high end. Any less than this and you're probably not getting enough relevant coverage, and any more than this and you're probably trying to cover too much ground at this stage.
Once you have your suite of smoke tests defined, you're going to want to think about automation. Given that the point of smoke testing is to determine quickly the state of the release and its readiness for full QA, you don't want your suite to take more than a few minutes to execute. If you're running 50 tests, and it takes half a day (or more) to run these tests, then your testing cycle is going to be too slow. This is why you want to automate.
There are many challenges with Selenium and Selenium-based automation projects. As applications continue to increase in complexity, these problems will multiply. Selenium itself is buggy across browsers, and your typical Selenium-based automation projects are brittle, for various reasons. We're solving these and many other problems related to testing automation, so talk to us if you are trying to improve your automation.
Whatever automation framework you end up choosing, automating both your smoke suite and your entire testing suite after that should be top priorities for your team.
There are many good reasons for automating your smoke testing but the best reason is that you can get through tests faster. You should aim to complete this set of tests in 5 minutes or less.
Once you have your suite created and an automation framework selected, you'll want to think about a couple of key areas in order to keep your tests running healthy and providing valuable feedback:
The most important thing to remember is that speed is better than thoroughness when it comes to smoke testing. Don't worry about trying to get everything 100% right the first time, you can always improve later. Just think about the things you would normally do in your application and make a list, where each line is a separate test. For example, if you were writing a smoke test for Shopify, the popular e-commerce platform, you might write your initial test plan as follows:
As you can see, these are very simple sets of tests that you can easily pass to a manual QA test to perform, at a minimum, without much instruction. Even if you were going to go to the trouble of writing Selenium code, you could handle these steps easily. There is quite a bit of mess, however, to deal with in setting up your browser testing environment, dealing with a new code repository, and more.
With Functionize, it would take about 30-60 minutes to completely automate this smoke suite, which is something we will cover in our offer below.
Once you have your smoke suite created manually and then have automated it, think about a few next steps. A great next step is to integrate your suite of tests into a continuous integration (CI) tool. CI tools will kick off the test, and send you reports and alerts about the outcome of the test.
Before we started Functionize, the best answer would have been to build an automation project using a framework like Selenium. We've made that requirement go away completely. With Functionize, we can help you create a free smoke test suite of up to 20 tests. You're then free to use it either standalone using our internal CI tools or with your own CI tools. You can use it for both pre-production and production environments, up to once a day.
In order to see if you qualify, just fill out your name and contact information. You'll get an email from us. Tell us you read this post and that you want to qualify for the free smoke suite.
We'll ask a few simple questions about your team. We only provide this to a limited number of teams per month, so the sooner you apply, the better. Sign up here!