QA
Q&A
Question: What is a Smoke Test?
Answer: A subset of all defined/planned test cases that cover the main functionality of a component or system, to ascertain that the most crucial functions of a program work, but not bothering with finer details. A daily build and smoke test is among industry best practices. [ISTQB]
When you want to quickly assess the state of a software build, you often turn to a Smoke Test (sometimes called a Build Verification Test). Essentially, you are trying to do something that is relatively quick and inexpensive, that will give you a general feeling for the software. You aren't looking for all the bugs. You aren't trying all the scenarios. You just want a general idea if the software works or not.
Often, the goal of this Smoke Test is to decide if more in-depth, more expensive testing is worthwhile, or if the software is too broken to be bothered.
Usually, you exercise only the basic paths of the software, and avoid all the unusual conditions. And usually, you don't test deeply at all - you just skim the surface.
In her book Effective Software Testing: 50 Specific Ways To Improve Your Testing, Elfriede Dustin says that a Smoke Test is "a condensed version of a regression test suite". And many companies do just that - extract a portion of their overall regression test suite, and use it repeatedly as a Smoke Test.
Manual Smoke Testing
Sometimes, a manual Smoke Test just involves "playing with the software for a while". For an experienced user/tester of the software this may make sense. Such a tester may have a good sense of what to try, and just how deep to go.
But often, a written script tells the manual Smoke Tester what to test and how.
Either way, the manual Smoke Test should be brief and to the point. If the software is in generally good shape, the manual Smoke Test should pass. If not, the Smoke Test should fail.
Automated Smoke Testing
I see this as one of the basic uses for my Test Automation Suite. I want to be able to quickly run a test after the build, so I can assess whether or not my team should bother to dig in and spend their time testing it. If the system-under-test passes the Smoke Test, we can proceed. If not, we reject the build until it is fixed.
If the builds occur overnight, I like to be able to schedule this Smoke Test so that it runs after the build and so that the results are ready and waiting for me when I get in the next morning. Sometimes, this allows me to run a larger overnight test and still have the results ready for the morning.
When I start to automate the testing of a system, the Smoke Test is usually the first automation I create.
In their excellent book How We Test Software At Microsoft, Alan Page, Ken Johnston, and Bj Rollison list these attributes of an automated Smoke Test (Build Verification Test):
- Automate Everything - run it on every build, the same every time.
- Test a Little - verify basic functionality.
- Test Fast - should execute quickly
- Fail Perfectly - fail only when the build isn't suitable for further testing
- Test Broadly, Not Deeply - focus on primary usage scenarios
- Debuggable and Maintainable - easy to fix, and keep up to date
- Trustworthy - you must be able to trust these tests
- Critical - requires time and careful thought to get it right
