Smoke Testing vs. Sanity Testing – Key Differences

Smoke and sanity tests are different filtering methods used to quickly determine the stability of a particular software system build before it moves on to more rigorous and expensive test stages.

Smoke tests are typically performed as part of a daily build/regression process, whereas sanity tests are often a gateway to more detailed acceptance testing. Both types of testing ward off many hours of fruitless detailed quality tests that require significant setup and configuration but are unable to complete due to fundamental defects that could have been easily detected in earlier testing stages.

Smoke Testing

Smoke tests are the first to run immediately after a new build of the software. The name derives from the old practice of applying power to newly constructed circuits to see if they catch on fire or if specific components develop “smoke leaks.” Their primary purpose is to provide the most basic “signs of life” test for newly compiled software. They are a suite of elemental troubleshooting tests that determine if the most critical parts of the application under test can initialize and complete their startup procedures, such as presenting a working GUI or connecting to a database or network.

Smoke test procedures are based upon detailed, documented functional requirements of the software developed before coding began. As such, some or all of these tests may be incorporated into the code itself as internal self-tests that verify that all components have loaded and are in an initial stable state. Smoke tests may also turn up problems with the build itself, such as linking to the wrong version of a particular library. They also determine that all third-party resources required for the application to run are present.

Smoke tests can be run by both developers and testers as part of daily regression testing immediately following each software build. To provide the greatest value, the suite of smoke tests should verify build and execution log files to confirm the absence of unexpected errors.

Sanity Testing

Sanity testing always follows smoke testing. Though it has the same general goal as smoke testing, which is to find defects early enough to avoid expending resources on more detailed testing, it has more specific targets. Sanity tests focus on new functionality or defects in previously existing functionality that are reported as fixed. These tests rely on the high-level stability of the system under tests that smoke tests have confirmed. This dependency avoids any confusion as to the source of defects that sanity testing turns up.

Sanity testing is typically performed with only light or no documentation, which may consist solely of a bug report. Since sanity tests focus on specific components or behaviour, they are less likely to be repeated at this stage in subsequent builds, so they are often done manually using exploratory testing methods.

Care must be exercised by testers if functionality other than that under test has a dependency on the particular defect fix being tested. In this case, there should be a sanity test on those components as well to verify that new defects were not created.

Particular sanity tests may migrate to follow-on acceptance test suites after the area under test has stabilized. Tests selected for migration are fully documented and included in a larger suite of automated acceptance tests.

Since it is impossible to determine an absolutely complete set of sanity tests a priori, it may also be the case that subsequent detailed testing turns up showstopper issues that are best handled at the sanity test level or even at the smoke test phase.

Other Uses for Smoke and Sanity Testing

Though the typical placement of smoke and sanity testing is near the front end of the development chain, the concepts can be applied at any point in a development/test/release cycle.

For instance, before configuring and kicking off a lengthy session of stress testing, smoke tests will ensure that all hardware, software, database and network dependencies are present and working. If glitches are found, the tester saves a great deal of preparation time.

Though these two testing terms, smoke and sanity, are often confused, they are actually quite distinct with their own specific purposes and methods. Smoke tests touch the software at the highest, broadest level as early as possible, whereas sanity tests are precise tests that probe as deeply as is necessary.

A smoke test is equivalent to checking that your toaster is in fact plugged in before placing slices of bread in the slots. Once you have determined that power is flowing, sanity tests make sure the toaster components such as the switch, heating elements and timer work according to specifications.

Both test methods are invaluable throughout the software development cycle. They provide a triage function that sorts out problems rapidly and in the most efficient manner. They have saved countless hours of ultimately futile testing effort in later test stages where test overhead is much greater.