Exploratory Testing vs. Scripted Testing

In scripted testing, test designers develop detailed test scenarios that testers transform into scripts. Scripted tests provide rigid guidance based on intimate knowledge of the software application under test and are expected to produce a well-defined set of outcomes.

Exploratory testers, however are able to make lane changes as they from a looser set of test parameters that often traverse less-traveled paths through the code, which can produce unexpected, yet equally valuable, results.

These testing approaches are not mutually exclusive. In fact, experiments have shown them to be complementary when applied simultaneously. Each works best for different test scenarios and their efficiency depends on different skill-sets from testers.

The Advantages of Each Approach

Scripted tests are favoured for their stability and ability to guide the work of less skilled testers. They produce measurable results and reasonable estimates for completion of the work. Due to their high level of repeatability, they are easily automated.

Exploratory, or context driven, testing requires guidance from designers, but this guidance takes the form of recommendations or outlines versus a list of specific steps. Testers have greater freedom to explore alternate testing paths based on experience they acquire during testing. Generally, exploratory testing takes less time to prepare, but requires self-motivated testers possessing strong analytical skills.

When to Use Scripted Testing and Exploratory Testing

Exploratory testing is highly efficacious in the earliest stages of software development when code is undergoing rapid changes. Developers themselves can apply exploratory techniques to unit tests, while testers use these techniques to acquire application familiarity rapidly. Experience gained from exploratory tests is invaluable in guiding subsequent test scripts.

In agile environments, context-driven testing is an effective way to keep up with short scrum cycles that prohibit developing formal test design and scripting. Test outlines are developed almost on the fly, which saves time while still uncovering defects in code or design. As each agile cycle ends, the most valuable exploratory tests can be captured for subsequent scrums.

Structuring Exploratory Approaches

Skilled exploratory testers employ a number of effective techniques to attain high defect discovery rates in the absence of formal tests. These include describing software behaviour in terms of nouns and verbs, testing multiple paths to achieve a single result and testing time, data and workflow boundary conditions.

Naturally, test team managers would feel uncomfortable if exploratory testing were a completely open-ended free-for-all, so formal methods have been developed to rein in exploratory testing while permitting the necessary freedom for effective testing.

Charters

Exploratory test charters are akin to recommendations as compared to highly specific engineered test scripts in the same way that stop signs are regarded in third-world countries versus developed countries. In the former, ignoring the sign usually invokes a harsh penalty, whereas in the latter it typically induces mild annoyance while keeping traffic moving smoothly.

Test charters outline an area or areas to be tested and features or functions of particular importance, but do not tell testers precisely how to test within those parameters or even require that the parameters be strictly followed. Specific charters are expendable based on the results that were produced.

Sessions

Session-based testing usually includes a test charter, but is fundamentally bounded by time rather than specific functionality or features. The amount of time for each session may be half an hour to several hours during which a tester attempts to produce as many results, i.e. defects, as possible. The idea is to bound and measure the activity of testers whose defect discovery rate typically diminishes the longer they test. Each session is evaluated for its efficacy and either continued or a new charter is applied.

Mind Mapping

The technique of producing contextual mind maps is a good fit with exploratory testing. It can be used before testing begins as a way to define the goals of a session or test charter. During testing, the relationships between applied tests, functions, features and discoveries augment the original mind map or are captured in a separate map. These are a low overhead way to categorize and organize tests as they are performed and to guide further exploratory testing.

Regardless of which structuring technique is applied to exploratory testing, results evaluation should be frequent and an emphasis put on finding the highest risk areas in the software as testing progresses.

 

Exploratory testing should never be random or ad hoc in order to be most effective. It is also not generally applicable to all areas of testing, such as performance, load or stress testing or wherever a more predictive approach is required. Despite its proven value in defect discovery, it does not supplant scripted testing, which is necessary to maintain system stability or where strict measurements are required.

In the final analysis, exploratory testing is one of many invaluable tools at the disposal of any test team. It can help them cover more ground, more quickly especially in the earliest stages of development or when agile processes are in effect. Properly bounded and performed by skilled testers, it has been shown to be as or more effective than formal, scripted testing in the right contexts.