Defects that are seemingly non-repeatable are the nemesis of both developers and testers alike. They consume an inordinate amount of time compared to other bugs and significantly reduce confidence in the quality of the software under development besides having the potential to drive everyone to the edge of madness.
Many such defects should actually be classified as “hard to reproduce” or intermittent but developers and testers throw them in the cannot-reproduce bin after an initial, earnest effort in order to proceed with fixing bugs that are more tractable. There are strategies and tactics, however, to reduce the number of non-reproducible defects to a minimum.
Keen Observation Skills
A key element in replicating a bug that insists on hiding is to practice a strict discipline of detailed observation when reproducing any bug. These testing details might be disregarded for the easy-to-produce bug, but once a “one-time” bug is encountered, you will be quite thankful to have access to a detailed description of every step taken up to the point it occurred.
Such information must include the state of the software and hardware on which the test is run. The versions of the program under test, the operating system, OS updates and any third-party services or applications must be noted carefully. Much of this information can be collected automatically.
Testers should also record any other applications running on the system and the history of the machine and software up to the point the defect occurred. For instance, the time since last reboot, the state of a browser cache or the specific network adapter in use may have something to do with the bug.
If the bug is reported by an end-user or client, the recording of prior state is more difficult, but an effort should be made to collect as much detail as possible as early as possible before the user’s recollections fade. Obtaining screenshots or logs from the user may help establish relevant operating conditions at or near the time of the bug occurrence.
All such details are especially important to whomever is designated to re-produce the bug, which may not be the person who originally discovered it.
Often when a defect stubbornly refuses to reveal itself, it is useful to look at other bug reports that may be related. They may exhibit the same or closely-related error code or may have occurred in the same area of software functionality. For instance, if the error occurred while contact information was being filled in, look for other bugs related to that page or other pages where user textual information is input.
If you fail to find similar defect patterns in the software under test, take to the Internet to seek out technical forums whose topics are related to the genre of software you are working on and look for reports of similar bugs even if they seem less relevant. Such searches can often inspire other lines of attack or suggestions as to what can go wrong. Be careful to bound the time used for this kind of searching, however, as it can lead down non-productive rabbit holes.
The Application of Creativity
Most of the bugs developers and testers encounter are “routine” in the sense that they are relatively easy to reproduce. That is part of what makes dealing with a non-reproducible defect so frustrating. However, re-discovery of such bugs can be blocked by your own thinking ruts. Often, making an extra effort to apply creativity to the problem yields productive results:
- Challenge any obvious assumptions you may be making, such as being sure you are testing the correct software version. If you have out-of-hand discarded certain possibilities, re-visit those.
- Look for other tools that may provide insights from a different angle into the effects of the steps taken to reproduce the bug.
- Try stressing the system under test while going through the steps to reproduce the defect.
- If you are a tester, ask the developer responsible for the code that appears to have failed to set up an ad hoc code review focused only on that code.
Looking for irreproducible bugs demands a great deal of thought and patience. It also requires balancing the effort to find them against the impact on quality if you cannot find it in a reasonable amount of time. Keep in mind that some of these bugs will be fixed by the next software release as a side-effect of other defect repairs or a refactoring of code. However, that does not excuse you from recording the defect in detail in case it does return in a later release.
To bolster your mood when dealing with hard-to-reproduce defects, think ahead to the story of triumph you will tell when you finally latch onto that phantom bug and its cause. But do not let your pride get the better of you, because you know other non-reproducible bugs are still lurking in the software. After reading these tips and with additional experience, however, they are far more likely to succumb to your improved defect hunting skills.