There is a large overlap between the capabilities of manual and automated testing, but neither can completely replace the other. Cases where one approach is clearly superior over the other are easy enough to discern, but the tradeoffs between each in overlap areas requires more careful consideration.
Characteristics of Manual Testing
Functional testing at the unit, integration or system level is often done manually. During early development phases, manual tests make a good deal of sense when developers are exploring implementation choices and testers are performing risk analysis and designing subsequent test tactics. Tests at this stage have a short half-life, and the advantages of human cognitive skills and intuition are more valuable than high-maintenance automated test suites.
When the software user interface begins to emerge up until final release is an excellent time to apply manual testing also. User interfaces frequently are in a constant state of flux right up until the final stages of development. Additionally, the usability and aesthetics of the UI require human cognitive skills to evaluate. For both these reasons, it is difficult to utilize automated testing methods.
Manual testing of software even by non-technical staff is possible at any stage of development, of course. This sort of ad hoc testing often leads to the discovery of unexpected defects or insights into how the software could be improved.
Characteristics of Automated Testing
Automated testing is almost universally applied to build regression testing for the purpose of validating that new code changes have not resulted in new defects once the integration phase has been reached. This is typically the starting point for a formal test automation process, although earlier automation at the unit test level is also possible.
Beyond regression tests, automation is very useful for any kind of highly repetitive tests. Whenever test suites can be divided and run in parallel to save time, automation is also the preferred method over hiring teams of manual testers. In order to run performance or load tests that simulate hundreds or thousands of concurrent users or transactions, automation is essential.
Although there are record/playback tools that generate test scripts via UI interactions, these are not typically the best use of automation because of the fragility of the test scripts if the UI is undergoing even slight changes. It is especially inappropriate if the purpose of the UI test scripts is actually to test layers below the UI. Those should be tested directly via APIs.
Summary of Manual vs Automated Testing Advantages and Disadvantages
Manual Testing Pros and Cons
- Exploratory testing is best done manually
- Manual testing is cost-efficient for tests run only a few times
- Manual testing of UIs is most effectively done by humans
- A software’s UX can only be evaluated by human testers
- Relative to the amount of code coverage, manual tests are inefficient
- Manual testing is prone to human error, so may lead to inconsistent or misleading results
- Human test resources are more expensive than test machines
- The simulation of large numbers of users or configurations is nearly impossible to accomplish manually
Automated Testing Pros and Cons
- Far more code is covered in a shorter time using automation
- It relieves human testers from highly repetitive, low-value test work
- Automation can allow non-technical staff or customers to run tests
- Automation enables time savings due to parallelization of test execution
- Automated tests can simulate thousands of users and configurations easily
- Automation and maintenance of test suites requires significant investment in tools and skills
- Automation cannot replace human cognitive skills required for evaluating software’s UI or UX
- It is not appropriate for exploratory testing used to improve the software’s design
No software organization would dare attempt to test their products solely via manual testing these days. Neither can they hope to achieve 100 percent test automation, nor would this be a good idea given the limitations of automation in some areas.
It is the in-between spaces where the effectiveness and costs of both approaches must be carefully evaluated. Such evaluation leads to the most sensible choices relative to the capabilities of the organization. In general, however, informed decisions to automate testing where possible typically lead to better test coverage and improved product quality.