Test Automation: Myths and Facts

Automation is an area of software testing encompassing a multitude of techniques and tools with potential application throughout the software development process. Some organizations with automation experience may hesitate to expand its use due to concerns regarding costs or a perceived lack of relevant skills. Others may be overly gung-ho to automate their development processes end-to-end based on unrealistic expectations. In both cases, certain “myths” of test automation must be addressed in order for development and testing teams to correctly apply it in the most cost-effective manner.

Myths That Impede Expanding Automation

Test Automation Costs Too Much

Taken at face value, this myth has a kernel of truth within it. Automation has significant upfront costs that are difficult to avoid. Proprietary frameworks require large licensing fees. Open source tools require consultants or in-house expertise to realize their full value. On the other hand, can you afford the costs of less competitive software quality, longer release cycles or missed customer requirements? Those can far exceed the price of automation.

Realistically, to extract value from test automation, a number of things are required:

  • An automation architect to match the software testing requirements against the capabilities of automation tool suites.
  • Automation engineers who prioritize test automation to specific test areas that save time and improve quality.
  • Plans to reuse automation for a string of future projects to profit from the initial investment.

We Have to Replace Manual Testers

Related to this belief is a myth that test automation completely replaces manual testing, which we will get to later. Suffice it to say that there is always a need for manual testing.

As agile development methodologies and continuous delivery processes grow in use, automation frameworks have responded by accommodating the collaboration of both technical and non-technical staff test development and execution. For example, Selenium IDE allows non-coders to build basic but valuable test automation. Selenium IDE scripts can be exported to Selenium WebDriver or other higher-level tools for further enhancement and expansion within the grasp of retrained manual testers.

Furthermore, as an organization’s commitment to test automation increases, two things happen:

  1. The development/testing culture spontaneously institutes a virtuous feedback cycle that heightens awareness of additional automation opportunities.
  2. Test resources bogged down in repetitive, manual testing activities are redeployed higher-value testing activities that actually increase software quality.

Myths That Automation Is a Panacea

All Tests Will Be Automated

Test automation has its limits. Ignoring those limits wastes time and political collateral. Automation works best with highly repetitive or highly parallel test scenarios, such as build regression testing or load testing with large data sets or multiple users. It cannot test for user experience or for unpredictable results that require human cognitive skills to assess correctness.

Automating UI testing is another area fraught with complications and ineffectiveness due to the fragility in under-development UIs and the high cost of test maintenance and execution. It is especially ineffective when the test target is code beneath the UI.

Faster Software Release Cycles

Expecting test automation to speed up development/test cycles misses a key point. The goal of automation should be improving software quality rather than push software out the door faster. Properly applied, automation accomplishes deeper and broader code coverage. More defects are uncovered and in a more timely manner.

The closer to actual code writing that test automation is implemented, the more efficient is this effect. In an agile development environment practicing continuous delivery, end-to-end automation does indeed speed up delivery.

Automation Pays for Itself Quickly

It can be true that tactically applied test automation, such as automated developer unit testing within the IDE, pays for itself relatively quickly. Otherwise, automation must be viewed as a long-term commitment. It takes meaningful investments of time and money as the organization learns to use it in the most effective manner. Expect 10 to 20 software releases before fully recovering that investment.

A reluctance to adopt or expand test automation capabilities is often rooted in beliefs that automation is too costly and requires wholesale replacement of manual tests and testers. In fact, the costs of not automating are higher, manual testing is still required and personnel are redirected to higher-value tasks.

A wholesale adoption of automation, on the other hand, may be based on unrealistic expectations of what can and cannot be automated, an expectation that release cycles will shorten or that costs will be recouped quickly.

In either case, a careful evaluation of the advantages and disadvantages of test automation, appropriate tool selection and its effective application will dispel these myths. A deeper understanding of automation and long-term commitment to its use reap the greatest rewards.