At first glance, it may appear that a Continuous Integration model of software development leaves testers out in the cold. One such feature is the increased responsibility for defect detection falling into the laps of developers. This so-called “left shift” of testing is intended to find bugs sooner, faster and facilitate their repair.
CI also significantly reduces the time between lines of code written to them showing up in production systems. Ideally, a well-done CI implementation compresses this cycle from months down to weeks or even days. In the context of waterfall-like environments, the demands of CI release frequency would be all but impossible to maintain for conventional testing roles.
How QA Maintains the Pace
Even given that more left-end testing is handled by developers, when the day is done they are still developers and not testers. Furthermore, that testing is mainly going to be of a functional flavor with perhaps some regression, performance, exploratory and UI testing thrown in. Loads of other testing, such as stress, compatibility, UX, BVT, API, security, scalability and so forth are unlikely to be taken on by developers lest they stop writing code altogether.
On the other hand, if any of those testing domains were to remain as sequential or manual processes, they risk the persistence of bottlenecks that undermine the fundamental benefits of Continuous Integration.
What changes for QA in the context of CI is not so much testing domains, but an increase in responsibility for creating and managing a more effective testing pipeline via various means:
- Automation – Testers must become experts in automation from every angle. They do this under the direction of one or more automation engineers/architects. Opportunities for automation must be sought out aggressively.
- Concurrency – This aspect refers not only to running test suites in parallel, but also to aligning testing processes to the same timelines that development uses. Test scripts or scenarios are developed from requirements before code is written, risk planning takes place as early as possible, testers participate in scrums, planning and business meetings, all test code is maintained within the same revision control system with the code plus testers are paired with developers as code is written.
- Virtualization – In order to get the biggest bang for the buck via automation and concurrency, QA needs to increase their expertise in the virtualization of development, testing and production systems. They are in a perfect position to anticipate the need for development/test assets. In fact, it makes sense for QA to take on a partial Operations role in developing instant acquisition of correctly provisioned and configured systems for developers and anyone else in the CI pipeline.
Potential Pitfalls for QA in CI
CI itself, let alone QA’s enhanced role within CI, does not come about overnight. It requires several months or more for most organizations to get the hang of it. Meanwhile, software production cannot idle while it is put in place. In this sort of intense, highly focused effort, it is easy to overlook hard numbers regarding ROI. It is vital to keep in mind that if upfront and maintenance costs to QA to maintain pace with CI are not accurately weighed against the long-term benefits, the organization risks playing a zero-sum game or worse.
Furthermore, QA must recognize that automation, concurrency and virtualization all have their limits. They are not appropriate solutions for every testing problem that arises. There will always be a need for manual testing such as exploratory tests, UI shakeouts or live user test sessions. Automation also produces increased overhead, such as test script maintenance or the acquisition, processing and scrubbing of live user data at the production end of the pipeline.
CI adoption has proven time and again that it can have an enormous positive impact in terms of reducing the time to market of new software features while cutting down on production overhead. To make it work right, however, the test organization must become a concurrent partner alongside development, up their use of automation and take charge of producing effective, efficient solutions to slow-downs in the CI testing pipeline wherever they occur. Having done so, QA will continue to provide a vital function in ensuring that an enterprise’s software is delivered fast and to the highest quality standards.