What Is a Test Automation Framework?
Automated testing frameworks provide test environment structure that is typically missing from underlying test tools. Each framework style offers unique rules, guidelines, protocols and procedures for test creating, organization and execution.
Six types of test automation frameworks are regularly encountered. Those covered here increase in complexity and levels of indirection to achieve their goals. The aspects to evaluate include scalability, re-usability, maintenance effort and the cost of technical skills.
In most cases, the choice of an initial style of framework is less important than the establishment of any framework, which brings standardization to development and testing processes and improved efficiency.
Module-Based Testing Framework
This framework assigns independent test scripts to specific software modules, components or functions. Avoiding script inter-dependencies is crucial to framework stability and maintainability.
Each module script embeds both actions and data. Any change in the test data requires script changes or a new, separate script. Where data changes are frequent, a data-driven testing framework would be preferable.
Common Library Testing Framework
This framework is a module-based framework that segregates common script functions into a shared library that each test script calls. This approach avoids having to re-implement common functions within each test script. This reduces the lengths of scripts and repetitive effort. A drawback is that if a common function is incorrectly implemented, then the error propagates to many scripts.
Data-Driven Testing Framework
Where tests utilize varied sets of input and output test data, a data-driven approach is more efficient and more easily managed. Test scripts obtain all input and output data from an external database, which improves their re-usability. By convention, test data are stored as key-value tuples in a standardized fashion. Data extraction within scripts is facilitated by a common library.
This type of framework significantly reduces the number of test scripts compared to a module-based framework. The test data matrix can be changed independently of the scripts as long as required data are not deleted entirely. This framework introduces more complexity overall and requires intermediate programming skills to set up and maintain.
Keyword-Driven Testing Framework
This framework is also called table-driven. It separates test automation from test case design by adding an additional layer of abstraction via action keyword-value tuples in the data matrix. The keywords are used to drive tests independently using underlying test execution tools. This framework style enables less technically skilled staff to create test scripts. It also improves test script readability.
The implementation and maintenance of such a framework does require programmers who can create the keyword execution mechanism or who can leverage open source tools such as Robot Framework.
Hybrid Testing Framework
A hybrid test framework is a combination of two or more of the previous framework types. It attempts to leverage the most desirable benefits of other frameworks for the particular test environment it manages.
For instance, one hybrid approach would be the use of a common library coupled with a data repository that combines both test I/O data plus action keywords. Each tuple in the repository may contain a name, description, action keyword, UI locator and text string, if necessary, for an input field.
Naturally, a hybrid approach may be more complex to set up initially. However, it could provide the greatest flexibility if the tradeoffs between the frameworks it incorporates are carefully evaluated and implemented.
Behavior Driven Development Framework
This framework is unlike previous frameworks in that it tends to be more a development process than a test management structure per se. Its purpose is to facilitate test specification by business stakeholders as early in the development process as possible. It requires increased collaboration between development and test teams as well.
It is centred on the use of a non-technical, semi-formal, natural language for test case specification. This process is facilitated by higher-level tools like Cucumber or RBehave in conjunction with testing tools such as Selenium.
A key drawback to this approach is that BDD scenarios are so abstracted that they are often at odds with technical reality. Furthermore, in practice, it is difficult to obtain participation from business stakeholders in creating or evaluating BDD scenarios
This brief survey is not an exhaustive list of test automation frameworks, but the most popular styles are covered. Other than BDD, these frameworks have no inherent incompatibilities. An organization could easily begin with a module-based approach and incrementally grow their framework to more complex styles over time depending on development and testing needs and the organization’s technical expertise.