Posts

Cloud-Based Test Automation Tools and Benefits

Acquiring and maintaining in-house testing platforms for even small-market software applications is expensive and time-consuming. If your software products include mobile applications that support thousands or millions of users, then the sheer scale of testing can overwhelm even well-stocked IT departments. Fortunately, with the availability of a multitude of cloud-based testing environments, there is a better way to meet your testing platform requirements that will cost less and improve efficiency.

Multi-Faceted Cost Savings

To start with, renting cloud-based testing environments means your company avoids huge capital equipment expenditures for servers, networking equipment, OS licenses and so on. Additionally, hardware and software maintenance costs plus the cost of personnel to manage such resources is borne by the cloud service. These are meaningful advantages for any company, especially for cash-strapped startups.

Furthermore, you avoid depreciation costs on test platforms regardless of whether they are in-use or not. That advantage, along with knowing that test platform backups, upgrades, provisioning and configurations are taken care of automatically, brings a strong measure of stability to ongoing cost projections.

On-Demand Testing Expedites Planning

Multiple surveys have indicated that, due to the periodic nature of testing, most companies that maintain in-house testing infrastructure achieve less than 50 percent utilization of that infrastructure. Most cloud-based testing services enable scaling test infrastructure up and down at will, so you only pay for what you actually use at any given time.

Instant scalability of hard testing resources provides an enormous benefit from a planning perspective. It eliminates the scenario where test platforms are acquired or re-commissioned in preparation for large-scale performance and load testing only to have them sit idle after development is delayed.

Additionally, many cloud-based mobile-app testing services provide virtual access to large arrays of released or soon-to-be released mobile devices and various OS versions. When it comes time for full-scale compatibility testing, few companies are equipped with the necessary resources to duplicate such a large and diverse testing environment at almost a moment’s notice.

Reducing False Defects Due to Inconsistent Configurations

A large percentage of software defects can be traced to incorrect test environment configuration. For example, provisioning a platform with a specific OS version plus service packs, databases and third-party middleware or applications on which your software depends is an error-prone process to say the least. Cloud-based testing services excel at providing consistent test platform preparation backed by service level agreements, so that you can avoid wasting time and resources tracking down phantom bugs.

Reducing In-House Test Applications

Besides the advantages inherent in renting versus buying your test platform infrastructure, the best cloud-based testing services also offer world-class testing frameworks with which you can create, run and manage your test suites. This avoids the costs of licensing in-house testing tools plus the costs to install and maintain in-house tools.

Increased Collaboration Improves Time-to-Market

Furthermore, most cloud-based tools make a point of being user-friendly and only require access via a browser. Thus, greater accessibility to the tools, test suites and reports is achieved across development, test and business units. Access can be granted outside the company as well to provide transparency to clients.

Most importantly, because testing resources and tools are virtualized, collaboration between developers and testers increases regardless of their physical locations and the time zones within which they perform their work. Testers can share results instantly via screen sharing with developers, which reduces communication failures and delays immeasurably.

As software products become more complex, more distributed and release cycles are increasingly shorter, maintenance of internal testing infrastructure is increasingly impractical in terms of cost, quality and time-to-market. Although virtualization of such resources via cloud-based testing services will never completely replace internal testing, an emphasis on its use brings immense benefits. The reduction of capital expenditures, ongoing cost reductions, test platform scalability, planning flexibility and enhanced collaboration are advantages too great to ignore.

A final key advantage to cloud-based testing services is that they can be sampled without incurring a large commitment by enterprises who desire hands-on experience of cloud-based testing relative to their particular situation. An experienced partner can save time and provide enlightening insights in this regard.

5 Advantages of Automated Software Testing

automated-testing-6 5 Advantages of Automated Software TestingAutomated vs Manual testing has always been a point of debate among software professionals. Most of us realize the utilities of both, but the confusion stems from the extent of automated testing for your software. Here, we make a case for intelligently deployed automated software testing and its benefits.

Benefits of Automated Software Testing

Test Automation is essentially using code to create programs that perform automated tests for your software. The way this is different from manual testing is, instead of actually performing the test, one creates an automated testing scenario and supervises it. Test automation is extensively used for regression testing, which seeks out new bugs in a program and separates them. Regression tests are generally extremely tedious and time consuming. Here is where automated tests come in and make life easy for a software testing professional. Apart from this code-driven testing type, the other arena for automated testing is user environment simulation. Testing software can be created to replicate typical user environment using automated keystrokes and mouse clicks. The software GUI response is recorded and analysed as per the automated input.

Efficient Testing

Test automation is a way to make the testing process extremely efficient. The testing team can be strategically deployed to tackle the tricky, case specific tests while the automation software can handle the repetitive, time consuming tests that every software has to go through. This is a great way to not only save up on time, money and resources, but also to generate a high ROI.

Upgradation and Reusability

One of the best aspects of test automation is that the testing software is reusable. Not only that, but with every new test and every new bug discovery, the testing software directory can be upgraded and kept up-to-date. Thus, even though one of the main criticisms against test automation is the expense, one has to realize that automation software is a long lasting, reusable product which can justify its cost.

Consistency

Test automation provides a consistent platform for your testing needs. The tests for which automation is usually deployed are extremely tedious. Automation drastically reduces the margin of error in the testing scenario by going through pre-recorded instructions. Regression tests verify whether the pre-existing functionalities are suited for new versions, which is critical when new development in the existing software takes place. This novel consistency provides a much needed reliability for your testing protocols.

Unique Programmability and Shelf Life

Not only can  test automation software be built to exact testing specifications, it serves as a prime component for future testing scenarios. In-house automated software developed by testing firms are modelled such that they have enough flexibility to handle a unique product, while complying with the latest security and testing protocols. This makes  test automation a powerful tool for time-saving, resourceful and top notch results.

User Environment Simulation

One unique way in which testing automation affects the testing procedure is through simulation of a typical user environment through categorically deployed mouse clicks and keystrokes. GUI testing is one of the most time consuming and redundant procedures because the tester has to deploy the same procedures in mock user driven environments and check for issues in the responsiveness of the GUI. With automated testing this process becomes incredibly easy.

Get in touch with OptimusQA for more information regarding automated testing. As we saw above, automated testing can make things in the testing lab very easy and tester friendly. At the same time, it gives the software creator a unique advantage of quick, hassle free testing and a great way to save precious resources and stay ahead of the timeline.

Choosing Between Automated and Manual Testing

Finding the right balance between manual and automated testing is a regular challenge in any software development life cycle. There is a place for each in any software project, although internal bias and habits may color which one receives emphasis. That is why an objective evaluation of the merits and disadvantages of each approach is important.

Situations Where Manual Testing Is Appropriate

There are three fundamental scenarios where manual testing is most effective:

  • Manual testing is typically the most effective approach during early code development. Developers make quickie unit tests, while testers begin familiarizing themselves with the software through manual functional tests. In both cases, the fluidity of the code precludes investing in test script overhead. As the code matures, however, opportunities for automating both unit and functional tests emerge.
  • As the code base stabilizes and more components are integrated, manual testing continues to provide an efficient method for exploration of features, the UI and workflow. These facilitate differentiation of higher and lower risk areas from which the prioritization of subsequent automation efforts is made.
  • When the UI is well established but still in flux, manual testing is the only way to go. Automating UI tests even on a stable system requires significant effort. Plus, the overhead of maintaining scripts is prohibitively expensive until the UI is nearly done.

Where Automation Is Most Appropriate

There are many opportunities during the software development life cycle where test automation should be considered as the most efficient method:

  • At any point where testers find their testing efforts are becoming excessively repetitive, these tests should be automated. Some test frameworks allow manual tests to be recorded into scripts, which eases the conversion process. However, there must be periodic culling of low-value scripts to limit undue maintenance overhead.
  • Daily build regression testing should always be automated as much as possible. Execution of such tests should be simple enough that both developers and testers can run them.
  • Error prone tests should be automated in the pursuit of consistency. This avoids many wasted hours of tester and developer time tracking down phantom bugs. This principle is especially important when creating testing benchmarks during performance and load testing.
  • Every opportunity to run tests in parallel automatically should be aggressively pursued. It is far less expensive to employ additional virtual or physical test servers than hiring more manual testers to speed up test cycles. Parallelization is essential for tests that simulate hundreds or thousands of simultaneous users too.
  • Automation is the appropriate approach when the software has a large set of possible configurations that do not differ significantly between each other and these configuration settings can be automated.
  • As much automation of all test phases is critical if the organization is employing continuous delivery methodologies due to the inherent short end-to-end test cycles.

Limitations of Automation

Despite the many opportunities for test automation within any software development project, there are limitations to its use that may muddle whether or not manual testing might be a better alternative:

  • The ease with which test frameworks can generate test scripts can create excessive execution and maintenance overhead if script value criteria are not in place.
  • Test scripts are essentially software and the same principles guiding software development, such as modularization, reusability, testability and KISS should be borne in mind.
  • Automation has significant upfront costs. If the organization is not committed to acquiring skilled automation resources or is not planning on using it for future projects, then it is probably wise to defer its use.
  • It cannot be effectively used in situations where human cognitive skills are superior in producing defects or evaluating complex behavior.

Besides the guidelines outlined above for determining when and where a manual or automatic approach to testing is most appropriate, each organization must evaluate its own capabilities to support either method. Clearly, both manual and automated testing have their uses and the trick is in finding the correct balance of each in achieving the software organization’s quality goals.

Care must be taken, however, to avoid influencing the decision based on the current status quo. Employing either type of testing simply because it is the way things have been done is not a basis for evaluating the way forward.

Test Automation: Myths and Facts

Automation is an area of software testing encompassing a multitude of techniques and tools with potential application throughout the software development process. Some organizations with automation experience may hesitate to expand its use due to concerns regarding costs or a perceived lack of relevant skills. Others may be overly gung-ho to automate their development processes end-to-end based on unrealistic expectations. In both cases, certain “myths” of test automation must be addressed in order for development and testing teams to correctly apply it in the most cost-effective manner.

Myths That Impede Expanding Automation

Test Automation Costs Too Much

Taken at face value, this myth has a kernel of truth within it. Automation has significant upfront costs that are difficult to avoid. Proprietary frameworks require large licensing fees. Open source tools require consultants or in-house expertise to realize their full value. On the other hand, can you afford the costs of less competitive software quality, longer release cycles or missed customer requirements? Those can far exceed the price of automation.

Realistically, to extract value from test automation, a number of things are required:

  • An automation architect to match the software testing requirements against the capabilities of automation tool suites.
  • Automation engineers who prioritize test automation to specific test areas that save time and improve quality.
  • Plans to reuse automation for a string of future projects to profit from the initial investment.

We Have to Replace Manual Testers

Related to this belief is a myth that test automation completely replaces manual testing, which we will get to later. Suffice it to say that there is always a need for manual testing.

As agile development methodologies and continuous delivery processes grow in use, automation frameworks have responded by accommodating the collaboration of both technical and non-technical staff test development and execution. For example, Selenium IDE allows non-coders to build basic but valuable test automation. Selenium IDE scripts can be exported to Selenium WebDriver or other higher-level tools for further enhancement and expansion within the grasp of retrained manual testers.

Furthermore, as an organization’s commitment to test automation increases, two things happen:

  1. The development/testing culture spontaneously institutes a virtuous feedback cycle that heightens awareness of additional automation opportunities.
  2. Test resources bogged down in repetitive, manual testing activities are redeployed higher-value testing activities that actually increase software quality.

Myths That Automation Is a Panacea

All Tests Will Be Automated

Test automation has its limits. Ignoring those limits wastes time and political collateral. Automation works best with highly repetitive or highly parallel test scenarios, such as build regression testing or load testing with large data sets or multiple users. It cannot test for user experience or for unpredictable results that require human cognitive skills to assess correctness.

Automating UI testing is another area fraught with complications and ineffectiveness due to the fragility in under-development UIs and the high cost of test maintenance and execution. It is especially ineffective when the test target is code beneath the UI.

Faster Software Release Cycles

Expecting test automation to speed up development/test cycles misses a key point. The goal of automation should be improving software quality rather than push software out the door faster. Properly applied, automation accomplishes deeper and broader code coverage. More defects are uncovered and in a more timely manner.

The closer to actual code writing that test automation is implemented, the more efficient is this effect. In an agile development environment practicing continuous delivery, end-to-end automation does indeed speed up delivery.

Automation Pays for Itself Quickly

It can be true that tactically applied test automation, such as automated developer unit testing within the IDE, pays for itself relatively quickly. Otherwise, automation must be viewed as a long-term commitment. It takes meaningful investments of time and money as the organization learns to use it in the most effective manner. Expect 10 to 20 software releases before fully recovering that investment.

A reluctance to adopt or expand test automation capabilities is often rooted in beliefs that automation is too costly and requires wholesale replacement of manual tests and testers. In fact, the costs of not automating are higher, manual testing is still required and personnel are redirected to higher-value tasks.

A wholesale adoption of automation, on the other hand, may be based on unrealistic expectations of what can and cannot be automated, an expectation that release cycles will shorten or that costs will be recouped quickly.

In either case, a careful evaluation of the advantages and disadvantages of test automation, appropriate tool selection and its effective application will dispel these myths. A deeper understanding of automation and long-term commitment to its use reap the greatest rewards.

Best Practices for Structuring Automated Tests

Test automation is an essential ingredient in any software organization’s overall testing strategy. Applied to code requiring frequent or repetitive testing, it offers increased efficiency, better test coverage and the opportunity to significantly improve software quality.

To extract maximum value from automation, it must be applied and managed wisely. Start by being smart about structuring your automated tests, their execution and maintenance by instilling tried and true best practices.

“Pyramid” Your Test Automation Priorities

Never will there be a dearth of opportunities for test automation, but all such opportunities are not equal in their contribution to improving software quality. Definite targets are manual tests that are data-driven, repetitive, require long wait times or are prone to human error. However, inserting automation here and there could be more disruptive than helpful.

A more fruitful approach is to prioritize your automation efforts in an architectural style alá Mike Cohn’s Test Pyramid. This technique places greater emphasis on unit testing than on end-to-end or GUI testing. It de-emphasizes testing through the GUI, which is time-consuming, fragile and notoriously difficult to automate.

Although some of that difficulty is mitigated by using a headless browser, Cohn’s Test Pyramid concept argues that greater testing value and less thrash is achieved via a larger portfolio of unit and service layer tests below the GUI. Logically, this implies that any GUI test failure means the test suite is missing one or more unit tests. Thus, from an automation point of view, automating unit tests should be a top priority.

Start Automating by Performing Manual Tests First

Continue to write test cases manually following best practices such as explicitly identifying pre-conditions, test data, step-by-step instructions and expected results. Every test case should be independent of others. The test case is then validated by running it manually and revising it as necessary before submitting it for automation.

This practice has two benefits. First, assuming a skilled test script translation is made, it shakes out any “bugs” in the test case and its execution, which are more difficult to correct once the test is automated. Secondly, experience shows that up to 50 percent of defects in the software are found at this early stage than during later automation runs.

Set Effective Time Targets

Following the ide of the Test Pyramid, the bulk of your automation should be centered on unit tests. If you have ruthlessly modularized unit tests, then they are quick to debug and quick to run, especially if you take advantage of parallelizing their execution.

A further efficiency can be had if tests are organized in suites corresponding to software components for which a developer team or individual is responsible. Then, at each check-in, the developers themselves can run the tests.

Especially if agile development is employed, the entire unit test suite should run in 10 minutes or less. Short run times are crucial if your developers are scattered across time zones to give the responsible developer time to correct a defect before he or she leaves the plant and passes the code to other regions.

Integration tests tend to run slower, but if their scope is limited to the interfaces between internal and external system components their run times should be under 20 minutes. Grab any opportunity for parallel execution where possible.

Acceptance or End-to-End tests may take even longer, but there should be far fewer of these. Also, they are run less frequently. Any excuse to replace an E2E test with one or more unit tests should be seized however.

Follow Software Development Best Practices

A test environment that embraces automation must be structured according to code development best practices. Test cases act as functional requirements. Test script development follows smart coding techniques. Continual maintenance is performed with regard to test suites and the framework in which they operate.

Script reviews should be performed for both test case documents and the test scripts or programs that derive from those. Tests should be highly modularized to facilitate debugging, framework refactoring and test suite maintenance.

Regular audits, supported by automated test metrics such as coverage, weed out ineffective tests and designate upgrades based on the most recent code status. All documents, test data, scripts and results must be under version control.

While test automation brings many benefits, it has significant up-front costs. Thus, it behooves any test organization to plan its automation strategy thoroughly. When targeting which tests to automate, consider re-ordering priorities with regard to the classes of tests that are most valuable in a context of automation. Continue to use manual tests to validate test cases, reduce testing times to under 20 minutes and institute techniques used in software development to improve test quality and ease maintenance.

Tips You Should Read Before Automating Your Testing

Many organizations unfamiliar with test automation view it as a panacea to missed deadlines and poor software quality. It is neither. It can save significant time and resources, especially when applied to repetitive testing tasks. It does improve an organization’s ability to find more defects, find them quickly and find them earlier in the development process both directly and indirectly.

However, unless an enterprise first considers its goals for automation and the costs of acquisition and maintenance, automation could create false hopes and be unable to fix underlying problems in the organization’s testing environment.

These tips will help you assess your automation readiness and gain the highest benefit from its application.

Plan Ahead

Not all areas of testing benefit effectively from automation. Choose carefully where you apply it. In general, automation offers the biggest payback when used for tests that are frequent, repetitive, and require minimal human intervention:

  • Unit testing for new code as it is being developed
  • Regression testing at each build
  • Functional testing which can take advantage of large, varied input sets
  • Performance, load and stress testing where test configurations differ slightly between runs
  • Large, data-driven tests that require long processing time.

Prioritize among these activities to ascertain where automation gives the biggest bang for the buck with the least risk. Unit testing and regression testing are good places to start.

Prioritize Test Execution

As you target areas for automation, categorize tests according to various criteria pertaining to their value:

  • The type of test being performed, e.g. white box, black box, load, etc.
  • The stability of the software area being tested
  • The amount of code that is covered by the test
  • The degree to which the tests focus on 3rd-party APIs
  • The amount of risk, such as cost of failure, the area under test poses

Prioritize test execution according to these criteria so that the most valuable tests always run first in sequence. Keep in mind that as development progresses, the value of an individual test is subject to change.

Develop Test Maintenance Procedures

Automation in no way relieves anyone of test environment upkeep duties. If ill-managed, test automation can create more problems than it solves. Therefore, it is important to implement basic maintenance procedures:

  • Keep individual tests simple and separated to increase their reusability and decrease debugging time.
  • Avoid spending too much time developing detailed tests for software components that have not stabilized yet.
  • Segregate tests that are outputting inconsistent results, which requires too much manual analysis. Either the tests are bad or the area under test is too unstable.
  • Periodically, review all tests to assess their value. Eliminate redundant ones, those no longer applicable to the current state of the software and those that are rarely executed. Update those that are ill-documented, incomplete or provide false positives.

Limits and Risks of Automation

Recognizing the limits of automation helps to apply it more effectively:

  • You cannot automate user experience testing to any significant degree.
  • They are not a replacement for manual tests that depend on tester intuition.
  • When requirements and code are in high flux, it may lose value because of high maintenance costs. This should not be the case in an agile environment however.
  • Its use can raise unrealistic expectations that it will fix a broken test environment, especially one that has a low resistance to disruption.

There are risks and costs to consider also:

  • It can create a false sense of confidence in your test coverage that, ironically, leads to missed defects.
  • Automation often requires skills that existing resources do not possess, so there are potential redeployment or training costs.
  • Automation tools are not cheap and neither is test script development.
  • Unless there is widespread buy-in for automation in the enterprise, there will be an uphill battle to prove its value.

Test automation has enormous potential value if applied and maintained correctly. It can meaningfully improve overall software quality through earlier defect detection and running repetitive, data-driven tests, which permits manual testers to apply their talents on higher-value tasks.

As organizations become more adept with test automation, they also benefit from the ability to create visible artifacts such as graphs and reports to gain defect tracking insights that indicate the quality progress of the software.

Clearly, there are limits to what can be automated and what test automation can accomplish, but those limits continue to recede in the face of new development methodologies such as agile and with new advances in the technology. It pays any organization who has not yet adopted test automation to give it a try after weighing all the pros and cons.

Creating a Successful Mobile Test Automation Strategy

Mobile software testing can quickly get out of hand if cross-platform software that accounts for multiple carriers, networks, phone models and platform configurations is being developed. Complications include deferring conformance, compatibility, performance, stress and load testing toward the end of the development cycle, which can create a disheartening situation for even the most experienced testing professionals.

A comprehensive mobile test automation strategy applied under the right conditions, however, can ensure that a quality mobile software product will be delivered on time.

Test Automation’s Fundamental Value

The first step in any automation test strategy is normalizing expectations within the organization relative to the value automation provides. Automation fundamentally expands test coverage and facilitates allocation of test resources to the highest value tasks. It enables both broader and deeper testing.

What and What Not to Automate

The next strategy task is deciding where to apply automation most effectively. Choose tests that have the highest return for the effort in automating them:

  • Tests that are time-consuming or error-prone when performed manually such as data-driven tests, repetitive tests with minimal input variance and tests with significant waiting time between results
  • High frequency tests such as daily regression verification
  • Suites of test cases applied to numerous platform combinations with minimal baseline configuration changes
  • Overnight load/stress tests that log results and restart automatically without human intervention
  • UI tests that can be scripted from initial manual tests

Besides identifying test classes that benefit from automation, each test candidate can be further prioritized according to additional criteria:

  • The degree to which a test’s automation frees up test resources for higher value activities
  • The degree to which automating a test increases test coverage
  • The sensitivity of tests to source code or test case changes
  • Test scaling efficiency as test cases or platform/OS/network combinations expand

The overall goal is to significantly narrow the scope of automation, which leads to higher effectiveness and success.

Actual Versus Virtual Mobile Test Automation

For highest effectiveness, a mobile test automation strategy should apply a staged application of automated test tools and methods.

Unit testing and early functional testing can be automated via device emulators within development and test teams. Once tests show stability, actual devices are added to the mix. If your mobile software targets the two mobile OS leaders, test on 7-10 actual iOS devices and about twice that for Android.

Expand actual device tests via automation tools simulating cellular network behavior over WiFi and cloud-based test services with access to additional real-world devices and real carrier networks. Additionally, select cloud-based mobile test frameworks with which automated UI scripts can be created from initial manual tests.

When deciding on the mix of actual and virtual devices consider that actual devices cost around $500 each, whereas cloud access to real devices is about $25 per hour. Aim for gaining complete proficiency on a few virtual devices before adding more in order to reduce the hourly cost.

Define Automated Test Tool Criteria

Evaluate test automation tools according to capabilities and how well they integrate with current tools. Ensure test reports are customizable and easily shareable. Test the ease with which the framework integrates with your test management process, build systems, revision control processes and bug-tracking tools. The best systems will support agile methodologies by integrating with developer SDKs as well.

Where Test Automation Strategies Fail

The best mobile test automation strategy can falter due to numerous obstacles:

  • Unrealistic expectations for up-front costs and effort to automate lead to short-term disappointment
  • Developer/tester silos hamper test automation integration and maintenance
  • Automating too much, too fast leads to frustration
  • Inadequate test automation tools or services severely limit automation benefits
  • A lack of advanced skills among testers and failure to designate dedicated automation personnel
  • The expectation that automation can fix a broken test/QA process

Organizations that tend to be successful with automation have created skilled test teams with peer status relative to developers, have developed management support/visibility and have made the effort to define the roles of automation engineer and architect. These organizations typically experience the highest ROI via automation.

Test automation is not a panacea for the increasingly competitive and complex environment in which mobile software testing must deliver the highest quality possible within rapid development cycles. Furthermore, it can never be a 100 percent replacement for manual testing. However, automation remains an essential ingredient for success.

Mobile test automation strategy must carefully consider the scope of automation, the selection of enabling tools and services plus the ease with which it integrates with current in-house processes and resources in order to achieve the greatest benefits. A well-thought out analysis followed by staged, scalable application of the strategy usually works best.

10 Tips to Get Started with Automated Testing

Deeper and wider test coverage enabled by test automation increases testing’s value and leads to improved software product quality. However, test automation is often misunderstood or misapplied, which can create more problems than it solves. To ensure that introducing test automation to your software development process goes smoothly take note of the following tips.

1 – Identify Tests That Benefit Most from Automation

  • Repetitive tests such as regression tests
  • Tests subject to human error such as manual interface tests
  • Data-driven tests that utilize multiple data sets
  • Configuration tests run over many OS or hardware platforms
  • Testing load and performance by a large number of simultaneous users

2 – Avoid Situations Where Test Automation Is Least Effective

  • When project requirements are still in flux, which leads to frequent re-coding
  • User experience testing except to the extent that test platforms are automatically reset
  • UI testing where UI component identification is not standardized
  • Unit testing when the test framework does not integrate with developer IDEs
  • When the value of tests in terms of defect risk or reusability cannot be measured

3 – Avoid Mixing Manual and Automated Test Teams

Test automation requires some specific skills not required of manual testers. Effective test automation teams often require a skill level equal to that of the developers. Oversight by a test automation architect is also invaluable.

4 – Do Not Use Automation to Speed up Product Delivery

Automation allows tests to be run faster, but the saved time should be directed toward more thorough testing for high risk or high value defects. Rarely does automation improve time-to-market for a given set of development resources, but it does improve the software’s stability, reliability, usefulness and value.

5 – Do Not Force Feed Automation

Especially if your software development process is new to test automation or uses it in a limited way, apply it gradually or to small projects to work out the kinks. If developers and testers perceive that test automation is being hammered into place, it is likely to receive substantial negative feedback and its effectiveness will be undermined.

6 – Automate Beyond the Test Team

Once the value of test automation is understood, additional automation opportunities should be identified, especially within development teams. For instance, a test framework may enable unit testing directly within existing developer IDEs that ties into test teams’ defect reporting systems. Opportunities to automate collaboration between testers and coders should be capitalized upon too.

6 – Automation Does Not Fix Poor Test Design

Test design must take into consideration all the areas of testing to be performed, but it should also identify high risk areas or other specific areas where test automation would add the most value rather than leaving such decisions to be made ad hoc once development is in later stages.

7 – Do Not Let Automation Create More Work Than It Saves

If you over-automate or fail to maintain automation processes, then you create unnecessary overhead. Tests become irrelevant over time as software requirements evolve, features are deprecated or platforms become obsolete. Test value is something that needs to be regularly evaluated and used to purge ineffective test suites.

8 – Automate When You Are Bored

Anytime a tester has mastered a suite of manual tests to the point where he or she begins daydreaming at the console, there is very likely a test automation opportunity. That opportunity to automate often falls victim to short-term demands on time. Thus, test teams must be instilled with a cultural expectation for them to prioritize automation versus repetition.

9 – Account for the Costs of Automation

Sometimes, test automation is only applied for automation’s sake. There may be tests that are amenable to automation, but their expiration date, scope or value may not justify the amount of time or resources required to automate them. In short, the cost of setting up the automation might be equal to or greater than what it costs to perform the tests manually.

10 – Automation Cannot Fix a Broken QA Process

When an organization’s QA/Testing organization has trouble meeting concrete goals, does not have adequate staffing or skill or fails to follow best practices, there is nothing that automation can do to correct these deficiencies. In such environments, it is more likely that applying automation will simply arrive at a bad result faster.

 

Before putting effort into test automation, be sure that you have identified the quantitative, measurable results you hope to achieve and that the deployment effort is understood and acceptable to both testers and developers. Clearly comprehend what types of testing benefit most from automation and instill a mindset that seeks out any area that can benefit from automation based on that comprehension.

Best Practices for Test Automation

Everyone involved in the development of software shares similar goals, which are to improve the functionality and reliability of the product in order to increase customer satisfaction. Since these goals are constrained by limits of time and money, this leads many to assume that automation is applied to shorten the development cycle. In fact, its primary purpose is to improve the quality and reliability of the final product. Employing a few best practices regarding automation make this point clear and increase its effectiveness.

Outline Test Automation Advantages

Test automation works most effectively with certain kinds of testing:

  • Test that are performed frequently, such as daily build regression testing
  • Repetitive testing with minimal human intervention
  • Data-driven tests that are difficult to perform manually
  • Baseline tests applied to a large set of platforms or configurations
  • UI tests derived automatically from manual tests

Identify Test Automation Limitations

However, automation does have limitations:

  • Automation can create excessive, low-value tests that increase maintenance costs.
  • User experience testing is not amenable to automation.
  • Automation tools cannot evaluate individual test value nor the quality of the test input data.
  • Automation is less effective when requirements and code are changing rapidly.
  • Test automation does not fix a broken QA process. In fact, it may exacerbate such a problem.

Plan for Automation

Knowing which tests can be most effectively automated and where the limitations to automation lie will be reflected in the project test plan. The plan sets forth the goals of testing and identifies specific benefits from automating certain areas:

  • Speeding up unit testing by developers
  • More frequent testing throughout the development cycle
  • Improving test coverage
  • Improving visibility into test results, defect detection, defect tracking and repair
  • Streamlining the generation of tests cases and test scripts

Quantifying these testing benefits provides a basis for evaluation of test automation tools or frameworks most appropriate to the project.

Evaluate Automation Tools against Testing Goals

If test automation is new to a team or new tools are being introduced, initially apply their use to well-scoped areas of the project or smaller, separate projects to evaluate their use.

Evaluate carefully basic automation tool capabilities:

  • The kinds of tests the tool handles such as regression tests, functional tests, UI tests, etc.
  • Its ability to generate test cases and scripts with minimal programming skills
  • The types of data sources with which it can interface
  • Its ability to record manual tests and generate scripts from logs
  • Report generation and analytical capabilities
  • Level of integration with developer IDEs

Frameworks or automation tools that integrate well with existing developer tools and the style of development process your organization uses will be the most effective and cause the least disruption.

Although it is important to choose a tool that can grow with the organization in terms of future capabilities and platform support, resist buying into expensive features for which there is currently no use.

Organize Automated Testing Efforts

Testers’ activities should be divided according to their experience with developing test scripts. With the best automation tools, non-programmers among the team can create useful script suites. Testers with script skills can be employed in developing more complex or detailed test scripts or fine-tuning generated scripts. At least one engineer who specializes in test automation tools and technique should oversee all automated testing efforts.

Automated tests should be organized according to the type of testing being performed, such as black box, white box, stress and so on, but also ranked by value criteria that prioritize their application. Such criteria include whether a test is being applied to a high risk area, a particular API or the integration of particular components. In this way, tests are organized such that the most valuable tests are run first in each test cycle. The earlier that high value defects are found, the more quickly and efficiently they can be repaired by the development team.

It is productive to further identify tests that have a high reusability factor. These tests have additional value in that they can be used for quickly developing additional tests for the current or future projects or be used by new, less skilled testers.

 

Getting off on the right track when applying test automation by following fundamental best practices improves its overall acceptance, effectiveness and contributes to a product with higher customer satisfaction. It must be kept in mind that test automation is not applied to shorten time-to-market, however. Its greatest value is in improving test coverage, identifying high value defects and making the best use of limited testing resources.

Using Test Automation For More Than Regression Test Cases

Many people understand by test automation the process of automating regression test cases for a specific application. This is probably the most popular use of test automation. To automate test cases that:

  • are executed often
  • are not very interesting
  • take a lot of time.

The benefit for the tester is that he will have more time for more complex testing. But can test automation be used for more than automating regression test cases? Lets take a few examples:

1. We have an application that can be used by its users in many languages. The users select their preferred language on the application’s home page. In this case, test automation is beneficial not only for verifying that the most important user scenarios work in different languages. A second benefit comes from using automation for checking if all translations in the user language are correct.

Test automation can help with language testing.

2. Some applications are highly customized and they have apparently a setting available for everything. It takes a long time to verify that the application works correctly for the default settings values. To verify that the application still behaves normally for non-default settings will take even longer. How about using test automation for:

  • changing the value of a setting
  • executing user scenarios from the happy path
  • confirming that the application works correctly

Test automation can help with configuration testing.

3. There are situations when applications crash randomly. Testers try to reproduce the crashes without results. One could suspect that a memory leak exists and that the application memory consumption goes up in time. Running test automation scripts continuously can prove if this is the case or not. I have done this successfully in 2 occasions so far.

Test automation can help with memory leak situations.

4. What if you deal with an application that implements complicated workflows for many user types? Lets assume also that each workflow type can be customized. Testing manually will take a very long time. Automating the workflow testing would bring a lot of time savings.

Test automation can help with workflow testing.

5. Finally, test automation cannot help with performance testing. But test automation can improve the performance testing in some cases. If you use JMETER for performance testing, it is possible to add Selenium WebDriver automation scripts to the load test plan. This allows you to see what the virtual users of the load test are actually doing.

So, test automation is not only for automating regression test cases. Much more can be done with it.

Guest post contributed by Alex Siminiuc (Test Automation For Manual Testers).

Want to get started with test automation? Read this Free tutorial today!