Regression Testing Mobile Apps

cross-platform-testing Regression Testing Mobile AppsMobile app development is fast paced. In the last few short years, smartphones have exploded in popularity and user’s expectations of their software have dramatically increased. Release cycles have shortened and if apps aren’t regularly improving, users will start to look for alternatives. With this comes a pain: regression testing. It’s fun to design, develop, and deploy new features but it’s important to make sure your new changes don’t break existing functionality, introduce new bugs, or reintroduce previously resolved bugs. This kind of work can be tedious and less exciting that working on the new stuff. That’s why we do a mix of manual and automated regression testing before any major release. After establishing a testing strategy, the regression testing can be optimized to get the most value out of the efforts. Specifically, there is rarely a need to test every possible combination of platforms, operating systems, and devices – so the strategy is to identify the high risk areas and focus on those. Those areas will uncover 90% of defects.

Manual Regression Testing VS Automated Regression Testing

Automating regression tests can be very beneficial. It will give your team the ability to move faster, run through huge amounts of test scripts/test data in short periods of times, and get a higher percentage of the functional components and edge cases tested. However, testing directly on mobile devices can be challenging, but there is low hanging fruit when it comes to automating the testing of the backend/API. API test automation is not challenging and can be setup using open source tools. This level of testing makes it possible to ensure that at least the backend is working as expected without having to first troubleshoot from a mobile device. It also opens up the door to performance and scalability testing. And when you do need to test directly on device, there are three good strategies:

  1. Build a device library (or rent from local providers) that consists of a good mix of devices. The library should have the latest and greatest as well as the worst devices being supported. Then automate your deployment as much as possible and capture crash logs remotely.
  2. Get remote access to devices and test either manually or automated using services like TestFlight and Perfecto. These are cumbersome to setup, but in theory give you a great opportunity to automate tests on a very wide array of devices all over the world.
  3. Work with a partner that provides testing services. Either they write their own test cases/plans or you provide them, but at the end of the day you send a build and they return test results. This is the quickest and most reliable way to get consistent test results. The quality here is determined by how invested you and the partner are in the relationship and how closely you work together. In an agile world where things move quickly, the partner needs to have complete clarity on what to test, what’s expected to work, and what is out of scope for testing; otherwise, known issues will be repeatedly reported.

The best approach to regression testing is often a mix of manual and automated, but keep in mind that the benefit of automation isn’t to reduce cost or effort, but to increase the scope of what’s being tested.

Using Test Automation For More Than Regression Test Cases

Many people understand by test automation the process of automating regression test cases for a specific application. This is probably the most popular use of test automation. To automate test cases that:

  • are executed often
  • are not very interesting
  • take a lot of time.

The benefit for the tester is that he will have more time for more complex testing. But can test automation be used for more than automating regression test cases? Lets take a few examples:

1. We have an application that can be used by its users in many languages. The users select their preferred language on the application’s home page. In this case, test automation is beneficial not only for verifying that the most important user scenarios work in different languages. A second benefit comes from using automation for checking if all translations in the user language are correct.

Test automation can help with language testing.

2. Some applications are highly customized and they have apparently a setting available for everything. It takes a long time to verify that the application works correctly for the default settings values. To verify that the application still behaves normally for non-default settings will take even longer. How about using test automation for:

  • changing the value of a setting
  • executing user scenarios from the happy path
  • confirming that the application works correctly

Test automation can help with configuration testing.

3. There are situations when applications crash randomly. Testers try to reproduce the crashes without results. One could suspect that a memory leak exists and that the application memory consumption goes up in time. Running test automation scripts continuously can prove if this is the case or not. I have done this successfully in 2 occasions so far.

Test automation can help with memory leak situations.

4. What if you deal with an application that implements complicated workflows for many user types? Lets assume also that each workflow type can be customized. Testing manually will take a very long time. Automating the workflow testing would bring a lot of time savings.

Test automation can help with workflow testing.

5. Finally, test automation cannot help with performance testing. But test automation can improve the performance testing in some cases. If you use JMETER for performance testing, it is possible to add Selenium WebDriver automation scripts to the load test plan. This allows you to see what the virtual users of the load test are actually doing.

So, test automation is not only for automating regression test cases. Much more can be done with it.

Guest post contributed by Alex Siminiuc (Test Automation For Manual Testers).

Want to get started with test automation? Read this Free tutorial today!