Mobile app development is fast paced. In the last few short years, smartphones have exploded in popularity and user’s expectations of their software have dramatically increased. Release cycles have shortened and if apps aren’t regularly improving, users will start to look for alternatives. With this comes a pain: regression testing. It’s fun to design, develop, and deploy new features but it’s important to make sure your new changes don’t break existing functionality, introduce new bugs, or reintroduce previously resolved bugs. This kind of work can be tedious and less exciting that working on the new stuff. That’s why we do a mix of manual and automated regression testing before any major release. After establishing a testing strategy, the regression testing can be optimized to get the most value out of the efforts. Specifically, there is rarely a need to test every possible combination of platforms, operating systems, and devices – so the strategy is to identify the high risk areas and focus on those. Those areas will uncover 90% of defects.
Manual Regression Testing VS Automated Regression Testing
Automating regression tests can be very beneficial. It will give your team the ability to move faster, run through huge amounts of test scripts/test data in short periods of times, and get a higher percentage of the functional components and edge cases tested. However, testing directly on mobile devices can be challenging, but there is low hanging fruit when it comes to automating the testing of the backend/API. API test automation is not challenging and can be setup using open source tools. This level of testing makes it possible to ensure that at least the backend is working as expected without having to first troubleshoot from a mobile device. It also opens up the door to performance and scalability testing. And when you do need to test directly on device, there are three good strategies:
- Build a device library (or rent from local providers) that consists of a good mix of devices. The library should have the latest and greatest as well as the worst devices being supported. Then automate your deployment as much as possible and capture crash logs remotely.
- Get remote access to devices and test either manually or automated using services like TestFlight and Perfecto. These are cumbersome to setup, but in theory give you a great opportunity to automate tests on a very wide array of devices all over the world.
- Work with a partner that provides testing services. Either they write their own test cases/plans or you provide them, but at the end of the day you send a build and they return test results. This is the quickest and most reliable way to get consistent test results. The quality here is determined by how invested you and the partner are in the relationship and how closely you work together. In an agile world where things move quickly, the partner needs to have complete clarity on what to test, what’s expected to work, and what is out of scope for testing; otherwise, known issues will be repeatedly reported.
The best approach to regression testing is often a mix of manual and automated, but keep in mind that the benefit of automation isn’t to reduce cost or effort, but to increase the scope of what’s being tested.