As the world decidedly transitions from desktops and laptops to mobile devices, application developers and testers face enormous challenges to ensure that their products meet basic metrics of functionality, performance and usability. These are requirements for desktop apps too, but they are compounded for mobile apps due to the immense variance in platform resources, input methods, screen sizes, operating systems, operating system versions, networks and application architectures on mobile devices.
Furthermore, end users, who now have access to more platform and application choices than ever before, are completely unforgiving of faults in mobile applications. Failures result in almost instantaneous loss of customers according to most studies. A disappointing mobile application experience causes a quarter of users to immediately delete the app. A third will instead use a competitor’s application and nearly half will be skeptical of trying other apps from that supplier.
Getting it right for your customers, whether they be the captive audience for your enterprise workforce apps or the public, requires a particular mindset in regards to planning, development, testing and support that may be unfamiliar to teams oriented toward PC software.
Approaches to mobile application testing require bearing in mind several considerations that ensure your company’s mobile app testing is on target and successful.
10 Key Considerations for Your Mobile Application Testing
1 –The Potential for Combinatorial Test Case Explosion
It is easy to be carried away with the idea that your next great app will conquer the mobile device world. In order to accomplish that, you must go live on at least Android and IOS and probably Windows 10 Mobile.
Supporting more than one OS presents a significant cross-platform development challenge, but the reality for testing is even more onerous. The testers must assess the application across devices from a dozen manufacturers, each with a portfolio of products with different screen sizes, platform capabilities and resource configurations.
If the app is HTML5 based, testing can use a standardized browser, but hybrid and native apps require platform-specific testing especially with regard to the UI and on-board sensors. There will be additional test cases for network interoperability and security as well.
Unless your organization already has deep experience in developing cross-platform mobile apps, you are better off reducing the scope of both development and testing to something manageable by choosing a single OS and a subset of manufacturers and devices supporting that OS in order to produce the greatest value to your business.
2 – Your Current Design-Code-Test-Support Methodology
Testing within an organizational structure that utilizes a throw-it-over-the-wall approach between developers and testers is not going to cut it in the fast-paced mobile app ecosystem. Agile development methods greatly reduce lag times between coding, testing, delivery, defect detection and defect remedy. Not only do such methods increase team efficiency and allow for faster time-to-market but they improve customer support and team cohesion too.
If your development processes are not agile, then your efforts to deal with the challenges of mobile testing can be easily doubled. Though the introduction of agile methodologies within ongoing projects is likely to be counter-productive, pilot projects should be started sooner rather than later to begin a gradual switchover to methods proven to be more effective. Be careful not to fall into the trap of simply accelerating waterfall processes and thinking you have improved agility.
3 – Real versus Virtual Device Testing
The bottom line when choosing whether to use real or emulated devices for testing comes down to the fact that all of your end users are going to be using real devices. Although it is impractical to use only real devices for all test phases, no application must be released without full testing on actual devices on which the app is intended to run.
Especially during earlier development phases, emulator testing is a must for efficiency reasons. Some IDEs include emulation capabilities, which is critical for unit testing and UI evaluation. In general, it is easier to automate test suites using emulated devices, though that is changing with the advent of cloud services aimed specifically at mobile device testing.
Cloud testing services allow both developers and testers to interact in real-time on live networks with nearly any real mobile device on the market. Tests are accomplished via online dashboards that realistically display the device UI. Tests can be developed manually and saved to a script for regression tests or to demonstrate a bug to the developers. As this technology continues to improve, the need for an organization to acquire, maintain and plug into a test harness a multitude of actual devices diminishes.
4 – Automation
Whether or not your teams use real or emulated devices, the need for test automation should be paramount. It is not possible to automate everything of course. Most automated tests will have first been run manually because they originated from a defect found by a user or because new test case suites are developed. To whatever extent possible, broad and deep automation is essential for an agile, continuous delivery organization if it is to keep up with the mobile app markets.
There are increasingly sophisticated tools for test automation available. The best support end-to-end testing from requirements to test cases to regression testing and defect testing. These can streamline the entire testing and support process. However, simply finding the right framework is only half the challenge.
Automation requires increased cultural awareness of its benefits and a willingness to seek out and implement automation wherever it is practically possible. An organization supports this mentality via team incentives and designating an Automation Engineer. The occupant of that role drives automation strategy, ensures automation tools and initiatives are used correctly and measures the effectiveness of automated processes toward producing frequent, timely, high-quality app releases.
5 – User Experience
The ultimate testing challenge may be testing User Experience, or UX. While the functionality of most mobile application UIs can be tested thoroughly by the best automation frameworks, even those fall short in the face of sophisticated interactions such as gestures, hand waves and eye movements. An even greater imposition is automated testing of interactions via intelligent agents such as Siri or Google Now. Throw in contextual information such as tilt, acceleration, geolocation and ambient sound and even the best test automation efforts will struggle.
Plan for UX testing with real users under lab conditions or by field beta testers. In addition to testing for smooth operation, the app’s workflow, responsiveness and aesthetics should be reviewed. Application instrumentation that tracks how users interact with the app often reveal surprising insights for the designers and developers.
Keep in mind overall that the test with the most value is to ensure that the user receives intended and valuable results from the application.