Forget for a moment the nascent Big Bang of the Internet of Things and focus on today’s explosion of Mobile Devices and Apps. Every day, thousands of new apps are uploaded, billions downloaded and they must run successfully on tablets, smartphones, vehicles, watches, kitchen appliances and a slew of new devices arriving at “ludicrous speed.”
What then of the plight of testing professionals who must cope with this incredibly diverse universe of devices, OSs, configurations, malleable user interfaces, paucity of platform resources and convoluted connectivity options? Furthermore, keep in mind that over half of companies report having inadequate mobile testing tools and insufficient device coverage that would allow them the luxury of feeling sanguine about their chances of satisfying users’ high standards.
Before hands are thrown in the air and everyone scrambles for the exits, consider these solutions to five major challenges in mobile testing.
5 Mobile Testing Challenges and Solutions
1) Device and Platform Diversity
There are today more mobile phones than people on the planet. iOS and Android dominate the OSs and five mobile device manufacturers cover half of the market with dozens of lesser device makers following close behind.
The easy, but expensive, way to confront platform diversity is off-loading it to an external lab, but most organizations instead choose a mix of internal mobile lab and cloud-based mobile testing services. Depending on shop size and the number of platforms supported, the ratio of in-house to cloud-based lab facilities will vary.
Cloud-based mobile testing is typically more efficient during late development stages when code is stable, performance and compatibility testing are the main areas of focus and access to a large mix of actual devices without the capital and maintenance expenses of in-house acquisition is required.
2) Network Diversity
Network carriers are even more fragmented than OS and device providers with a dozen companies sharing half of all connections and dozens more vying for the remainder. Testing must also account for this aspect of mobile testing diversity.
Network emulators can simulate cell network bandwidth and connectivity variances over Wi-Fi. These provide excellent mileage, but at some point testing on real networks becomes necessary. An interim solution is the use of device emulators plus an operator’s web or test proxy, which avoids airtime charges and lets testing proceed with an instrumented test stack.
Short of acquiring in-house accounts with target networks, cloud-based network test services provide the most realistic testing scenario. Many cloud-based device testing services include remote carrier coverage.
3) Mobile App Types
Another factor in mobile testing are the three approaches to mobile app architecture: native, Web/HTML5 and hybrid apps. Test case scenarios for each differ, especially regarding performance, stress, conformance and compatibility testing. Native apps have a reduced testing scope overall, whereas Web and hybrid apps require both on- and off-platform test cases and must account for complex connectivity and back-end issues.
Native and hybrid apps must be tested for successful download, execution, platform interaction and update behavior. Web-only apps depend less on the platform OS and more on the choice of browser and its versions of which all instances must be tested.
Since native, hybrid and web-based apps are all subject to other mobile testing challenges, the common solution for avoiding the additional complexity of supporting multiple app architectures is to eliminate at least one or both of the alternatives by executive decision.
4) Applying Test Automation
There are trade-offs between efficiency and coverage during various development and test phases when applying test automation. Early development functional and UI testing benefits most from frameworks that let developers emulate devices and facilitate their unit testing. Automation at later development phases must be selectively applied to achieve the highest ROI, since automation has substantial upfront costs of time and money.
Mobile testing automation typically has the highest payoff for data-driven, repetitive, frequent tests that require minimal human intervention and are relatively insensitive to code changes. If the tests chosen for automation can scale easily as platform diversity expands and be managed by less skilled testers, higher-skilled resources can be re-deployed to higher-valued test roles.
5) Finding Useful Test Tools
Mobile testing organizations are faced with an enormous diversity in test automation tools as well. Tool capabilities must be matched to the skill of test resources, the existing development tool set and the organization’s budget. Key features to acquire include the ability to build scripts from manual tests, which increases automated test coverage. Such an ability also increases collaboration between testers and developers during detect-verify-fix bug cycles for user errors.
Such solutions to the formidable challenges mobile app testers face should mitigate fears arising from the many and varied challenges that confront mobile testing teams and keep them moving forward.