Even within the walled garden of enterprise workforce apps, top performance is essential for app adoption and increased user productivity. If you produce mobile apps for public consumption, performance expectations increase by an order of magnitude. Many an app has suffered premature death due to inadequate performance regardless of how well it met users’ functional needs.
Thus, application level performance testing is essential. Such testing must evaluate availability, response times and the timely and accurate completion of business logic transactions. Furthermore, performance must be evaluated within diverse combinations of platforms, OSs, connectivity options and back-end services.
Five Pieces of Advice for Mobile Performance Testing
1) Measure Early, Accurately and Often
Well before actual performance testing begins, some planning needs to happen. Stakeholders must develop a list of Key Performance Indicators against which performance test results will be measured. Start with metrics important from a user’s point of view and work downward to metrics within the app that support the user-level performance metrics.
Determine how metrics will be collected. Code instrumentation, measurement libraries, external tools or scripts are a few of the options. Next, plan on when and where data are to be collected. Finally, plan for how a performance baseline will be created, maintained and upgraded at all test stages up to production. It is wise to continue to update the performance baseline even after deployment.
2) Test under Diverse Connectivity Scenarios
Connectivity issues are a prime generator of performance-impacting issues such as hangs, lag, degradation or failed transactions, especially with regard to cellular carrier networks. You cannot count on the user correctly attributing such performance issues to the network and not your app, however.
Therefore, it is vital to perform connectivity testing over as many types and instances of network infrastructures as practically possible. Design tests such that elements including platform loading are isolated from connectivity factors to develop an accurate picture of app performance over any connection. Augment testing by using a variety of carriers and Internet backbones. To achieve this degree of variation may require employing cloud testing services.
3) Test the Backend with and without the App
If your mobile app utilizes backend services, understanding their performance is equally important as testing for connectivity issues. Fortunately, it is likely to be of smaller scope and less complex. Backend instrumentation should support both standalone stress tests on the backend infrastructure as well as tests on both the mobile device and backend during interactions. The backend test strategy must include load and stress tests with unrelated services present that compete for server and network resources.
4) Test for Extraordinary Success
Although you app’s customer requirements or backend SLAs may not demand it, it is wise to probe what happens on both the mobile app and the backend should user demands exceed expected maximum capacity. It does not happen frequently, but “viral” success of an app could end up leaving egg on the face of the organization should too much success lead to disappointment for excess users. They will not appreciate the irony either.
Testing for double or triple maximum capacity reveals the degree to which your app’s performance degradation is graceful. Often, this level of testing reveals defects that went undetected during less extreme conditions. This result alone could make such testing worthwhile.
5) Use an Actual Production Environment
Initial app performance testing necessarily takes place under constrained conditions as code development is in its early stages. Although initial tests are valuable in revealing flaws in assumptions or architecture and in contributing to the ongoing performance baseline, their results should be assessed cautiously. It is especially important not to extrapolate the figures to expected deployments.
At some point, however, app performance including functional, speed and reliability evaluations must be performed in an environment as nearly like production/deployment as possible. Variables to account for include hardware and OS configurations, platform resources, network connectivity and the use of production databases.
Take prudent steps to reduce the possibility of mangling your production system during such testing:
- Test during maintenance periods
- Utilize a fraction of the production framework
- Test in secret before the release’s public debut
- Test during off-peak periods
A survey by Equation Research revealed that well over half of mobile app users expect their apps to exhibit performance equal to that encountered on desktop PCs. Other surveys have consistently demonstrated that mobile users’ tolerance for sub-par performance is unquestionably low. Glitches and long or inconsistent wait times, for instance, quickly lead to deletion of the app and an increased reluctance to try other apps from the same vendor. Therefore, accurate performance testing is crucial. Thorough planning, execution and measurement all have critical roles in mobile app performance testing. Increased effort in this area leads to improved quality and heightened user satisfaction.