For application programmers, APIs are the user-interfaces upon which they build their own APIs, services and applications. They have been in use almost since software programming began. Until recently, their main acceptance criteria were their ease of use and functionality. However, in today’s world, they are on the critical path for determining application end-user performance, usually measured by how promptly they respond to user actions and requests.
Thus, API testing must encompass more than correct functioning of the API in terms of inputs and outputs or whether it fails gracefully in the face of errors. It hardly matters to end-users if the functionality is right or wrong if they cannot access it in a reasonable amount of time. API performance under load, therefore, must be measured and fine-tuned to remove processing bottlenecks.
Ensure Functional Stability First
API functionality must be verified first before meaningful performance testing can progress. This includes evaluating the API and its documentation to ensure that API calls and their descriptions line up and are self-explanatory. Start with happy-path test scripts that take the documentation literally. These tests might be enhanced versions of developer unit tests.
Next, stress the functionality by exploring border conditions, passing random or missing parameter values, large amounts of data, non-ASCII character sets and so on. Working with code that has compile-time instrumentation built-in greatly reduces the amount of time spent tracking down bugs that appear as a result of these tests. Be sure to log all inputs and outputs during test runs also.
Performance Testing Preparation
Once functionality is stable and before designing performance, load or stress tests, be sure you know what you are testing for. Examine the software requirements to determine important real-life performance metrics the API performance expectations. Metrics to look for include request throughput, peak throughput, the distribution of throughput per API endpoint and the maximum number of concurrent users supported.
With these data in hand, start general load tests that progressively increase demands on the API. These may shake out bugs in the API as well as the test environment. They will provide early baselines of the maximum load the API can serve without breaking.
At this point, apply tests that more accurately reflect the expected real-world usage of the API. If an existing version of the API is already in use, API call frequency distributions are invaluable in determining the most effective use of your testing resources. Alternatively, utilize previous production logs for the API and feed these to automated testing tools to recreate realistic scenarios.
Popular API Testing Tools
It is often useful to take advantage of cloud services, such as AWS, to provide a testing infrastructure that can expand and shrink according to the demands of your performance tests. Along with that, there are many testing tools available to create an effective testing environment quickly of which we cover a few.
This is an open-source command-line tool whose main focus is to produce a steady request per second rate specified by the tester. It is simpler to use than many of the other testing tools mentioned here and is ideal for setting up performance baselines.
The free version of this cloud testing service from SendGrid permits up to 10K requests per second, which is a meaningful load for most APIs. It is simple to set up and produces informative reports.
This tool configures all tests via a command line interface. It is relatively easy to use for generating any target rate of HTTP requests you need. It is multi-threaded, so has a higher performance than other tools, but the reporting is non-graphical.
JMeter is probably the best-known open-source performance testing tool. It uses a full-featured GUI for creating detailed test plans. The number of execution threads, parameters for HTTP requests and listeners used to display results are some of the parameters that can be specified. Its downside is its complexity and steep learning curve.
API performance as an acceptance criterion has become equal in importance to API functionality and usability. This is due to increased expectations of end users who are intolerant of delays and have many similar application options to choose from in a growing market of applications.
Thus, it is critical that testing gives API performance its due as part of the overall test plan. Fortunately, there are clear best practices and many tools to assist in performance evaluation and to help developers and testers balance both performance and functionality of the APIs being produced.