Testing the Internet of Things

The precise definition of the Internet of Things varies according to the beholder, but in general constitutes a broad array of sensors, actuators, devices that communicate with their creators, users and other devices. More technically, any IP-addressable thing with a digital brain qualifies, although the smartest devices, i.e. PCs and smartphones, are typically not considered members of the IoT club on par with smart home devices, smart kitchen appliances, smart cars, smart cities, etc. Your kid’s teddy bear should be joining the IoT shortly, however.

Why IoT Testing is Different

Testers embedded in the world of PCs, servers and mobile apps will readily detect a substantial difference in IoT’s input/output model. Keyboards, displays, touchscreens and audio are decidedly minimal for IoT devices, if they exist at all. These devices might interact via motion or thermal detectors, cameras, and by monitoring and recording our behaviors such as when we open and close the refrigerator door. In fact, many IoT devices never interact directly with humans, which requires that their “correctness” is evaluated indirectly based on interactions within a system context that may include other IoT devices.

Less complex I/O and compute power of embedded devices reduces the number of functional and non-functional test cases when they are sitting on the workbench. However, a core concept of IoT is extreme connectivity, with interactions among a web of devices. This introduces a brittleness that may be outside the realm of many testers’ experiences.

Conformance, Interop and Security Testing Is Critical

To even come close to the heightened expectations for IoT, well-developed standards for interoperability and communication are vital. Otherwise, for example, communications protocol breakdowns between even a few IoT device types could create an enormous ripple effect depending on the roles of those devices and to how many other devices they are connected. Conformance and security testing in the IoT world is even more crucial than it is in the mobile device world today.

IoT Brings UX Testing to a New Level

The user experience for IoT-based services is potentially broad, subtle and complex. UI testing for a single device may be inconsequential, but IoT UX testing must encompass a larger context of intuitiveness, interactions, effects, expectations and satisfaction regarding the service in which the single device may play a small, but integral part. Today’s UX testing for PCs and mobile devices could appear trivial in comparison.

It is not difficult to imagine other non-critical side effects that are possible, even likely, when dealing with highly distributed interacting “things” with limited compute/storage/power resources, subject to asynchronous configuration changes or data delays going in or out of the device. Not the least of these are the possibilities for hacks and the release of private data.

For example, Philips IoT-enabled, LED lighting API allows Netflix to dynamically change ambient lighting in response to the brightness and color of an in-progress movie. Depending on a home’s layout, what other occupants are engaged in, etc., automatic lighting adjustments could be annoying or even disruptive. Such side-effect situations are within the domain of IoT UX testing.

It is easy to envision an in-home, elderly care medical monitoring service composed of IoT devices from disparate manufacturers, such as the Proteus ingestible pill sensor, a Metria™ wearable health monitor and a SmartThings home automation system going awry because one or more components silently failed an update or a communication protocol was subtly modified. Consequently, unintentional alerts to doctors or family members could disturb doctors and family and generate significant expense.

The Evolution of IoT Testing

Today’s IoT devices are easier to test due to their singularity of purpose, minimal connectivity and tentative expectations from human users. That situation is changing rapidly, however. IoT stakeholders, including test professionals, must raise awareness of the critical importance of standards and conformance if an IoT network effect is ever to get off the ground. In parallel, innovation in testing methodologies and tools must also sustain a sharp pace.

Fortunately, this is not altogether unfamiliar territory. Such complexity challenges have faced testers at every technology inflection point. The Internet evolved from static data repositories to an extremely dynamic, highly distributed, asynchronous system upon which “mashups” of complex services reside today. Improved standards, better tools, sophisticated testing methodologies and roll-up-your-sleeves innovation enabled test teams to keep up.

Highly connected, embedded IoT devices are consuming and generating data at an enormous rate. Harnessing and acting upon this information will create an array of new services heretofore unimaginable. To maintain the pace, test/QA teams must match this IoT innovation wave, especially with regard to evaluating human experiences in a context of services built upon countless devices with unprecedented connectivity.