Testing the Internet of Things

“The Internet of things, will augment your brain” – Eric Schmidt

Why IoT is such a hot topic today?

Before we delve into details of testing of Internet of Things (IoT), it will be good to see why IoT is such a hot topic today. Gartner estimates that IoT product and service suppliers will generate incremental revenue exceeding $300 billion in 2020. It will not only impact the business, but our lives as well. Some of the futuristic quotes on IoT from different source. In brief, a big money is involved and business will follow.

Internet of Things Market Statistics – 2015
Some Bottom Lines on the Internet of Things (IoT)
6 Quotes that Predict the Future of the Internet of Things
The ‘Internet of Things’ Quotes to Consider

While the IoT is still taking shape, it’s already making incredible strides as a new frontier for the connected world in which people, devices, environments and virtual objects are all connected and capable of interaction but that brings new testing problems and complexity. Tiny computers are embedded in everything from light bulbs and refrigerators to school alarm bells. The number of these devices can explode in ten of billions in the next few years. Customers expect all these devices to simply work – and work well – with each other.

Why IoT testing is different? Challenges faced by testing team?

Many IoT devices never interact directly with humans, which requires that its “correctness” is evaluated indirectly based on interactions within the system context that may include other IoT devices. The added complexity comes with huge volume, velocity and variety of data generated across a smart and connected ecosystem. The QA needs to experience testing in a dynamic environment than defined environment. With several million sensors and different types of devices, mere functionality validation is not enough and covering all real time use cases could be a mammoth task. Setting up an environment to test real time IoT is another headache and expensive.

Testing the Internet of Things
The Internet of Things: QA Unleashed

IoT testing areas

The IoT’s key components are “Things”, “Communication” & “Computing”. QA organization can split test areas into two broad categories (1) The device interaction layer and (2) The user interaction layer. In device interaction layer, we need to check if different devices and sensors conforms with standards? Are different devices interoperable? With billion of sensors in the making, will data privacy and the security concerns be tackled? In User Interaction layer we need to check if Network capability is sufficient? Have user experience & usability is being considered? How about the functionality of IoT services and data analytics validation? Apart from these performance, compatibility and exploratory testing should be considered. QA should be developing a testing strategy to test the application in totality w/o having manufacturing hardware first.

The Internet of Things: QA Unleashed
Internet of Things (IoT) Testing – Challenges, Applications And Sample Test Cases

Solution and Framework Opportunities

So far we discussed on challenges in testing for IoT, but are there any solution and framework available to take care of testing of IoT? What are some of the solution and framework might require to verify and validate IoT system? Have you considered any protocol simulator? Do you have “Data Recorder” in place who can capture data from different devices? Do you have the capability to virtualize different IoT interfaces to simulate a dynamic test environment?

The Internet of Things: QA Unleashed
Functional Testing for IoT

Some existing solution who have tried to resolve IoT challenge is by developing simulators for MQTT (a light weigh messaging protocol for sensors) and CoAP (The Constrained Application protocol, a web transfer protocol for use with constrained nodes and constrained network), simulator for WebAPIs.

Internet Of Things Solutions

How can we conduct performance testing of IoT?

Performance testing is about simulating different communications in the system with large number of users. In IoT, different communication that needs simulation are device interacts with other devices (D2D), collected device data sent to server infrastructure (D2S) and then server infrastructure needs to share this device data to program, devices or other server (S2S)

Performance Testing 101: How to Approach the Internet of Things