Internet of Things (IoT) Testing: Challenges, Tools and Testing Approach

Testing Internet of Things (IoT):

When a question such as “What are the basic necessities of life” is thrown at people, most of them would answer, “Food, Shelter, Clothing”.

But that was the case a century ago. Human beings have evolved to develop a handful of extra necessities for living. We have evolved to make our life simpler, better, easier.

We have stopped using switches to control lights, have stopped paying at kiosks to submit toll charges, we have been monitoring our health status smartly, tracking vehicular movements more efficiently and there is a lot on the list.

How do we do it now? How is it so different? 

We need to understand these first after which we can learn how to test them.

IOT Testing

What is the Internet of Things (IoT)?

IoT is the connection of vehicles, home appliances, medicinal equipment using embedded electronics, microchips etc. to collect and exchange data of a different kind is called IoT. This technology allows the user to control devices remotely over a network.

IoT Examples

Some of the examples of IoT in real life implementation are as follows:

#1) In wearable tech:

Wearable gadgets such as Fitbit bands and Apple watches synchronize easily with mobile devices.

This helps in capturing necessary information such as health, heart rate monitoring, sleep activity etc. This also helps in displaying data and notifications from mobile devices onto them.

#2) Infrastructure and development:

With the use of an application such as CitySense, it is easier to get real time outdoor lighting data and based on these, the street lights are switched on or off. There are also various applications to control traffic signals and parking availability in a sophisticated city layout.

#3) Healthcare:

There are numerous applications to monitor the health conditions of the patients.

Based on the benchmarked data, the services control the dosage of medicine at different times in a day. There are applications such as UroSense which can monitor the fluid levels in the patient’s body and based on the need can initiate the fluid transfer. At the same time, the data can be transmitted wirelessly to various stakeholders.

Technology used in IoT

The following are a few of the most used technologies in IoT:

1. RFID [Radio Frequency Code] tags and EPC [Electronic Product Code]
2. NFC [Near Field Communication] is used to enable two-way interactions between electronic devices. This is basically for smartphones and is mostly used to make contactless payment transactions.
3. Bluetooth: This is used where short range communications are enough to get away with the problem. This is mostly used in wearable technologies.
4. Z-Wave: This is a low power RF comm technology. This is primarily used for home automation, lamp control, etc.
5. WiFi: This is the most commonly used choice for IoT. When on a LAN, this helps in transferring files, data and messages seamlessly.

Testing IoT

Let’s take an example of a medical healthcare tracking system in which the instrument monitors the health, heart rate, fluid intake details and sends out a report to the physicians. The data is recorded in the system and the historical data can be viewed whenever required.

Physicians can initiate drug intake and fluid supplements based on the data. This can be triggered remotely from any of the devices [computers or mobile devices] to which the medical device is connected to.

Now to test such an architecture, we need to go through multiple testing approaches such as;

IoT Test Approaches

#1) Usability:

• We need to make sure the usability of each of the devices used here.
• The medical healthcare tracking device used should be portable enough to be moved into different segments of the medical.
• The equipment should be smart enough to push not only the notifications but also the error messages, warnings etc.
• The system should have an option to log out of all events to provide clarity to the end user. If it is not capable of doing that, the system should push those as well to a database to store it.
• The notifications should be shown and handling of the display should be done properly in the devices [computers/mobile devices].
• Usability in terms of displaying data, processing data, pushing job tasks from the devices should be tested thoroughly.

=> Read more about general Usability testing here

#2) IoT Security:

• IoT Security challenges: IoT is data centric where all devices/systems connected operate based on the data that is available.
• When it comes to the data flow between devices, there is always a chance that the data can be accessed or read when getting transferred.
• From a testing standpoint, we need to check if the data is protected/encrypted when getting transferred from one device to the other.
• Wherever there is an UI, we need to make sure there is a password protection on it.

=> Read more about general security testing here

#3) Connectivity:

• As it is a healthcare solution, connectivity plays a vital role.
• The system should be available at all times and should have seamless connectivity with the stakeholders.
• As per connectivity, two things are very important to test;
  • Connectivity, transfer of data, and receiving job tasks from the devices should be seamless when the connection is UP and running.
  • The other condition is the connection down scenario. It doesn’t matter how robust the system and the network are, there is a chance that the system will go offline. Being a tester, we should test the offline conditions as well. Once the system is not available on the network, there has to be an alert which can prompt the physicians so that they can start to monitor the health conditions manually not depending on the system till it is up. On the other hand, there has to be a mechanism in the system which can store all the data in it during the offline period. Once the system comes online, all that data should get propagated. Data loss should not be there in any condition.

#4) Performance:

• When we are talking about a system for a healthcare domain, we need to make sure the system is scalable enough for the whole hospital.
• When the testing is carried out, it is done for 2-10 patients at a time and the data is propagated to 10-20 devices.
• When the whole hospital is connected and 180-200 patients are connected to the system, the data that is propagated is much bigger than the tested data.
• As testers, we need to make sure the system performs the same even though the added data is propagated.
• We should also test the monitoring utility to display the system usage, power usage, temperature etc.

#5) Compatibility Testing:

• Looking at the complex architecture of an IoT system, compatibility testing is a must.
• Testing items such as, multiple operating system versions, browser types and respective versions, generations of devices, communication modes [For e.g. Bluetooth 2.0, 3.0] is necessary for IoT compatibility testing.

=> Read more about general security testing here

#6) Pilot Testing:

• As far as the IoT is concerned, Pilot testing is a must.
• Only testing in the lab makes sure the product/system works fine. However, this may backfire badly when exposed to real-time conditions/steps/scenarios.
• During pilot testing, the system is exposed to a limited number of users in the real field. They use the application to give feedback on the system.
• These comments come in handy, making the application robust enough for the production deployment.

#7) Regulatory Testing:

• This being a healthcare system that needs to pass through multiple regulatory/compliance checkpoints.
• Think of a scenario where the product passes through all the testing steps but fails in the final compliance checklist [testing performed by regulatory body].
• It is best practice to meet the regulatory requirements at the start of the development cycle itself. The same should be done as part of the testing checklist.
• By doing so, we will make sure the product is certified as part of the regulatory checklist as well.

#8) Upgrade testing:

• IoT is a combination of multiple protocols, devices, operating systems, firmware, hardware, networking layers etc.
• When an upgrade is performed, be it for the system or for any of the involved items as stated above, thorough regression testing should be carried out/strategy should be adopted so as to overcome upgrade related issues.

IoT Testing Challenges

The challenges a tester faces in IoT are as follows:

#1) Hardware-Software Mesh

IoT is an architecture, which is closely coupled among various hardware and software components. It is not only the software applications that make the system but also the hardware ones, sensors, communication gateways etc. too play a vital role.

Only functional testing does not help in completely certifying the system. There is always a dependency on each other in terms of the environment, data transfer etc. So, it becomes a tedious job as compared to testing a generic system [only software/hardware component].

#2) Device Interaction module

As this is an architecture between different set(s) of hardware and software, it becomes mandatory that they talk to each other in real time/near real time. When they both integrate with each other, things such as security, backward compatibility, and upgrade issues become a challenge for the testing team.

#3) Real-time data testing

As we discussed earlier, Pilot testing/regulatory testing is mandatory for a system like this, it also becomes very tough to get such data.

Being on the testing team, getting regulatory checkpoints or getting the system deployed in the pilot is very tough. The step becomes even tougher if the system is related to Healthcare as per our example. That stays as a big challenge for the testing team.

#4) UI

The IoT is spread across devices belonging to every platform [iOS, Android, Windows, Linux]. Currently, testing that out on devices can be done but testing it on all possible devices is almost impossible.

We cannot omit the possibility of the UI being accessed from a device which we don’t possess or simulate. That’s a challenge which is tough to overcome.

#5) Network availability

Network connection plays a vital role as IoT is all about data being communicated at faster speeds all the time. IoT architecture has to be tested for all kinds of network connectivity/speeds.

To test this, virtual network simulators are mostly used to vary the network load, connectivity, stability etc. However, real time data/network is always a new scenario and the testing team doesn’t know where the bottleneck will develop in the long run.

IoT Testing Tools

There are various tools which are used during testing IoT systems.

They can be classified based on the target and are given as below:

#1) Software:

1. Wireshark: This is an open source application used to monitor traffic at the interface, source/destination host addresses etc.
2. tcpdump: This does a similar job to that of Wireshark except, this does not have a GUI. This is a command line based utility which helps the user in displaying the TCP/IP and other packets that are transmitted or received over a network.

#2) Hardware:

1. JTAG Dongle: This is similar to a debugger in PC applications. This helps in debugging the target platform code and showing variable step by step.
2. Digital Storage Oscilloscope: This is used to check various events with time stamps, glitches in power supply, and signal integrity checks.
3. Software Defined Radio: This is used to emulate receivers and transmitters for a large range of wireless gateways.

In the developing world around us, IoT is a growing market and has a lot of opportunities. It is not long before IoT becomes essential for the testers to survive in the developing world.

IoT-enabled gadgets, smart device applications, and communication modules play a vital role in studying and evaluating the performance and behavior of various IoT services.

Poor design of IoT-enabled devices and services can hamper the correct functioning of the application and in turn negatively affect the end-user experience.

To Summarize

IoT testing approach can be different based on the system/architecture involved. Testers should concentrate more on the Test-As-A-User [TAAS] approach rather than testing based on the requirements.

Another major player in IoT testing is Integration testing. IoT is successful if the Integration test plan is accurate and robust enough to catch flaws in the system.

IOT testing may be a tough/challenging job, but it is also very exciting as well for the testing team to certify such a complicated mesh of devices, protocols, hardware, operating systems, firmware etc.

About the author: This is a guest post by Subhasis. He is working as a team lead with over 8 years of corporate experience working for Fortune 500 IT companies in the field of Software Quality Assurance, Software Development and Testing experience.

UPDATE: We have added useful Infographics on IoT topics. Thanks to webeeky.com for sharing this with us.  

Post your IoT queries/comments below.