The project Eclipse IoT-Testware is delivering free open-source test tools and programs for the industry and companies developing Internet-of-Things (IoT) solutions. At TestCon Moscow 2019 Axel Rennoch, senior scientist at Fraunhofer FOKUS, spoke about quality assurance for IoT.
Today and in the future, IoT products and solutions will be omnipresent; they do appear in most of our daily environments at home, in industry, agriculture or traffic situations, said Rennoch. IoT solutions are generally characterized by openness, distribution, dynamics, scaling, and a long service life, argued Rennoch. IoT devices and services should be tested with a focus on conformance, interoperability, robustness, and security.
Rennoch argued that misuse or misbehavior following bad quality assurance or cyber attacks may cause serious implications on humans or our environment. He mentioned vulnerabilities, like Spectre and Meltdown in 2018, and weaknesses resulting from poor implementations of common IoT protocols that may accept invalid data values, as examples.
The testing industry and community has to deal with several messaging standards like MQTT, HTTP or CoAP and multiple IoT platforms. Rennoch mentioned that "we need some universal adaptable testing harness for the users and developers".
Rennoch presented the IoT-T project. He explained that this project has two major goals working with two different communities. The first goal is to deliver free open-source test tools and programs for the industry and companies to improve their own products and solutions. In addition, there is a need to establish a wide acceptable test catalog usable as a reference for any certification activities by commercial test labs or authorities.
The IOT-T project works together with the Eclipse open-source communities in the project Eclipse IoT-Testware to publish testware. For standardization activities and publications, it has created a new working group at the European Telecommunication Standards Institute (see ETSI TC MTS Testing WG) dedicated to addressing the areas in IoT testing that have not been covered elsewhere.
InfoQ spoke with Axel Rennoch after his talk at TestCon Moscow 2019.
InfoQ: What are the main testing challenges in the IoT world?
Rennoch: Following the IoT developers Survey around the developers and users within the Eclipse Foundation, the main concerns and therefore our challenges are security, data analytics, connectivity and interoperability. There is a need to check basic security requirements like session lock or password lifetime, but also different semantic interpretations. The later issues may appear if two IoT platforms using e.g. different data models (incompatible data formats) and representations (incompatible description languages) for IoT resources/devices do not match properly (see Interoperability in Internet of Things: Taxonomies and Open Challenges).
The test engineers need to apply knowledge of all known testing techniques and in particular complexity, asynchronism, resource constraints, and long operation phase We have to apply protocol testing, API software testing, system testing, security testing and performance testing. Due to economic reasons, it is necessary to introduce and use test automation as much as possible.
InfoQ: How can we deal with these IoT testing challenges?
Rennoch: Due to the openness, a large number of heterogeneous resources and potential vulnerabilities of IoT systems, such testing techniques, help to evaluate the interactions and to protect the public infrastructure from massive attacks.
QA has to consider advanced security techniques like Fuzzing testing. Furthermore, we need to apply interoperability testing and plugfest events involving heterogeneous IoT platforms to discover potential semantic issues.
These techniques are not new and do not appear with IoT for the first time. The novelty is that we need to apply multiple testing techniques during the whole lifecycle of the IoT solutions, in particular also after traditional acceptance testing, since IoT products will be in operation for many years in the field and will be subject for updates.
InfoQ: What has the IOT-T project delivered so far? What are the upcoming deliveries?
Rennoch: Today you may download and run the IoT-Testware from the Eclipse Foundation. It is already available for MQTT, CoAP and partly for OPC-UA. It includes a user-friendly dashboard to support the application without deeper knowledge of the execution runtime environment and comes in a docker compartment to avoid complicated installations.
At ETSI, we have related drafts for the standardized test purpose catalogs for MQTT, CoAP and in particular, security tests for LoRaWan and the industrial security requirements published by IEC 62443.