ACM Digital Library home
ACM home
Advanced Search
research-article

PHiLIP on the HiL: Automated Multi-Platform OS Testing With External Reference Devices

ACM Transactions on Embedded Computing SystemsVolume 20Issue 5sArticle No.: 91pp 1–26https://doi.org/10.1145/3477040
Published:22 September 2021Publication History
  • 1citation
  • 113
  • Downloads
Metrics
Total Citations1
Total Downloads113
Last 12 Months44
Last 6 weeks12
Skip Abstract Section

Abstract

Developing an operating systems (OSs) for low-end embedded devices requires continuous adaptation to new hardware architectures and components, while serviceability of features needs to be assured for each individual platform under tight resource constraints. It is challenging to design a versatile and accurate heterogeneous test environment that is agile enough to cover a continuous evolution of the code base and platforms. This mission is even more challenging when organized in an agile open-source community process with many contributors such as for the RIOT OS. Hardware in the Loop (HiL) testing and Continuous Integration (CI) are automatable approaches to verify functionality, prevent regressions, and improve the overall quality at development speed in large community projects.

In this paper, we present PHiLIP (Primitive Hardware in the Loop Integration Product), an open-source external reference device together with tools that validate the system software while it controls hardware and interprets physical signals. Instead of focusing on a specific test setting, PHiLIP takes the approach of a tool-assisted agile HiL test process, designed for continuous evolution and deployment cycles. We explain its design, describe how it supports HiL tests, evaluate performance metrics, and report on practical experiences of employing PHiLIP in an automated CI test infrastructure. Our initial deployment comprises 22 unique platforms, each of which executes 98 peripheral tests every night. PHiLIP allows for easy extension of low-cost, adaptive testing infrastructures but serves testing techniques and tools to a much wider range of applications.

References

  1. ACM. Jan., 2017. Result and Artifact Review and Badging. http://acm.org/publications/policies/artifact-review-badging. (Jan., 2017).Google ScholarGoogle Scholar
  2. Cedric Adjih, Emmanuel Baccelli, Eric Fleury, Gaetan Harter, Nathalie Mitton, Thomas Noel, Roger Pissard-Gibollet, Frederic Saint-Marcel, Guillaume Schreiner, Julien Vandaele, and Thomas Watteyne. 2015. FIT IoT-LAB: A large scale open experimental IoT testbed. In 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT). 459–464.Google ScholarGoogle Scholar
  3. Bestoun S. Ahmed, Miroslav Bures, Karel Frajtak, and Tomas Cerny. 2019. Aspects of quality in internet of things (IoT) Solutions: A systematic mapping study. IEEE Access 7 (Jan. 2019), 13758–13780.Google ScholarGoogle Scholar
  4. Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, J. Jenny Li, and Hong Zhu. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (August 2013), 1978–2001.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. ARM Ltd.2020. Mbed OS. https://www.mbed.com, last accessed 07-17-2020. (2020).Google ScholarGoogle Scholar
  6. ARM Mbed. 2021. ARM Mbed OS Testing Tools Documentation. https://os.mbed.com/docs/mbed-os/v5.15/tools/testing.html, last accessed 13-07-2021. (2021).Google ScholarGoogle Scholar
  7. ARM Software. 2020. CMSIS System View Description. https://arm-software.github.io/CMSIS_5/SVD/html/svd_Format_pg.html, last accessed 01-01-2021. (2020).Google ScholarGoogle Scholar
  8. Emmanuel Baccelli, Cenk Gündogan, Oliver Hahm, Peter Kietzmann, Martine Lenders, Hauke Petersen, Kaspar Schleiser, Thomas C. Schmidt, and Matthias Wählisch. 2018. RIOT: An open source operating system for low-end embedded devices in the IoT. IEEE Internet of Things Journal 5, 6 (December 2018), 4428–4440. http://dx.doi.org/10.1109/JIOT.2018.2815038Google ScholarGoogle ScholarCross RefCross Ref
  9. Emmanuel Baccelli, Oliver Hahm, Mesut Günes, Matthias Wählisch, and Thomas C. Schmidt. 2013. RIOT OS: Towards an OS for the internet of things. In Proc. of the 32nd IEEE INFOCOM. Poster. IEEE Press, Piscataway, NJ, USA, 79–80.Google ScholarGoogle Scholar
  10. Miroslav Bures, Tomas Cerny, and Bestoun S. Ahmed. 2019. Internet of Things: Current challenges in the quality assurance and testing methods. In Proc. of International Conference on Information Science and Applications (LNEE), Kuinam J. Kim and Nakhoon Baek (Eds.), Vol. 514. Springer Singapore, Singapore, 625–634.Google ScholarGoogle Scholar
  11. Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, USA, 209224.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Cristian Cadar, Patrice Godefroid, Sarfraz Khurshid, Corina S. Păsăreanu, Koushik Sen, Nikolai Tillmann, and Willem Visser. 2011. Symbolic execution for software testing in practice: Preliminary assessment. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). ACM, New York, NY, USA, 10661071.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Cadence. 2021. All Products A-Z. https://www.cadence.com/en_US/home/tools/tools-a-z.html, last accessed 01-01-2021. (2021).Google ScholarGoogle Scholar
  14. Woei-Kae Chen, Chien-Hung Liu, William W.-Y. Liang, and Ming-Yi Tsai. 2018. ICAT: An IoT device compatibility testing tool. In Proc. of 25th Asia-Pacific Software Engineering Conference (APSEC). IEEE, NJ, USA, 668–672.Google ScholarGoogle ScholarCross RefCross Ref
  15. Kai Cong, Fei Xie, and Li Lei. 2013. Symbolic execution of virtual devices. In Proc. 13th Intern. Conference on Quality Software (QSIC’13). IEEE Computer Society, USA, 1–10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Mariela Cortés, Raphael Saraiva, Marcia Souza, Patricia Mello, and Pamella Soares. 2019. Adoption of software testing in internet of things: A systematic literature mapping. In Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing (SAST 2019). Association for Computing Machinery, New York, NY, USA, 3–11.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. João Pedro Dias, Flávio Couto, Ana C. R. Paiva, and Hugo Sereno Ferreira. 2018. A brief overview of existing tools for testing the internet-of-things. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 104–109.Google ScholarGoogle ScholarCross RefCross Ref
  18. dSPACE. 2021. dSPACE ECU Testing. https://www.dspace.com/de/gmb/home/products/systems/ecutest.cfm, last accessed 01-01-2021. (2021).Google ScholarGoogle Scholar
  19. Adam Dunkels, Björn Grönvall, and Thiemo Voigt. 2004. Contiki - A lightweight and flexible operating system for tiny networked sensors. In Proc. of IEEE Local Computer Networks (LCN). IEEE Computer Society, Los Alamitos, CA, USA, 455–462.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Eric Blake. 2018. Understanding QEMU devices. https://www.qemu.org/2018/02/09/understanding-qemu-devices/, last accessed 28-05-2021. (2018).Google ScholarGoogle Scholar
  21. Bo Feng, Alejandro Mera, and Long Lu. 2020. P2IM: Scalable and hardware-independent firmware testing via automatic peripheral interface modeling. In 29th USENIX Security Symposium. USENIX Association, 1237–1254.Google ScholarGoogle Scholar
  22. Niels Gandras̎, Michel Rottleuthner, and Thomas C. Schmidt. 2021. Work-in-Progress: Large-scale timer hardware analysis for a flexible low-level timer-API design. In Proceedings of EMSOFT 2021. ACM, New York, NY, USA. Accepted for publication.Google ScholarGoogle Scholar
  23. Vahid Garousi, Michael Felderer, Çağrı Murat Karapıçak, and Uğur Yılmaz. 2018. Testing embedded software: A survey of the literature. Information and Software Technology 104 (12 2018), 14–45.Google ScholarGoogle Scholar
  24. Kai Geissdoerfer, Mikołaj Chwalisz, and Marco Zimmerling. 2019. Shepherd: A portable testbed for the batteryless IoT. In Proc. 17th Conf. on Embedded Networked Sensor Systems (SenSys’19). ACM, New York, NY, USA, 83–95.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. I. Georgiev and I. Georgiev. 2019. Simulation-based self-testing in IoT-enabled manufacturing. In Proc. of International Conference on Information Technologies (InfoTech). IEEE, Piscataway, NJ, USA.Google ScholarGoogle Scholar
  26. Anna Katrina Gomez and Simi Bajaj. 2019. Challenges of testing complex internet of things (IoT) devices and systems. In Proc. of 11th Intern. Conf. on Knowledge and Systems Engineering (KSE). IEEE, Piscataway, NJ, USA.Google ScholarGoogle ScholarCross RefCross Ref
  27. Cenk Gündogan, Peter Kietzmann, Martine S. Lenders, Hauke Petersen, Michael Frey, Thomas C. Schmidt, Felix Shzu-Juraschek, and Matthias Wählisch. 2021. The impact of networking protocols on massive m2m communication in the industrial IoT. IEEE Transactions on Network and Service Management (TNSM) (2021). https://doi.org/10.1109/TNSM.2021.3089549Google ScholarGoogle ScholarCross RefCross Ref
  28. Vlado Handziski, Andreas Köpke, Andreas Willig, and Adam Wolisz. 2006. TWIST: A scalable and reconfigurable testbed for wireless indoor experiments with sensor networks. In REALMAN’06: Proceedings of the 2nd international workshop on Multi-hop ad hoc networks: from theory to reality. ACM.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Michael Karlesky, Greg Williams, William Bereza, and Matt Fletcher. 2007. Mocking the embedded World: Test-driven development, continuous integration, and design patterns. In Embedded Systems Conference (ESC 413). 1518–1532.Google ScholarGoogle Scholar
  30. Jannis S. Keränen and T. D. Räty. 2012. Model-based testing of embedded systems in hardware in the loop environment. IET Software 6, 4 (Aug. 2012), 364–376.Google ScholarGoogle ScholarCross RefCross Ref
  31. Peter Kietzmann, Lena Boeckmann, Leandro Lanzieri, Thomas C. Schmidt, and Matthias Wählisch. 2021. A performance study of crypto-hardware in the low-end IoT. In International Conference on Embedded Wireless Systems and Networks (EWSN). ACM, New York, USA, 12.Google ScholarGoogle Scholar
  32. Peter Kietzmann, Thomas C. Schmidt, and Matthias Wählisch. 2021. A Guideline on Pseudorandom Number Generation (PRNG) in the IoT. ACM Comput. Surv. 54, 6 (July 2021), 112:1–112:38. https://dl.acm.org/doi/10.1145/3453159Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Hiun Kim, Abbas Ahmad, Jaeyoung Hwang, Hamza Baqa, Franck Le Gall, Miguel Angel Reina Ortega, and JaeSeung Song. 2018. IoT-TaaS: Towards a prospective iot testing framework. IEEE Access 6 (April 2018), 15480–15493.Google ScholarGoogle Scholar
  34. Labcenter Electronics North America. 2021. Proteus Design Suite. https://www.labcenter.com, last accessed 01-01-2021. (2021).Google ScholarGoogle Scholar
  35. Roman Lim, Federico Ferrari, Marco Zimmerling, Christoph Walser, Philipp Sommer, and Jan Beutel. 2013. FlockLab: A testbed for distributed, synchronized tracing and profiling of wireless embedded systems. In Proc. 12th International Conference on Information Processing in Sensor Networks (IPSN’13). ACM, New York, NY, USA, 153–166.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Roman Lim, Balz Maag, Benjamin Dissler, Jan Beutel, and Lothar Thiele. 2015. A testbed for fine-grained tracing of time sensitive behavior in wireless sensor networks. In 2015 IEEE 40th Local Computer Networks Conference Workshops (LCN Workshops). IEEE.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Weiwei Lin, Hongwei Zeng, Honghao Gao, Huaikou Miao, and Xiaolin Wang. 2018. Test sequence reduction of wireless protocol conformance testing to internet of things. Security and Communication Networks 2018 (2018), 1–13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Linaro Limited. 2019. LAVA. https://docs.lavasoftware.org/lava/index.html, last accessed 01-01-2021. (2019).Google ScholarGoogle Scholar
  39. Babur Hayat Malik, Myda Khalid, Maliha Maryam, M. Nauman Ali, Sheraz Yousaf, Mudassar Mehmood, Hammad Saleem, and Viktoria Stray. 2019. IoT Testing-as-a-Service: A new dimension of automation. International Journal of Advanced Computer Science and Applications 10, 5 (2019), 364–371.Google ScholarGoogle ScholarCross RefCross Ref
  40. Torvald Mårtensson. 2019. Continuous Integration and Delivery Applied to Large-Scale Software-Intensive Embedded Systems. Ph.D. Dissertation. University of Groningen.Google ScholarGoogle Scholar
  41. MathWorks. 2021. Simulink. https://mathworks.com/products/simulink.html, last accessed 01-01-2021. (2021).Google ScholarGoogle Scholar
  42. Jani Metsa, Mika Katara, and Tommi Mikkonen. 2007. Testing non-functional requirements with aspects: An industrial case study. In Seventh International Conference on Quality Software (QSIC’07). 5–14.Google ScholarGoogle ScholarCross RefCross Ref
  43. Ghadeer Murad, Aalaa Badarneh, Abdallah Quscf, and Fadi Almasalha. 2018. Software testing techniques in IoT. In Proc. of 8th Intern. Conf. on Computer Science and Information Technology (CSIT). IEEE, NJ, USA, 17–21.Google ScholarGoogle ScholarCross RefCross Ref
  44. Marius Muresan and Dan Pitica. 2012. Software in the loop environment reliability for testing embedded code. In IEEE 18th Intern. Symposium for Design and Technology in Electronic Packaging (SIITME). 325–328.Google ScholarGoogle ScholarCross RefCross Ref
  45. Pedro Martins Pontes, Bruno Lima, and João Pascoal Faria. 2018. Izinto: A pattern-based iot testing framework. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops. ACM, New York, NY, USA, 125–131.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. QEMU. 2021. QEMU - the FAST! processor emulator. https://www.qemu.org, last accessed 28-05-2021. (2021).Google ScholarGoogle Scholar
  47. John Regehr. 2005. Random testing of interrupt-driven software. In Proc. 5th ACM International Conference on Embedded Software (EMSOFT’05). ACM, New York, NY, USA, 290–298.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Renode. 2021. Renode Homepage. https://renode.io, last accessed 28-05-2021. (2021).Google ScholarGoogle Scholar
  49. RIOT. 2021. RIOT Emulators. https://api.riot-os.org/emulators.html, last accessed 28-05-2021. (2021).Google ScholarGoogle Scholar
  50. Michel Rottleuthner, Thomas C. Schmidt, and Matthias Wählisch. 2021. Sense your power: The ECO approach to energy awareness for IoT devices. ACM Transactions on Embedded Computing Systems (TECS) 20, 3 (March 2021), 24:1–24:25. https://doi.org/10.1145/3441643Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Per Runeson. 2006. A survey of unit testing practices. IEEE Software 23, 4 (July 2006), 22–29.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Raimondas Sasnauskas, Olaf Landsiedel, Muhammad Hamad Alizai, Carsten Weise, Stefan Kowalewski, and Klaus Wehrle. 2010. KleeNet: Discovering insidious interaction bugs in wireless sensor networks before deployment. In Proc. 9th ACM/IEEE Intern. Conf. on Information Processing in Sensor Networks (IPSN’10). ACM, New York, NY, USA, 186–196.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Quirin Scheitle, Matthias Wählisch, Oliver Gasser, Thomas C. Schmidt, and Georg Carle. 2017. Towards an ecosystem for reproducible research in computer networking. In Proc. of ACM SIGCOMM Reproducibility Workshop. ACM, New York, NY, USA, 5–8.Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Aritra Sengupta, Tanakorn Leesatapornwongsa, Masoud Saeida Ardekani, and Cesar A. Stuardo. 2019. Transactuations: Where transactions meet the physical world. In 2019 USENIX Annual Technical Conference. USENIX Association, Renton, WA, USA, 91–106.Google ScholarGoogle Scholar
  55. Jooyoung Seo, Yuhoon Ki, Byoungju Choi, and Kwanghyun La. 2008. Which spot should i test for effective embedded software testing? In Second International Conference on Secure System Integration and Reliability Improvement. 135–142.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Jooyoung Seo, Ahyoung Sung, Byoungju Choi, and Sungbong Kang. 2007. Automating embedded software testing on an emulated target board. In Proc. Second Intern. WS on Automation of Software Test (AST’07). IEEE Computer Society, USA, 9.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Per Erik Strandberg. 2018. Automated System Level Software Testing of Networked Embedded Systems. Licentiate Theses 275. Mälardalen University Press, Embedded Systems.Google ScholarGoogle Scholar
  58. Ahyoung Sung, Byoungju Choi, and Seokkyoo Shin. 2007. An interface test model for hardware-dependent software and embedded OS API of the embedded system. Computer Standards and Interfaces 29, 4 (May 2007), 430–443.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Li Tan, Jesung Kim, Oleg Sokolsky, and Insup Lee. 2004. Model-based testing and monitoring for hybrid embedded systems. In Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004.IEEE, 487–492.Google ScholarGoogle ScholarCross RefCross Ref
  60. Teik-Boon Tan and Wai-Khuen Cheng. 2019. Software testing levels in internet of things (IoT) architecture. In Proc. of International Computer Symposium (ICS 2018). New Trends in Computer Technologies and Applications (CCIS), Vol. 1013. Springer Nature Singapore, Singapore, 385–390.Google ScholarGoogle ScholarCross RefCross Ref
  61. Pedro Taveras. 2018. A systematic exploration on challenges and limitations in middleware programming for iot technology. International Journal of Hyperconnectivity and the Internet of Things 2, 2 (Dec. 2018).Google ScholarGoogle ScholarCross RefCross Ref
  62. Roman Trüb, Reto Da Forno, Lukas Sigrist, Lorin Mühlebach, Andreas Biri, Jan Beutel, and Lothar Thiele. 2020. FlockLab 2: Multi-modal testing and validation for wireless IoT. In 3rd WS on Benchmarking Cyber-Physical Systems and Internet of Things (CPS-IoTBench 2020). ETH Zurich, TIK Laboratory.Google ScholarGoogle Scholar
  63. Josef Vejlupek, Robert Grepl, Petr Krejčí, František Lesák, and Karel Matouš. 2014. Hardware-in-the-loop simulation for automotive parking assistant control units. Proc. 16th International Conference on Mechatronics, Mechatronika 2014 (December 2014), 325–330.Google ScholarGoogle Scholar
  64. Darius Virzonis, T. Jukna, and D. Ramunas. 2004. Design of the embedded software using flexible hardware-in-the-loop simulation scheme. In Proc. 12th IEEE Mediter. Electrotechnical Conf. Vol. 1, 351–354.Google ScholarGoogle Scholar
  65. Matthias Woehrle, Christian Plessl, Jan Beutel, and Lothar Thiele. 2007. Increasing the reliability of wireless sensor networks with a distributed testing framework. In 4th WS on Embedded Networked Sensors (EmNets’07). ACM, 93–97.Google ScholarGoogle Scholar
  66. Zephyr Project. 2020. Zephyr. https://www.zephyrproject.org, last accessed 07-17-2020. (2020).Google ScholarGoogle Scholar

Index Terms

  1. PHiLIP on the HiL: Automated Multi-Platform OS Testing With External Reference Devices

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          Get this Article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          View Issue’s Table of Contents
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!