skip to main content
research-article

Engineering Model-Based Software Testing of WIMP Interactive Applications: A Process based on Formal Models and the SQUAMATA Tool

Authors Info & Claims
Published:29 May 2021Publication History
Skip Abstract Section

Abstract

The goal of software testing is to detect defects with the objective of removing them at a later stage in the development process. Interactive software development follows the User Centered Design approach that promotes continuous involvement of users both at design and evaluation phases. This process is meant to produce usable interactive software by gathering functional and non-functional requirements related to both user needs and context of use. However, taking into account these potentially very-complex-to-implement requirements increases the complexity of the software that is likely, without appropriate methods and tools, to encompass a large number of defects. One of the limitations of UCD approaches is that it provides no guidance on the engineering of the interactive application, which thus usually embeds numerous defects resulting in failures at the origin of user frustrations and performance drops. Even though a classification of interactive application defects has been proposed, interactive application testers remain only superficially supported in detecting them. This paper defines a model-based approach to engineer the testing activity for interactive applications. It proposes a process that bridges the gap between UCD artefacts and interactive software implementation by the production of a dedicated formal model exploited for testing purposes only. The application of the process is demonstrated on an interactive cockpit WIMP application. Finally, threats to validity (capability of the approach to detect defects and to ensure an acceptable coverage testing of the interactive application) are addressed by a longitudinal study on 61 variants of a simple application developed by 61 different developers. ?

References

  1. Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing 1, 1 (January 2004), 11--33. DOI:https://doi.org/10.1109/TDSC.2004.2 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. ARINC 661--7, Prepared by Airlines Electronic Engineering Committee. Cockpit Display System Interfaces to User Systems. Arinc Specification 661--7; 2020Google ScholarGoogle Scholar
  3. Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif M. Memon. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55, 10 (October 2013), 1679--1694. DOI:https://doi.org/10.1016/j.infsof.2013.03.004 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Eric Barboni, Stéphane Conversy, David Navarre, and Philippe Palanque. 2007. Model-Based Engineering of Widgets, User Applications and Servers Compliant with ARINC 661 Specification. In Interactive Systems. Design, Specification, and Verification (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 25--38. DOI:https://doi.org/10.1007/978--3--540--69554--7_3 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Eric Barboni, Célia Martinie, David Navarre, Philippe A. Palanque, Marco Winckler: Bridging the gap between a behavioral formal description technique and a user interface description language: Enhancing ICO with a graphical user interface markup language. Sci. Comput. Program. 86: 3--29 (2014)Google ScholarGoogle Scholar
  6. Ana Barbosa, Ana C.R. Paiva, and José Creissac Campos. 2011. Test case generation from mutated task models. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems (EICS '11). Association for Computing Machinery, New York, NY, USA, 175--184. DOI:https://doi.org/10.1145/1996461.1996516 Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Rémi Bastide, Philippe A. Palanque, Duc-Hoa Le, Jaime Munoz: Integrating Rendering Specifications into a Formalism for the Design of Interactive Systems. DSV-IS 1998: 171--190Google ScholarGoogle Scholar
  8. Rémi Bastide, David Navarre, Philippe Palanque, Amélie Schyn, and Pierre Dragicevic. 2004. A model-based approach for real-time embedded multimodal systems in military aircrafts. In Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04), Association for Computing Machinery, State College, PA, USA, 243--250. DOI:https://doi.org/10.1145/1027933.1027974 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Michel Beaudouin-Lafon and Wendy Mackay. 2002. Prototyping tools and techniques. The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. L. Erlbaum Associates Inc., USA, 1006--1031. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Barry Boehm. 2006. A view of 20th and 21st century software engineering. In Proceedings of the 28th international conference on Software engineering (ICSE '06). Association for Computing Machinery, New York, NY, USA, 12--29. DOI:https://doi.org/10.1145/1134285.1134288 Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Pierre Bourque, and Richard E. Fairley. 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 (3rd ed.). IEEE Computer Society Press, Los Alamitos, CA, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Judy Bowen and Steve Reeves. 2017. Generating Obligations, Assertions and Tests from UI Models. Proc. ACM Hum.-Comput. Interact. 1, EICS (June 2017), 5:1--5:18. DOI:https://doi.org/10.1145/3095807 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Åsa Cajander, Marta Larusdottir, and Jan Gulliksen. 2013. Existing but Not Explicit - The User Perspective in Scrum Projects in Practice. In Human-Computer Interaction -- INTERACT 2013 (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 762--779. DOI:https://doi.org/10.1007/978--3--642--40477--1_52Google ScholarGoogle Scholar
  14. Alexandre Canny, Elodie Bouzekri, Célia Martinie, and Philippe Palanque. 2019. Rationalizing the Need of Architecture-Driven Testing of Interactive Systems. In Human-Centered Software Engineering (Lecture Notes in Computer Science), Springer International Publishing, Cham, 164--186. DOI:https://doi.org/10.1007/978--3-030-05909--5_10.Google ScholarGoogle Scholar
  15. Alexandre Canny, David Navarre, José Creissac Campos, and Philippe Palanque. 2020. Model-Based Testing of Post-WIMP Interactions Using Object Oriented Petri-Nets. In Formal Methods. FM 2019 International Workshops (Lecture Notes in Computer Science), Springer International Publishing, Cham, 486--502. DOI:https://doi.org/10.1007/978--3-030--54994--7_35Google ScholarGoogle Scholar
  16. Alexandre Canny, Philippe Palanque, and David Navarre. 2020. Model-Based Testing of GUI Applications Featuring Dynamic Instanciation of Widgets. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 95--104. DOI:https://doi.org/10.1109/ICSTW50294.2020.00029Google ScholarGoogle Scholar
  17. Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 429--440. DOI:https://doi.org/10.1109/ASE.2015.89 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. José Creissac Campos, Camille Fayollas, Célia Martinie, David Navarre, Philippe Palanque, and Miguel Pinto. 2016. Systematic Automation of Scenario-based Testing of User Interfaces. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16), ACM, New York, NY, USA, 138--148. DOI:https://doi.org/10.1145/2933242.2948735 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. José Creissac Campos, Camille Fayollas, Marcelo Gonçalves, Célia Martinie, David Navarre, Philippe Palanque, and Miguel Pinto. 2017. A More Intelligent Test Case Generation Approach through Task Models Manipulation. Proc. ACM Hum.-Comput. Interact. 1, EICS, Article 9 (June 2017), 20 pages. DOI:https://doi.org/10.1145/3095811 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. José Creissac Campos, Camille Fayollas, Michael D. Harrison, Célia Martinie, Paolo Masci, and Philippe Palanque. 2020. Supporting the Analysis of Safety Critical User Interfaces: An Exploration of Three Formal Tools. ACM Trans. Comput.-Hum. Interact. 27, 5, Article 35 (October 2020), 48 pages. DOI:https://doi.org/10.1145/3404199 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Martin Cronel, Bruno Dumas, Philippe Palanque, and Alexandre Canny. 2019. MIODMIT: A Generic Architecture for Dynamic Multimodal Interactive Systems. In Human-Centered Software Engineering (Lecture Notes in Computer Science), Springer International Publishing, Cham, 109--129. DOI:https://doi.org/10.1007/978--3-030-05909--5_7Google ScholarGoogle Scholar
  22. European Aviation Safety Agency. 2017. CS-25 -- Certification Specifications and Acceptable Means of Compliance for Large Aeroplanes. (2017).Google ScholarGoogle Scholar
  23. Camille Fayollas, Célia Martinie, David Navarre, and Philippe Palanque. 2016. Engineering mixed-criticality interactive applications. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). Association for Computing Machinery, New York, NY, USA, 108--119. DOI:https://doi.org/10.1145/2933242.2933258 Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Jennifer Ferreira, Helen Sharp, and Hugh Robinson. 2010. Values and Assumptions Shaping Agile Development and User Experience Design in Practice. In Agile Processes in Software Engineering and Extreme Programming (Lecture Notes in Business Information Processing), Springer, Berlin, Heidelberg, 178--183. DOI:https://doi.org/10.1007/978--3--642--13054-0_15Google ScholarGoogle Scholar
  25. Google. Monkey : UI/Application Exercise Monkey, https://developer.android.com/studio/test/monkey , last accessed February 2021.Google ScholarGoogle Scholar
  26. Arnaud Hamon, Philippe Palanque, José Luís Silva, Yannick Deleris, and Eric Barboni. 2013. Formal description of multi-touch interactions. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '13). Association for Computing Machinery, New York, NY, USA, 207--216. DOI:https://doi.org/10.1145/2494603.2480311 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Arnaud Hamon, Philippe Palanque, Martin Cronel, Raphaël André, Eric Barboni, and David Navarre. 2014. Formal modelling of dynamic instantiation of input devices and interaction techniques: application to multi-touch interactions. In Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems (EICS '14). Association for Computing Machinery, New York, NY, USA, 173--178. DOI:https://doi.org/10.1145/2607023.2610286 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. International Software Testing Qualification Board. ISTQB Glossary: https://glossary.istqb.org/, last accessed February 2021.Google ScholarGoogle Scholar
  29. ISO 9241--210:2010 - Ergonomics of human-system interaction -- Part 210: Human-centred design for interactive systems. Retrieved June 7, 2018 from https://www.iso.org/standard/52075.htmlGoogle ScholarGoogle Scholar
  30. David Janzen and Hossein Saiedian. 2005. Test-driven development concepts, taxonomy, and future direction. Computer 38, 9 (September 2005), 43--50. DOI:https://doi.org/10.1109/MC.2005.314 Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Kati Kuusinen. 2015. Task Allocation Between UX Specialists and Developers in Agile Software Development Projects. In Human-Computer Interaction -- INTERACT 2015 (Lecture Notes in Computer Science), Springer International Publishing, Cham, 27--44. DOI:https://doi.org/10.1007/978--3--319--22698--9_3 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research Methods in Human-Computer Interaction. Morgan Kaufmann. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Marta Larusdottir, Jan Gulliksen, Åsa Cajander, A license to kill -- Improving UCSD in Agile development, Journal of Systems and Software, Volume 123, 2017, Pages 214--222, ISSN 0164--1212, https://doi.org/10.1016/j.jss.2016.01.024.Google ScholarGoogle ScholarCross RefCross Ref
  34. Valéria Lelli, Arnaud Blouin, and Benoit Baudry. 2015. Classifying and Qualifying GUI Defects. 1--10. DOI:https://doi.org/10.1109/ICST.2015.7102582Google ScholarGoogle Scholar
  35. Valéria Lelli, Arnaud Blouin, Benoit Baudry, and Fabien Coulon. 2015. On model-based testing advanced GUIs. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 1--10. DOI:https://doi.org/10.1109/ICSTW.2015.7107403Google ScholarGoogle ScholarCross RefCross Ref
  36. Célia Martinie, Philippe Palanque, Martina Ragosta, and Racim Fahssi. 2013. Extending procedural task models by systematic explicit integration of objects, knowledge and information. In Proceedings of the 31st European Conference on Cognitive Ergonomics (ECCE '13). Association for Computing Machinery, New York, NY, USA, Article 23, 1--10. DOI:https://doi.org/10.1145/2501907.2501954 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Atif M. Memon, Mary Lou Soffa, and Martha E. Pollack. 2001. Coverage Criteria for GUI Testing. In Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-9), ACM, New York, NY, USA, 256--267. DOI:https://doi.org/10.1145/503209.503244 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. John McDermid et Knut Ripken. Life cycle support in the ADA environment. University Press, 1984. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Eduardo Mosqueira-Rey, David Alonso-Ríos, and Vincente Moret-Bonillo. 2009. Usability taxonomy and context-of-use taxonomy for usability analysis. In 2009 IEEE International Conference on Systems, Man and Cybernetics, 812--817. DOI:https://doi.org/10.1109/ICSMC.2009.5346929 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. David Navarre, Philippe Palanque, Jean-Francois Ladry, and Eric Barboni. 2009. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact. 16, 4, Article 18 (November 2009), 56 pages. DOI:https://doi.org/10.1145/1614390.1614393 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Philippe Palanque, Eric Barboni, Célia Martinie, David Navarre, and Marco Winckler. 2011. A model-based approach for supporting engineering usability evaluation of interaction techniques. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems (EICS '11). Association for Computing Machinery, New York, NY, USA, 21--30. DOI:https://doi.org/10.1145/1996461.1996490 Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Philippe Palanque, Andy Cockburn, Léopold Désert-Legendre, Carl Gutwin, and Yannick Deleris. 2019. Brace Touch: A Dependable, Turbulence-Tolerant, Multi-touch Interaction Technique for Interactive Cockpits. In Computer Safety, Reliability, and Security (Lecture Notes in Computer Science), Springer International Publishing, Cham, 53--68. DOI:https://doi.org/10.1007/978--3-030--26601--1_4Google ScholarGoogle Scholar
  43. Mauro Pezzè, Paolo Rondena, and Daniele Zuddas. 2018. Automatic GUI Testing of Desktop Applications: An Empirical Assessment of the State of the Art. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (ISSTA '18), ACM, New York, NY, USA, 54--62. DOI:https://doi.org/10.1145/3236454.3236489 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Marc Rettig. 1994. Prototyping for tiny fingers. Commun. ACM 37, 4 (April 1994), 21--27. DOI:https://doi.org/10.1145/175276.175288 Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Winston W. Royce. Managing the Development of Large Software Systems. IEEE Wescon, pp 1--9, 1970Google ScholarGoogle Scholar
  46. Ken Schwaber and Mike Beedle, Agile Software Development with SCRUM. Upper Saddle River, NJ: Prentice-Hall, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Desirée Sy. 2007. Adapting Usability Investigations for Agile User-centered Design. Journal of Usability Studies, 2(3), 112--132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Harold Thimbleby. 2009. Contributing to safety and due diligence in safety-critical interactive systems development by generating and analyzing finite state models. In Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems (EICS '09). Association for Computing Machinery, New York, NY, USA, 221--230. DOI:https://doi.org/10.1145/1570433.1570474 Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Jessica Turner, Judy Bowen, and Steve Reeves. 2020. Model-based Testing of Interactive Systems using Interaction Sequences. Proc. ACM Hum.-Comput. Interact. 4, EICS, Article 85 (June 2020), 37 pages. DOI:https://doi.org/10.1145/3397873Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22, 5 (August 2012), 297--312. DOI:https://doi.org/10.1002/stvr.456 Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Jean Vanderdonckt, Abdo Beirekdar. Automated Web Evaluation by Guideline Review. J. Web Eng. 4(2): 102--117 (2005) Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Melody Y. Ivory, Marti A. Hearst. The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33(4): 470--516 (2001). Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Engineering Model-Based Software Testing of WIMP Interactive Applications: A Process based on Formal Models and the SQUAMATA Tool

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!