Abstract

The goal of software testing is to detect defects with the objective of removing them at a later stage in the development process. Interactive software development follows the User Centered Design approach that promotes continuous involvement of users both at design and evaluation phases. This process is meant to produce usable interactive software by gathering functional and non-functional requirements related to both user needs and context of use. However, taking into account these potentially very-complex-to-implement requirements increases the complexity of the software that is likely, without appropriate methods and tools, to encompass a large number of defects. One of the limitations of UCD approaches is that it provides no guidance on the engineering of the interactive application, which thus usually embeds numerous defects resulting in failures at the origin of user frustrations and performance drops. Even though a classification of interactive application defects has been proposed, interactive application testers remain only superficially supported in detecting them. This paper defines a model-based approach to engineer the testing activity for interactive applications. It proposes a process that bridges the gap between UCD artefacts and interactive software implementation by the production of a dedicated formal model exploited for testing purposes only. The application of the process is demonstrated on an interactive cockpit WIMP application. Finally, threats to validity (capability of the approach to detect defects and to ensure an acceptable coverage testing of the interactive application) are addressed by a longitudinal study on 61 variants of a simple application developed by 61 different developers. ?
- Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE Transactions on Dependable and Secure Computing 1, 1 (January 2004), 11--33. DOI:https://doi.org/10.1109/TDSC.2004.2 Google Scholar
Digital Library
- ARINC 661--7, Prepared by Airlines Electronic Engineering Committee. Cockpit Display System Interfaces to User Systems. Arinc Specification 661--7; 2020Google Scholar
- Ishan Banerjee, Bao Nguyen, Vahid Garousi, and Atif M. Memon. 2013. Graphical user interface (GUI) testing: Systematic mapping and repository. Information and Software Technology 55, 10 (October 2013), 1679--1694. DOI:https://doi.org/10.1016/j.infsof.2013.03.004 Google Scholar
Digital Library
- Eric Barboni, Stéphane Conversy, David Navarre, and Philippe Palanque. 2007. Model-Based Engineering of Widgets, User Applications and Servers Compliant with ARINC 661 Specification. In Interactive Systems. Design, Specification, and Verification (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 25--38. DOI:https://doi.org/10.1007/978--3--540--69554--7_3 Google Scholar
Digital Library
- Eric Barboni, Célia Martinie, David Navarre, Philippe A. Palanque, Marco Winckler: Bridging the gap between a behavioral formal description technique and a user interface description language: Enhancing ICO with a graphical user interface markup language. Sci. Comput. Program. 86: 3--29 (2014)Google Scholar
- Ana Barbosa, Ana C.R. Paiva, and José Creissac Campos. 2011. Test case generation from mutated task models. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems (EICS '11). Association for Computing Machinery, New York, NY, USA, 175--184. DOI:https://doi.org/10.1145/1996461.1996516 Google Scholar
Digital Library
- Rémi Bastide, Philippe A. Palanque, Duc-Hoa Le, Jaime Munoz: Integrating Rendering Specifications into a Formalism for the Design of Interactive Systems. DSV-IS 1998: 171--190Google Scholar
- Rémi Bastide, David Navarre, Philippe Palanque, Amélie Schyn, and Pierre Dragicevic. 2004. A model-based approach for real-time embedded multimodal systems in military aircrafts. In Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04), Association for Computing Machinery, State College, PA, USA, 243--250. DOI:https://doi.org/10.1145/1027933.1027974 Google Scholar
Digital Library
- Michel Beaudouin-Lafon and Wendy Mackay. 2002. Prototyping tools and techniques. The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. L. Erlbaum Associates Inc., USA, 1006--1031. Google Scholar
Digital Library
- Barry Boehm. 2006. A view of 20th and 21st century software engineering. In Proceedings of the 28th international conference on Software engineering (ICSE '06). Association for Computing Machinery, New York, NY, USA, 12--29. DOI:https://doi.org/10.1145/1134285.1134288 Google Scholar
Digital Library
- Pierre Bourque, and Richard E. Fairley. 2014. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 (3rd ed.). IEEE Computer Society Press, Los Alamitos, CA, USA. Google Scholar
Digital Library
- Judy Bowen and Steve Reeves. 2017. Generating Obligations, Assertions and Tests from UI Models. Proc. ACM Hum.-Comput. Interact. 1, EICS (June 2017), 5:1--5:18. DOI:https://doi.org/10.1145/3095807 Google Scholar
Digital Library
- Åsa Cajander, Marta Larusdottir, and Jan Gulliksen. 2013. Existing but Not Explicit - The User Perspective in Scrum Projects in Practice. In Human-Computer Interaction -- INTERACT 2013 (Lecture Notes in Computer Science), Springer, Berlin, Heidelberg, 762--779. DOI:https://doi.org/10.1007/978--3--642--40477--1_52Google Scholar
- Alexandre Canny, Elodie Bouzekri, Célia Martinie, and Philippe Palanque. 2019. Rationalizing the Need of Architecture-Driven Testing of Interactive Systems. In Human-Centered Software Engineering (Lecture Notes in Computer Science), Springer International Publishing, Cham, 164--186. DOI:https://doi.org/10.1007/978--3-030-05909--5_10.Google Scholar
- Alexandre Canny, David Navarre, José Creissac Campos, and Philippe Palanque. 2020. Model-Based Testing of Post-WIMP Interactions Using Object Oriented Petri-Nets. In Formal Methods. FM 2019 International Workshops (Lecture Notes in Computer Science), Springer International Publishing, Cham, 486--502. DOI:https://doi.org/10.1007/978--3-030--54994--7_35Google Scholar
- Alexandre Canny, Philippe Palanque, and David Navarre. 2020. Model-Based Testing of GUI Applications Featuring Dynamic Instanciation of Widgets. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 95--104. DOI:https://doi.org/10.1109/ICSTW50294.2020.00029Google Scholar
- Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated Test Input Generation for Android: Are We There Yet? (E). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 429--440. DOI:https://doi.org/10.1109/ASE.2015.89 Google Scholar
Digital Library
- José Creissac Campos, Camille Fayollas, Célia Martinie, David Navarre, Philippe Palanque, and Miguel Pinto. 2016. Systematic Automation of Scenario-based Testing of User Interfaces. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16), ACM, New York, NY, USA, 138--148. DOI:https://doi.org/10.1145/2933242.2948735 Google Scholar
Digital Library
- José Creissac Campos, Camille Fayollas, Marcelo Gonçalves, Célia Martinie, David Navarre, Philippe Palanque, and Miguel Pinto. 2017. A More Intelligent Test Case Generation Approach through Task Models Manipulation. Proc. ACM Hum.-Comput. Interact. 1, EICS, Article 9 (June 2017), 20 pages. DOI:https://doi.org/10.1145/3095811 Google Scholar
Digital Library
- José Creissac Campos, Camille Fayollas, Michael D. Harrison, Célia Martinie, Paolo Masci, and Philippe Palanque. 2020. Supporting the Analysis of Safety Critical User Interfaces: An Exploration of Three Formal Tools. ACM Trans. Comput.-Hum. Interact. 27, 5, Article 35 (October 2020), 48 pages. DOI:https://doi.org/10.1145/3404199 Google Scholar
Digital Library
- Martin Cronel, Bruno Dumas, Philippe Palanque, and Alexandre Canny. 2019. MIODMIT: A Generic Architecture for Dynamic Multimodal Interactive Systems. In Human-Centered Software Engineering (Lecture Notes in Computer Science), Springer International Publishing, Cham, 109--129. DOI:https://doi.org/10.1007/978--3-030-05909--5_7Google Scholar
- European Aviation Safety Agency. 2017. CS-25 -- Certification Specifications and Acceptable Means of Compliance for Large Aeroplanes. (2017).Google Scholar
- Camille Fayollas, Célia Martinie, David Navarre, and Philippe Palanque. 2016. Engineering mixed-criticality interactive applications. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). Association for Computing Machinery, New York, NY, USA, 108--119. DOI:https://doi.org/10.1145/2933242.2933258 Google Scholar
Digital Library
- Jennifer Ferreira, Helen Sharp, and Hugh Robinson. 2010. Values and Assumptions Shaping Agile Development and User Experience Design in Practice. In Agile Processes in Software Engineering and Extreme Programming (Lecture Notes in Business Information Processing), Springer, Berlin, Heidelberg, 178--183. DOI:https://doi.org/10.1007/978--3--642--13054-0_15Google Scholar
- Google. Monkey : UI/Application Exercise Monkey, https://developer.android.com/studio/test/monkey , last accessed February 2021.Google Scholar
- Arnaud Hamon, Philippe Palanque, José Luís Silva, Yannick Deleris, and Eric Barboni. 2013. Formal description of multi-touch interactions. In Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '13). Association for Computing Machinery, New York, NY, USA, 207--216. DOI:https://doi.org/10.1145/2494603.2480311 Google Scholar
Digital Library
- Arnaud Hamon, Philippe Palanque, Martin Cronel, Raphaël André, Eric Barboni, and David Navarre. 2014. Formal modelling of dynamic instantiation of input devices and interaction techniques: application to multi-touch interactions. In Proceedings of the 2014 ACM SIGCHI symposium on Engineering interactive computing systems (EICS '14). Association for Computing Machinery, New York, NY, USA, 173--178. DOI:https://doi.org/10.1145/2607023.2610286 Google Scholar
Digital Library
- International Software Testing Qualification Board. ISTQB Glossary: https://glossary.istqb.org/, last accessed February 2021.Google Scholar
- ISO 9241--210:2010 - Ergonomics of human-system interaction -- Part 210: Human-centred design for interactive systems. Retrieved June 7, 2018 from https://www.iso.org/standard/52075.htmlGoogle Scholar
- David Janzen and Hossein Saiedian. 2005. Test-driven development concepts, taxonomy, and future direction. Computer 38, 9 (September 2005), 43--50. DOI:https://doi.org/10.1109/MC.2005.314 Google Scholar
Digital Library
- Kati Kuusinen. 2015. Task Allocation Between UX Specialists and Developers in Agile Software Development Projects. In Human-Computer Interaction -- INTERACT 2015 (Lecture Notes in Computer Science), Springer International Publishing, Cham, 27--44. DOI:https://doi.org/10.1007/978--3--319--22698--9_3 Google Scholar
Digital Library
- Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research Methods in Human-Computer Interaction. Morgan Kaufmann. Google Scholar
Digital Library
- Marta Larusdottir, Jan Gulliksen, Åsa Cajander, A license to kill -- Improving UCSD in Agile development, Journal of Systems and Software, Volume 123, 2017, Pages 214--222, ISSN 0164--1212, https://doi.org/10.1016/j.jss.2016.01.024.Google Scholar
Cross Ref
- Valéria Lelli, Arnaud Blouin, and Benoit Baudry. 2015. Classifying and Qualifying GUI Defects. 1--10. DOI:https://doi.org/10.1109/ICST.2015.7102582Google Scholar
- Valéria Lelli, Arnaud Blouin, Benoit Baudry, and Fabien Coulon. 2015. On model-based testing advanced GUIs. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 1--10. DOI:https://doi.org/10.1109/ICSTW.2015.7107403Google Scholar
Cross Ref
- Célia Martinie, Philippe Palanque, Martina Ragosta, and Racim Fahssi. 2013. Extending procedural task models by systematic explicit integration of objects, knowledge and information. In Proceedings of the 31st European Conference on Cognitive Ergonomics (ECCE '13). Association for Computing Machinery, New York, NY, USA, Article 23, 1--10. DOI:https://doi.org/10.1145/2501907.2501954 Google Scholar
Digital Library
- Atif M. Memon, Mary Lou Soffa, and Martha E. Pollack. 2001. Coverage Criteria for GUI Testing. In Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-9), ACM, New York, NY, USA, 256--267. DOI:https://doi.org/10.1145/503209.503244 Google Scholar
Digital Library
- John McDermid et Knut Ripken. Life cycle support in the ADA environment. University Press, 1984. Google Scholar
Digital Library
- Eduardo Mosqueira-Rey, David Alonso-Ríos, and Vincente Moret-Bonillo. 2009. Usability taxonomy and context-of-use taxonomy for usability analysis. In 2009 IEEE International Conference on Systems, Man and Cybernetics, 812--817. DOI:https://doi.org/10.1109/ICSMC.2009.5346929 Google Scholar
Digital Library
- David Navarre, Philippe Palanque, Jean-Francois Ladry, and Eric Barboni. 2009. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. Comput.-Hum. Interact. 16, 4, Article 18 (November 2009), 56 pages. DOI:https://doi.org/10.1145/1614390.1614393 Google Scholar
Digital Library
- Philippe Palanque, Eric Barboni, Célia Martinie, David Navarre, and Marco Winckler. 2011. A model-based approach for supporting engineering usability evaluation of interaction techniques. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems (EICS '11). Association for Computing Machinery, New York, NY, USA, 21--30. DOI:https://doi.org/10.1145/1996461.1996490 Google Scholar
Digital Library
- Philippe Palanque, Andy Cockburn, Léopold Désert-Legendre, Carl Gutwin, and Yannick Deleris. 2019. Brace Touch: A Dependable, Turbulence-Tolerant, Multi-touch Interaction Technique for Interactive Cockpits. In Computer Safety, Reliability, and Security (Lecture Notes in Computer Science), Springer International Publishing, Cham, 53--68. DOI:https://doi.org/10.1007/978--3-030--26601--1_4Google Scholar
- Mauro Pezzè, Paolo Rondena, and Daniele Zuddas. 2018. Automatic GUI Testing of Desktop Applications: An Empirical Assessment of the State of the Art. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (ISSTA '18), ACM, New York, NY, USA, 54--62. DOI:https://doi.org/10.1145/3236454.3236489 Google Scholar
Digital Library
- Marc Rettig. 1994. Prototyping for tiny fingers. Commun. ACM 37, 4 (April 1994), 21--27. DOI:https://doi.org/10.1145/175276.175288 Google Scholar
Digital Library
- Winston W. Royce. Managing the Development of Large Software Systems. IEEE Wescon, pp 1--9, 1970Google Scholar
- Ken Schwaber and Mike Beedle, Agile Software Development with SCRUM. Upper Saddle River, NJ: Prentice-Hall, 2002. Google Scholar
Digital Library
- Desirée Sy. 2007. Adapting Usability Investigations for Agile User-centered Design. Journal of Usability Studies, 2(3), 112--132. Google Scholar
Digital Library
- Harold Thimbleby. 2009. Contributing to safety and due diligence in safety-critical interactive systems development by generating and analyzing finite state models. In Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systems (EICS '09). Association for Computing Machinery, New York, NY, USA, 221--230. DOI:https://doi.org/10.1145/1570433.1570474 Google Scholar
Digital Library
- Jessica Turner, Judy Bowen, and Steve Reeves. 2020. Model-based Testing of Interactive Systems using Interaction Sequences. Proc. ACM Hum.-Comput. Interact. 4, EICS, Article 85 (June 2020), 37 pages. DOI:https://doi.org/10.1145/3397873Google Scholar
Digital Library
- Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22, 5 (August 2012), 297--312. DOI:https://doi.org/10.1002/stvr.456 Google Scholar
Digital Library
- Jean Vanderdonckt, Abdo Beirekdar. Automated Web Evaluation by Guideline Review. J. Web Eng. 4(2): 102--117 (2005) Google Scholar
Digital Library
- Melody Y. Ivory, Marti A. Hearst. The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33(4): 470--516 (2001). Google Scholar
Digital Library
Index Terms
Engineering Model-Based Software Testing of WIMP Interactive Applications: A Process based on Formal Models and the SQUAMATA Tool
Recommendations
Multidimensional test coverage analysis: PARADIGM-COV tool
Currently, software tends to assume increasingly critical roles in our society so assuring its quality becomes ever more crucial. There are several tools and processes of software testing to help increase quality in virtually any type of software. One ...
Interactive System Testing: Beyond GUI Testing
EICS '18: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing SystemsThe increasing variety of Input/Output devices and functionalities in interactive systems raises concerns regarding the way they are tested. Indeed, while most of the existing testing techniques are suitable for interactive systems supporting WIMP ...
Designing and comparing automated test oracles for GUI-based software applications
Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle, a mechanism that determines whether a ...






Comments