Abstract
Ensuring that an interactive application allows users to perform their activities and reach their goals is critical to the overall usability of the interactive application. Indeed, the effectiveness factor of usability directly refers to this capability. Assessing effectiveness is a real challenge for usability testing as usability tests only cover a very limited number of tasks and activities. This paper proposes an approach towards automated testing of effectiveness of interactive applications. To this end we resort to two main elements: an exhaustive description of users' activities and goals using task models, and the generation of scenarios (from the task models) to be tested over the application. However, the number of scenarios can be very high (beyond the computing capabilities of machines) and we might end up testing multiple similar scenarios. In order to overcome these problems, we propose strategies based on task models manipulations (e.g., manipulating task nodes, operator nodes, information...) resulting in a more intelligent test case generation approach. For each strategy, we investigate its relevance (both in terms of test case generation and in terms of validity compared to the original task models) and we illustrate it with a small example. Finally, the proposed strategies are applied on a real-size case study demonstrating their relevance and validity to test interactive applications.
- 1998. ISO 9241--11: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11 : Guidance on usability (1 ed.). Technical Report. International Organization for Standardization, Geneva.Google Scholar
- Thomas A. Alspaugh, Debra J. Richardson, and Thomas A. Standish. 2005. Scenarios, State Machines and Purpose-driven Testing. In Proceedings of the Fourth International Workshop on Scenarios and State Machines: Models, Algorithms and Tools (SCESM '05). ACM, New York, NY, USA, 1--5. Google Scholar
Digital Library
- Domenico Amalfitano, Anna Rita Fasolino, and Porfirio Tramontana. 2011. A GUI Crawling-Based Technique for Android Mobile Application Testing. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW '11). IEEE Computer Society, Washington, DC, USA, 252--261. Google Scholar
Digital Library
- Judy Bowen and Steve Reeves. 2011. UI-driven Test-first Development of Interactive Systems. In Proceedings of the 3rd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '11). ACM, New York, NY, USA, 165--174. Google Scholar
Digital Library
- José C. Campos, Camille Fayollas, Célia Martinie, David Navarre, Philippe Palanque, and Miguel Pinto. 2016. Systematic Automation of Scenario-based Testing of User Interfaces. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA, 138--148. Google Scholar
Digital Library
- Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The Keystroke-level Model for User Performance Time with Interactive Systems. Commun. ACM 23, 7 (July 1980), 396--410. Google Scholar
Digital Library
- John P. Chin, Virginia A. Diehl, and Kent L. Norman. 1988. Development of an Instrument Measuring User Satisfaction of the Humancomputer Interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '88). ACM, New York, NY, USA, 213--218. Google Scholar
Digital Library
- Camille Fayollas, Célia Martinie, David Navarre, and Philippe Palanque. 2016. Engineering Mixed-criticality Interactive Applications. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA, 108--119. Google Scholar
Digital Library
- P. M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental PSychology 74 (1954), 381--391.Google Scholar
Cross Ref
- Andy Gimblett and Harold Thimbleby. 2010. User Interface Model Discovery: Towards a Generic Approach. In Proceedings of the 2nd ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '10). ACM, New York, NY, USA, 145--154. Google Scholar
Digital Library
- Yves Guiard. 2009. The Problem of Consistency in the Design of Fitts' Law Experiments: Consider Either Target Distance and Width or Movement Form and Scale. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'09). ACM, New York, NY, USA, 1809--1818. Google Scholar
Digital Library
- Marc Hassenzahl. 2004. Funology. Kluwer Academic Publishers, Norwell, MA, USA, Chapter The Thing and I: Understanding the Relationship Between User and Product, 31--42. http://dl.acm.org/citation.cfm?id=1139008.1139015 Google Scholar
Digital Library
- ISO/IEC. 2010. ISO/IEC 25010 - Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. Technical Report.Google Scholar
- Bonnie E. John and David E. Kieras. 1996. Using GOMS for User Interface Design and Evaluation: Which Technique? ACM Trans. Comput.-Hum. Interact. 3, 4 (Dec. 1996), 287--319. Google Scholar
Digital Library
- Barbara A. Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-Based Software Engineering. In Proceedings of the 26th International Conference on Software Engineering (ICSE '04). IEEE Computer Society, Washington, DC, USA, 273--281. http://dl.acm.org/citation.cfm?id=998675.999432 Google Scholar
Digital Library
- Valéria Lelli, Arnaud Blouin, Benoit Baudry, and Fabien Coulon. 2015. On Model-Based Testing Advanced GUIs. In Proceedings of the 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) -- Workshop on Advances in Model Based Testing (A-MOST 2015). Graz, Austria.Google Scholar
Cross Ref
- Jane Li, Laurie Wilson, Stuart Stapleton, and Patrick Cregan. 2006. Design of an Advanced Telemedicine System for Emergency Care. In Proceedings of the 18th Australia Conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (OZCHI '06). ACM, New York, NY, USA, 413--416. Google Scholar
Digital Library
- Célia Martinie, David Navarre, Philippe Palanque, and Camille Fayollas. 2015. A Generic Tool-supported Framework for Coupling Task Models and Interactive Applications. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '15). ACM, New York, NY, USA, 244--253. Google Scholar
Digital Library
- Célia Martinie, Philippe Palanque, and Marco Winckler. 2011. Structuring and Composition Mechanisms to Address Scalability Issues in Task Models. Springer Berlin Heidelberg, Berlin, Heidelberg, 589--609.Google Scholar
- Atif M. Memon. 2001. A comprehensive framework for testing graphical user interfaces. Ph.D. University of Pittsburgh. Google Scholar
Digital Library
- Atif M. Memon, Mary Lou Soffa, and Martha E. Pollack. 2001. Coverage Criteria for GUI Testing. In Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-9). ACM, New York, NY, USA, 256--267. Google Scholar
Digital Library
- Rodrigo M.L.M. Moreira and Ana C.R. Paiva. 2014. PBGT Tool: An Integrated Modeling and Testing Environment for Pattern-based GUI Testing. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 863--866. Google Scholar
Digital Library
- Inês Coimbra Morgado. 2016. Automated Pattern-Based Testing of Mobile Applications. Ph.D. Faculdade de Engenharia da Universidade do Porto.Google Scholar
- Glenford J. Myers and Corey Sandler. 2004. The Art of Software Testing. John Wiley & Sons. Google Scholar
Digital Library
- David Navarre, Philippe A. Palanque, Fabio Paternò, Carmen Santoro, and Rémi Bastide. 2001. A Tool Suite for Integrating Task and System Models Through Scenarios. In Proceedings of the 8th International Workshop on Interactive Systems: Design, Specification, and Verification-Revised Papers (DSV-IS '01). Springer-Verlag, London, UK, UK, 88--113. http://dl.acm.org/citation.cfm?id=646166.680841 Google Scholar
Digital Library
- Bao N. Nguyen, Bryan Robbins, Ishan Banerjee, and Atif Memon. 2014. GUITAR: an innovative tool for automated testing of GUI-driven software. Automated Software Engineering 21, 1 (2014), 65--105. Google Scholar
Digital Library
- Simeon Ntafos. 1998. On Random and Partition Testing. In Proceedings of the 1998 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA '98). ACM, New York, NY, USA, 42--48. Google Scholar
Digital Library
- Ana C. R. Paiva, João C. P. Faria, and Pedro M. C. Mendes. 2008. Reverse Engineered Formal Models for GUI Testing. Springer Berlin Heidelberg, Berlin, Heidelberg, 218--233. Google Scholar
Digital Library
- Fabio Paternò. 2004. ConcurTaskTrees: An Engineered Notation for Task Models. In The Handbook of Task Analysis for Human-Computer Interaction, Dan Diaper and Neville Stanton (Eds.). Lawrence Erlbaum Associates, Chapter 24, 483--501.Google Scholar
- Carmen Santoro. 2005. A task model-based approach for design and evaluation of innovative user interfaces. Ph.D. Presses universitaires de Louvain.Google Scholar
- J.C. Silva, C. Silva, R. Goncalo, J. Saraiva, and J.C. Campos. 2010. The GUISurfer tool: towards a language independent approach to reverse engineering GUI code. In Proceedings of the 2nd ACM SIGCHI Symposium on Engineering interactive computing systems. ACM, 181--186. ISBN: 978--1--4503-0083--4. Google Scholar
Digital Library
- J. L. Silva, J. C. Campos, and A. Paiva. 2008. Model-based user interface testing with Spec Explorer and ConcurTaskTrees. Electronic Notes in Theoretical Computer Science 208 (2008), 77--93. Google Scholar
Digital Library
- Wan-Hui Tseng and Chin-Feng Fan. 2013. Systematic Scenario Test Case Generation for Nuclear Safety Systems. Inf. Softw. Technol. 55, 2 (Feb. 2013), 344--356. Google Scholar
Digital Library
- Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A Taxonomy of Model-based Testing Approaches. Software: Testing, Verification and Reliability 22, 5 (Aug. 2012), 297--312. Google Scholar
Digital Library
Index Terms
A More Intelligent Test Case Generation Approach through Task Models Manipulation
Recommendations
Regression test suite prioritization using system models
During regression testing, a modified system is often retested using an existing test suite. Since the size of the test suite may be very large, testers are interested in detecting faults in the modified system as early as possible during this retesting ...
Scenario-Based Test Case Generation Using Event-B Models
VALID '09: Proceedings of the 2009 First International Conference on Advances in System Testing and Validation LifecycleIn this paper we present an extension of the previously reported model-based testing approach that is based on formal models and user-provided testing scenarios. In this approach, the user provides a testing scenario on the level of an abstract model. ...
Efficient Reduction of Model-Based Generated Test Suites through Test Case Pair Prioritization
MODEVVA '10: Proceedings of the 2010 Workshop on Model-Driven Engineering, Verification, and ValidationDuring the development and maintenance of software, test suites often reach a size that exceeds the costs allocated for test suite execution. In such a case, the test suite needs to be reduced. Many papers are dedicated to the problem of test suite ...






Comments