skip to main content
research-article
Open Access

Model-based, Mutation-driven Test-case Generation Via Heuristic-guided Branching Search

Published:25 January 2019Publication History
Skip Abstract Section

Abstract

This work introduces a heuristic-guided branching search algorithm for model-based, mutation-driven test-case generation. The algorithm is designed towards the efficient and computationally tractable exploration of discrete, non-deterministic models with huge state spaces. Asynchronous parallel processing is a key feature of the algorithm. The algorithm is inspired by the successful path planning algorithm Rapidly exploring Random Trees (RRT). We adapt RRT in several aspects towards test-case generation. Most notably, we introduce parametrized heuristics for start and successor state selection, as well as a mechanism to construct test cases from the data produced during the search.

We implemented our algorithm in the existing test-case generation framework MoMuT. We present an extensive evaluation of the proposed heuristics and parameters of the algorithm, based on a diverse set of demanding models obtained in an industrial context. In total, we continuously utilized 128 CPU cores on three servers for several weeks to gather the experimental data presented. We show that branching search works well and the use of multiple heuristics is justified. With our new algorithm, we are now able to process models consisting of over 2,300 concurrent objects. To our knowledge, there is no other mutation-driven test-case generation tool that is able to process models of this magnitude.

References

  1. Jean-Raymond Abrial. 2010. Modeling in Event-B—System and Software Engineering. Cambridge University Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. B. Aichernig, H. Brandl, E. Jöbstl, W. Krenn, R. Schlick, and S. Tiran. 2015. MoMuT::UML model-based mutation testing for UML. In Proceedings of the 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST’15).Google ScholarGoogle Scholar
  3. Bernhard K. Aichernig, Jakob Auer, Elisabeth Jöbstl, Robert Korošec, Willibald Krenn, Rupert Schlick, and Birgit Vera Schmidt. 2014. Model-based mutation testing of an industrial measurement device. In Proceedings of the International Conference on Tests and Proofs. Springer, 1--19.Google ScholarGoogle ScholarCross RefCross Ref
  4. Bernhard K. Aichernig, Harald Brandl, Elisabeth Jöbstl, Willibald Krenn, Rupert Schlick, and Stefan Tiran. 2015. Killing strategies for model-based mutation testing. Softw. Test., Verif. Reliab. 25, 8 (2015), 716--748. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Bernhard K. Aichernig, Elisabeth Jöbstl, and Stefan Tiran. 2015. Model-based mutation testing via symbolic refinement checking. Sci. Comput. Program. 97 (2015), 383--404. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Baris Akgun and Mike Stilman. 2011. Sampling heuristics for optimal motion planning in high dimensions. In Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’11). IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  7. James H. Andrews, Lionel C. Briand, and Yvan Labiche. 2005. Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th International Conference on Software Engineering (ICSE’05), Gruia-Catalin Roman, William G. Griswold, and Bashar Nuseibeh (Eds.). ACM, 402--411. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. John Aycock. 2003. A brief history of just-in-time. ACM Comput. Surv. 35, 2 (2003), 97--113. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Ralph-Johan Back and Reino Kurki-Suonio. 1983. Decentralization of process nets with centralized control. In Proceedings of the 2nd Annual ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing (PODC’83). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Marcello M. Bonsangue, Joost N. Kok, and Kaisa Sere. 1998. An approach to object-orientation in action systems. In Proceedings of the Mathematics of Program Construction (MPC’98), Johan Jeuring (Ed.), Vol. 1422. Springer, 68--95. http://link.springer.de/link/service/series/0558/bibs/1422/14220068.htm. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Michael S. Branicky, Michael M. Curtiss, Joshua A. Levine, and Stuart B. Morgan. 2003. RRTs for nonlinear, discrete, and hybrid planning and control. In Proceedings of the 42nd IEEE Conference on Decision and Control, Vol. 1. IEEE, 657--663.Google ScholarGoogle Scholar
  12. Manfred Broy, Bengt Jonsson, Joost-Pieter Katoen, Martin Leucker, and Alexander Pretschner. 2005. Model-based Testing of Reactive Systems: Advanced Lectures. Vol. 3472. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Timothy A. Budd, Richard J. Lipton, Richard A. DeMillo, and Frederick G. Sayward. 1979. Mutation Analysis.Technical Report. DTIC Document.Google ScholarGoogle Scholar
  14. Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive random testing: The art of test-case diversity. J. Syst. Softw. 83, 1 (2010), 60--66. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Tsun S. Chow. 1978. Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4, 3 (1978). Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Ilinca Ciupa, Andreas Leitner, Manuel Oriol, and Bertrand Meyer. 2008. ARTOO: Adaptive random testing for object-oriented software. In Proceedings of the 30th International Conference on Software Engineering. ACM, 71--80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Arilo C. Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H. Travassos. 2007. A survey on model-based testing approaches: A systematic review. In Proceedings of the Workshop on Empirical Assessment of Software Engineering Languages and Technologies. 31--36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Edsger W. Dijkstra. 1975. Guarded commands, nondeterminacy and formal derivation of programs. Commun. ACM 18, 8 (1975), 453--457. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Tommaso Dreossi, Thao Dang, Alexandre Donzé, James Kapinski, Xiaoqing Jin, and Jyotirmoy V. Deshmukh. 2015. Efficient guiding strategies for testing of temporal properties of hybrid systems. In Proceedings of the NASA Formal Methods Symposium. Springer, 127--142.Google ScholarGoogle Scholar
  20. Andreas Fellner, Willibald Krenn, Rupert Schlick, Thorsten Tarrach, and Georg Weissenbacher. 2017. Model-based, mutation-driven test case generation via heuristic-guided branching search. In Proceedings of the 15th ACM-IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE’17). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Dave Ferguson, Nidhi Kalra, and Anthony Stentz. 2006. Replanning with RRTs. In Proceedings of the 2006 IEEE International Conference on Robotics and Automation (ICRA'06). IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  22. Gordon Fraser and Andrea Arcuri. 2011. Evolutionary generation of whole test suites. In Proceedings of the 11th International Conference on Quality Software (QSIC’11), Manuel Núñez, Robert M. Hierons, and Mercedes G. Merayo (Eds.). IEEE Computer Society, 31--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Gordon Fraser and Andreas Zeller. 2012. Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38, 2 (2012), 278--292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Matthew J. Gallagher and V. Lakshmi Narasimhan. 1997. Adtest: A test data generation suite for ada software systems. IEEE Trans. Softw. Eng. 23, 8 (1997), 473--484. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In ACM Sigplan Notices, Vol. 40. ACM, 213--223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Christoph Hilken and Jan Peleska. 2015. Model-based testing against complex SysML models. In Proceedings of the Formal Modeling and Verification of Cyber-Physical Systems, and the 1st International Summer School on Methods and Tools for the Design of Digital Systems, Rolf Drechsler and Ulrich Kühne (Eds.). Springer, 284--286.Google ScholarGoogle Scholar
  27. William E. Howden. 1982. Weak mutation testing and completeness of test sets. IEEE Trans. Softw. Eng. 8, 4 (1982). Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Leonard Jaillet, Juan Cortés, and Thierry Siméon. 2008. Transition-based RRT for path planning in continuous cost spaces. In Proceedings of the 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  29. Claude Jard and Thierry Jéron. 2005. TGV: Theory, principles and algorithms. Int. J. Softw. Tools Technol. Transf. 7, 4 (2005), 297--315. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE . Softw. Eng. 37, 5 (2011), 649--678. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5487526. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE-22). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Bogdan Korel. 1990. Automated software test data generation. IEEE Trans. Softw. Eng. 16, 8 (1990). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Willibald Krenn, Rupert Schlick, and Bernhard K. Aichernig. 2009. Mapping UML to labeled transition systems for test-case generation: A translation via object-oriented action systems. In Proceedings of the 8th International Conference on Formal Methods for Components and Objects (FMCO’09). Springer-Verlag, Berlin, 186--207. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. James J. Kuffner and Steven M. LaValle. 2000. RRT-connect: An efficient approach to single-query path planning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’00), Vol. 2. IEEE, 995--1001.Google ScholarGoogle Scholar
  35. Chris Lattner and Vikram S. Adve. 2004. LLVM: A compilation framework for lifelong program analysis 8 transformation. In Proceedings of the 2nd IEEE / ACM International Symposium on Code Generation and Optimization (CGO’04). Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Steven M. LaValle. 1998. Rapidly-Exploring Random Trees: A New Tool for Path Planning. Technical Report.Google ScholarGoogle Scholar
  37. Lei Ma, Cyrille Artho, Cheng Zhang, Hiroyuki Sato, Johannes Gmeiner, and Rudolf Ramler. 2015. GRT: Program-analysis-guided random testing (T). In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE’15). 212--223.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Jan Malburg and Gordon Fraser. 2011. Combining search-based and constraint-based testing. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering (ASE’11) (ASE), Perry Alexander, Corina S. Pasareanu, and John G. Hosking (Eds.). IEEE Computer Society, 436--439. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Henry B. Mann and Donald R. Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 1 (1947), 50--60. http://www.jstor.org/stable/2236101.Google ScholarGoogle ScholarCross RefCross Ref
  40. Phil McMinn. 2004. Search-based software test data generation: A survey. Softw. Test. Verif. Reliabil. 14, 2 (2004), 105--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Carlos Pacheco and Michael D. Ernst. 2007. Randoop: Feedback-directed random testing for Java. In Proceedings of the Companion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’07). 815--816. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). IEEE Computer Society, 75--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Alexander Pretschner, Wolfgang Prenninger, Stefan Wagner, Christian Kühnel, Martin Baumgartner, Bernd Sostawa, Rüdiger Zölch, and Thomas Stauner. 2005. One evaluation of model-based testing and its automation. In Proceedings of the 27th International Conference on Software Engineering (ICSE’05). ACM, 392--401. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Klaus Reichl, Tomas Fischer, and Peter Tummeltshammer. 2016. Using formal methods for verification and validation in railway. In Proceedings of the 10th International Conference onTests and Proofs (TAP’16), Bernhard K. Aichernig and Carlo A. Furia (Eds.), Lecture Notes in Computer Science, Vol. 9762. Springer.Google ScholarGoogle ScholarCross RefCross Ref
  45. Manoranjan Satpathy, Anand Yeolekar, and S. Ramesh. 2008. Randomized directed testing (REDIRECT) for Simulink/Stateflow models. In Proceedings of the 8th ACM International Conference on Embedded Software. ACM, 217--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Muhammad Shafique and Yvan Labiche. 2010. A Systematic Review of Model Based Testing Tool Support. Carleton University, Canada, Tech. Rep. Technical Report SCE-10-04 (2010), 01--21.Google ScholarGoogle Scholar
  47. Muhammad Shafique and Yvan Labiche. 2015. A systematic review of state-based test tools. Softw. Tools Technol. Transf. 17, 1 (2015), 59--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Nigel Tracey, John Clark, Keith Mander, and John McDermid. 1998. An automated framework for structural test-data generation. In Proceedings of the 13th IEEE International Conference on Automated Software Engineering (ASE’98). IEEE. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Gerrit Jan Tretmans. 1992. A Formal Approach to Conformance Testing. Ph.D. Dissertation. University of Twente, Enschede, Netherlands.Google ScholarGoogle Scholar
  50. Jan Tretmans. 1996. Test generation with inputs, outputs and repetitive quiescence. Softw. Concept. Tools 17, 3 (1996), 103--120.Google ScholarGoogle Scholar
  51. Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliabil. 22, 5 (2012), 297--312. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Mattia Vivanti, Andre Mis, Alessandra Gorla, and Gordon Fraser. 2013. Search-based data-flow test generation. In Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE’13). IEEE, 370--379.Google ScholarGoogle ScholarCross RefCross Ref
  53. Joachim Wegener, Kerstin Buhr, and Hartmut Pohlheim. 2002. Automatic test data generation for structural testing of embedded software systems by evolutionary testing. In Proceedings of the Annual Conference on Genetic and Evolutionary Computation (GECCO’02), Vol. 2. 1233--1240. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Yuan Zhan and John A. Clark. 2005. Search-based mutation testing for simulink models. In Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation (GECCO’05). ACM, New York, NY, 1061--1068. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Model-based, Mutation-driven Test-case Generation Via Heuristic-guided Branching Search

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!