skip to main content
article

Avoiding useless mutants

Published:23 October 2017Publication History
Skip Abstract Section

Abstract

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

References

  1. 2017. PITest Mutation Testing Tool for Java. (2017). http://pitest.org/ Accessed: 2017-05-20.Google ScholarGoogle Scholar
  2. Konstantinos Adamopoulos, Mark Harman, and Robert M Hierons. 2004. How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Genetic and evolutionary computation conference. 1338-1349.Google ScholarGoogle Scholar
  3. Jr. Allen Troy Acree. 1980. On Mutation. Ph.D. Dissertation. Georgia Institute of Technology.Google ScholarGoogle Scholar
  4. Cyrille Artho and Lei Ma. 2016. Classification of randomly generated test cases. In Software Analysis, Evolution, and Reengineering (SANER), IEEE 23rd International Conference on. 29-32.Google ScholarGoogle Scholar
  5. Douglas Baldwin and Frederick Sayward. 1979. Heuristics for Determining Equivalence of Program Mutations. Technical Report. DTIC Document.Google ScholarGoogle Scholar
  6. Timothy Budd and Dana Angluin. 1982. Two notions of correctness and their relation to testing. Acta Informatica 18, 1 (1982), 31-45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Richard DeMillo, Richard Lipton, and Frederick Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34-41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. 416-419. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Bernhard JM Grün, David Schuler, and Andreas Zeller. 2009. The impact of equivalent mutants. In Software Testing, Verification and Validation Workshops. ICSTW. International Conference on. 192-199. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Rob Hierons, Mark Harman, and Sebastian Danicic. 1999. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability 9, 4 (1999), 233-262.Google ScholarGoogle ScholarCross RefCross Ref
  11. Daniel Jackson. 2012. Software Abstractions: Logic, Language, and Analysis. 2nd edition. The MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Daniel Jackson, Ian Schechter, and Hya Shlyahter. 2000. Alcoa: the Alloy constraint analyzer. In Proceedings of the 31st International Conference on Software Engineering (ICSE). IEEE Computer Society, 730-733. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649-678. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. René Just, Darioush Jalali, Laura Inozemtseva, Michael Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654-665. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. René Just, Bob Kurtz, and Paul Ammann. 2017. Inferring Mutant Utility from Program Context. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Rene Just, Franz Schweiggert, and Gregory Kapfhammer. 2011. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering. 612-615. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Marinos Kintis and Nicos Malevris. 2015. MEDIC: A static analysis framework for equivalent mutant identification. Information and Software Technology 68 (2015), 1-17.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Marinos Kintis, Mike Papadakis, Yue Jia, Nicos Malevris, Yves Le Traon, and Mark Harman. 2017. Detecting Trivial Mutant Equivalences via Compiler Optimisations. IEEE Transactions on Software Engineering PP, 99 (2017), 1-1.Google ScholarGoogle Scholar
  19. Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, and Nicos Malevris. 2016. Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In Source Code Analysis and Manipulation (SCAM), 16th International Working Conference on. 147-156.Google ScholarGoogle Scholar
  20. Yu-Seung Ma, Jeff Offutt, and Yong Rae Kwon. 2005. MuJava: An automated class mutation system. Software Testing, Verification and Reliability 15, 2 (2005), 97-133. Google ScholarGoogle ScholarCross RefCross Ref
  21. Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23-42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Melina Mongiovi. 2017. Scaling Testing of Refactoring Engines. Ph.D. Dissertation. Federal University of Campina Grande (UFCG).Google ScholarGoogle Scholar
  23. Melina Mongiovi, Gustavo Mendes, Rohit Gheyi, Gustavo Soares, and Márcio Ribeiro. 2014. Scaling testing of refactoring engines. In Software Maintenance and Evolution (ICSME), IEEE International Conference on. 371-380. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Jeff Offutt, Yu-Seung Ma, and Yong-Rae Kwon. 2006. The Class-level Mutants of MuJava. In Proceedings of the 2006 International Workshop on Automation of Software Test. 78-84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jeff Offutt and Jie Pan. 1996. Detecting equivalent mutants and the feasible path problem. In Computer Assurance, 1996. COMPASS, Systems Integrity. Software Safety. Process Security. Proceedings of the Eleventh Annual Conference on. 224-236.Google ScholarGoogle ScholarCross RefCross Ref
  26. Jeff Offutt and Jie Pan. 1997. Automatically detecting equivalent mutants and infeasible paths. Software Testing, Verification and Reliability 7, 3 (1997), 165-192.Google ScholarGoogle ScholarCross RefCross Ref
  27. Carlos Pacheco, Shuvendu Lahiri, Michael Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering. 75-84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. 936-946. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Corina S. Pasareanu and Neha Rungta. 2010. Symbolic PathFinder: symbolic execution of Java bytecode. In Proceedings of the IEEE/ACM international conference on Automated software engineering. 179-180. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. David Schuler, Valentin Dallmeier, and Andreas Zeller. 2009. Efficient Mutation Testing by Checking Invariant Violations. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis. 69-80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. David Schuler and Andreas Zeller. 2010. (Un-)Covering Equivalent Mutants. In Third International Conference on Software Testing, Verification and Validation. 45-54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. David Schuler and Andreas Zeller. 2013. Covering and Uncovering Equivalent Mutants. Software Testing, Verification and Reliability 23, 5 (2013), 353-374.Google ScholarGoogle ScholarCross RefCross Ref
  33. Gustavo Soares, Rohit Gheyi, and Tiago Massoni. 2013. Automated behavioral testing of refactoring engines. IEEE Transactions on Software Engineering 39, 2 (2013), 147-162. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Gustavo Soares, Rohit Gheyi, Dalton Serey, and Tiago Massoni. 2010. Making program refactoring safer. IEEE software 27, 4 (2010), 52-57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jeffrey Voas and Gary McGraw. 1997. Software fault injection: inoculating programs against errors. John Wiley & Sons, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Avoiding useless mutants

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!