skip to main content
research-article

Synthesizing method sequences for high-coverage testing

Published:22 October 2011Publication History
Skip Abstract Section

Abstract

High-coverage testing is challenging. Modern object-oriented programs present additional challenges for testing. One key difficulty is the generation of proper method sequences to construct desired objects as method parameters. In this paper, we cast the problem as an instance of program synthesis that automatically generates candidate programs to satisfy a user-specified intent. In our setting, candidate programs are method sequences, and desired object states specify an intent. Automatic generation of desired method sequences is difficult due to its large search space---sequences often involve methods from multiple classes and require specific primitive values. This paper introduces a novel approach, called Seeker, to intelligently navigate the large search space. Seeker synergistically combines static and dynamic analyses: (1) dynamic analysis generates method sequences to cover branches; (2) static analysis uses dynamic analysis information for not-covered branches to generate candidate sequences; and (3) dynamic analysis explores and eliminates statically generated sequences. For evaluation, we have implemented Seeker and demonstrate its effectiveness on four subject applications totalling 28K LOC. We show that Seeker achieves higher branch coverage and def-use coverage than existing state-of-the-art approaches. We also show that Seeker detects 34 new defects missed by existing tools.

References

  1. S. Anand, C. S. Pasareanu, and W. Visser. JPF-SE: A symbolic execution extension to Java PathFinder. In Proc. TACAS, pages 134--138, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. L. Baresi and M. Miraz. Testful: automatic unit-test generation for Java classes. In Proc. ICSE, pages 281--284, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In Proc. ISSTA, pages 123--133, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. In Proc. ASE, pages 443--446, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. U. Buy, A. Orso, and M. Pezze. Automated testing of classes. In Proc. ISSTA, pages 39--48, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. Cadar, D. Dunbar, and D. Engler. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In Proc. OSDI, pages 209--224, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. L. Clarke. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng., 2(3):215--222, 1976. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. C. Csallner and Y. Smaragdakis. JCrasher: An automatic robustness tester for Java. Softw. Pract. Exper., 34(11):1025--1050, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. A. Cypher, D. C. Halbert, D. Kurlander, H. Lieberman, D. Maulsby, B. A. Myers, and A. Turransky, editors. Watch what I do: Programming by demonstration. MIT Press, Cambridge, MA, USA, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. P. G. Frankl and E. J. Weyuker. An applicable family of data flow testing criteria. IEEE Trans. Softw. Eng., 14(10):1483--1498, 1988. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proc. PLDI, pages 213--223, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. Gulwani. Dimensions in program synthesis. In Proc. PPDP, pages 13--24, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Synthesis of loop-free programs. In Proc. PLDI (to appear), 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. Harman and P. McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Trans. Softw. Eng., 36:226--247, March 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. K. Inkumsah and T. Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In Proc. ASE, pages 297--306, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. H. Jaygarl, S. Kim, T. Xie, and C. K. Chang. OCAT: object capture-based automated testing. In Proc. ISSTA, pages 159--170, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Parasoft. Jtest manuals version 5.1. Online manual, 2006. http://www.parasoft.com.Google ScholarGoogle Scholar
  18. C. Kelleher and R. Pausch. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv., 37:83--137, June 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. J. C. King. Symbolic Execution and Program Testing. Communications of the ACM, 19(7):385--394, 1976. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. B. Korel. Dynamic method for software test data generation. Software Testing, Verification and Reliability, 2(3):203--213, 1992.Google ScholarGoogle ScholarCross RefCross Ref
  21. B. Korel. Automated test generation for programs with procedures. In Proc. ISSTA, pages 209--215, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. S. Koushik, M. Darko, and A. Gul. CUTE: A concolic unit testing engine for C. In Proc. ESEC/FSE, pages 263--272, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. K. Lakhotia, M. Harman, and P. McMinn. Handling dynamic data structures in search based testing. In Proc. GECCO, pages 1759--1766, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. K. Lakhotia, P. McMinn, and M. Harman. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. J. Syst. Softw., 83:2379--2391, December 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. H. Lieberman, editor. Your wish is my command: Programming by example. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2001.Google ScholarGoogle Scholar
  26. B. Liskov and J. Guttag. Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. G. Little and R. C. Miller. Translating keyword commands into executable code. In Proc. UIST, pages 135--144, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. H. Liu and H. Lieberman. Programmatic semantics for natural language interfaces. In Proc. CHI, pages 1597--1600, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. D. Mandelin, L. Xu, R. Bodik, and D. Kimelman. Jungloid mining: helping to navigate the API jungle. In Proc. PLDI, pages 48--61, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. M. R. Marri, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. An empirical study of testing file-system-dependent software with mock objects. In Proc. AST, pages 149--153, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  31. V. Martena, A. Orso, and M. Pezzè. Interclass testing of object oriented software. In Proc. ICECCE, pages 145--154, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. P. McMinn and M. Holcombe. Evolutionary testing of state-based programs. In Proc. GECCO, pages 1013--1020, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. A. Mockus, N. Nagappan, and T. T. Dinh-Trong. Test coverage and post-verification defects: A multiple case study. In Proc. ESEM, pages 291--301, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. A. Orso and B. Kennedy. Selective capture and replay of program executions. SIGSOFT Softw. Eng. Notes, 30(4):1--7, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proc. ICSE, pages 75--84, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. QuickGraph: A 100% C# graph library with Graphviz Support, Version 1.0, 2008. http://www.codeproject.com/KB/miscctrl/quickgraph.aspx.Google ScholarGoogle Scholar
  37. D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, N. James, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In Proc. ICISS, pages 1--25, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. S. Thummalapenta, J. de Halleux, N. Tillmann, and S. Wadsworth. DyGen: Automatic generation of high-coverage tests via mining gigabytes of dynamic traces. In Proc. TAP, pages 77--93, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. S. Thummalapenta and T. Xie. PARSEWeb: A programmer assistant for reusing open source code on the web. In Proc. ASE, pages 204--213, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. MSeqGen: Object-oriented unit-test generation via mining source code. In Proc. ESEC/FSE, pages 193--202, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. N. Tillmann and J. de Halleux. Pex - white box test generation for .NET. In Proc. TAP, pages 134--153, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. N. Tillmann and W. Schulte. Parameterized Unit Tests. In Proc. ESEC/FSE, pages 253--262, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. P. Tonella. Evolutionary testing of classes. In Proc. ISSTA, pages 119--128, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise identification of problems for structural test generation. In Proc. ICSE, pages 611--620, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proc. TACAS, pages 365--381, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. T. Xie, N. Tillmann, P. de Halleux, and W. Schulte. Fitness-guided path exploration in dynamic symbolic execution. In Proc. DSN, pages 359--368, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  47. Z3: An efficient theorem prover, 2005. http://research.microsoft.com/en-us/um/redmond/projects/z3/.Google ScholarGoogle Scholar
  48. W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with MUT-aware sequence recommendation. In Proc. ASE, pages 293--296, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Synthesizing method sequences for high-coverage testing

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM SIGPLAN Notices
              ACM SIGPLAN Notices  Volume 46, Issue 10
              OOPSLA '11
              October 2011
              1063 pages
              ISSN:0362-1340
              EISSN:1558-1160
              DOI:10.1145/2076021
              Issue’s Table of Contents
              • cover image ACM Conferences
                OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applications
                October 2011
                1104 pages
                ISBN:9781450309400
                DOI:10.1145/2048066

              Copyright © 2011 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 22 October 2011

              Check for updates

              Qualifiers

              • research-article

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!