Abstract
High-coverage testing is challenging. Modern object-oriented programs present additional challenges for testing. One key difficulty is the generation of proper method sequences to construct desired objects as method parameters. In this paper, we cast the problem as an instance of program synthesis that automatically generates candidate programs to satisfy a user-specified intent. In our setting, candidate programs are method sequences, and desired object states specify an intent. Automatic generation of desired method sequences is difficult due to its large search space---sequences often involve methods from multiple classes and require specific primitive values. This paper introduces a novel approach, called Seeker, to intelligently navigate the large search space. Seeker synergistically combines static and dynamic analyses: (1) dynamic analysis generates method sequences to cover branches; (2) static analysis uses dynamic analysis information for not-covered branches to generate candidate sequences; and (3) dynamic analysis explores and eliminates statically generated sequences. For evaluation, we have implemented Seeker and demonstrate its effectiveness on four subject applications totalling 28K LOC. We show that Seeker achieves higher branch coverage and def-use coverage than existing state-of-the-art approaches. We also show that Seeker detects 34 new defects missed by existing tools.
- S. Anand, C. S. Pasareanu, and W. Visser. JPF-SE: A symbolic execution extension to Java PathFinder. In Proc. TACAS, pages 134--138, 2007. Google Scholar
Digital Library
- L. Baresi and M. Miraz. Testful: automatic unit-test generation for Java classes. In Proc. ICSE, pages 281--284, 2010. Google Scholar
Digital Library
- C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In Proc. ISSTA, pages 123--133, 2002. Google Scholar
Digital Library
- J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. In Proc. ASE, pages 443--446, 2008. Google Scholar
Digital Library
- U. Buy, A. Orso, and M. Pezze. Automated testing of classes. In Proc. ISSTA, pages 39--48, 2000. Google Scholar
Digital Library
- C. Cadar, D. Dunbar, and D. Engler. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In Proc. OSDI, pages 209--224, 2008. Google Scholar
Digital Library
- L. Clarke. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng., 2(3):215--222, 1976. Google Scholar
Digital Library
- C. Csallner and Y. Smaragdakis. JCrasher: An automatic robustness tester for Java. Softw. Pract. Exper., 34(11):1025--1050, 2004. Google Scholar
Digital Library
- A. Cypher, D. C. Halbert, D. Kurlander, H. Lieberman, D. Maulsby, B. A. Myers, and A. Turransky, editors. Watch what I do: Programming by demonstration. MIT Press, Cambridge, MA, USA, 1993. Google Scholar
Digital Library
- P. G. Frankl and E. J. Weyuker. An applicable family of data flow testing criteria. IEEE Trans. Softw. Eng., 14(10):1483--1498, 1988. Google Scholar
Digital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proc. PLDI, pages 213--223, 2005. Google Scholar
Digital Library
- S. Gulwani. Dimensions in program synthesis. In Proc. PPDP, pages 13--24, 2010. Google Scholar
Digital Library
- S. Gulwani, S. Jha, A. Tiwari, and R. Venkatesan. Synthesis of loop-free programs. In Proc. PLDI (to appear), 2011. Google Scholar
Digital Library
- M. Harman and P. McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Trans. Softw. Eng., 36:226--247, March 2010. Google Scholar
Digital Library
- K. Inkumsah and T. Xie. Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution. In Proc. ASE, pages 297--306, 2008. Google Scholar
Digital Library
- H. Jaygarl, S. Kim, T. Xie, and C. K. Chang. OCAT: object capture-based automated testing. In Proc. ISSTA, pages 159--170, 2010. Google Scholar
Digital Library
- Parasoft. Jtest manuals version 5.1. Online manual, 2006. http://www.parasoft.com.Google Scholar
- C. Kelleher and R. Pausch. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv., 37:83--137, June 2005. Google Scholar
Digital Library
- J. C. King. Symbolic Execution and Program Testing. Communications of the ACM, 19(7):385--394, 1976. Google Scholar
Digital Library
- B. Korel. Dynamic method for software test data generation. Software Testing, Verification and Reliability, 2(3):203--213, 1992.Google Scholar
Cross Ref
- B. Korel. Automated test generation for programs with procedures. In Proc. ISSTA, pages 209--215, 1996. Google Scholar
Digital Library
- S. Koushik, M. Darko, and A. Gul. CUTE: A concolic unit testing engine for C. In Proc. ESEC/FSE, pages 263--272, 2005. Google Scholar
Digital Library
- K. Lakhotia, M. Harman, and P. McMinn. Handling dynamic data structures in search based testing. In Proc. GECCO, pages 1759--1766, 2008. Google Scholar
Digital Library
- K. Lakhotia, P. McMinn, and M. Harman. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. J. Syst. Softw., 83:2379--2391, December 2010. Google Scholar
Digital Library
- H. Lieberman, editor. Your wish is my command: Programming by example. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2001.Google Scholar
- B. Liskov and J. Guttag. Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2000. Google Scholar
Digital Library
- G. Little and R. C. Miller. Translating keyword commands into executable code. In Proc. UIST, pages 135--144, 2006. Google Scholar
Digital Library
- H. Liu and H. Lieberman. Programmatic semantics for natural language interfaces. In Proc. CHI, pages 1597--1600, 2005. Google Scholar
Digital Library
- D. Mandelin, L. Xu, R. Bodik, and D. Kimelman. Jungloid mining: helping to navigate the API jungle. In Proc. PLDI, pages 48--61, 2005. Google Scholar
Digital Library
- M. R. Marri, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. An empirical study of testing file-system-dependent software with mock objects. In Proc. AST, pages 149--153, 2009.Google Scholar
Cross Ref
- V. Martena, A. Orso, and M. Pezzè. Interclass testing of object oriented software. In Proc. ICECCE, pages 145--154, 2002. Google Scholar
Digital Library
- P. McMinn and M. Holcombe. Evolutionary testing of state-based programs. In Proc. GECCO, pages 1013--1020, 2005. Google Scholar
Digital Library
- A. Mockus, N. Nagappan, and T. T. Dinh-Trong. Test coverage and post-verification defects: A multiple case study. In Proc. ESEM, pages 291--301, 2009. Google Scholar
Digital Library
- A. Orso and B. Kennedy. Selective capture and replay of program executions. SIGSOFT Softw. Eng. Notes, 30(4):1--7, 2005. Google Scholar
Digital Library
- C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proc. ICSE, pages 75--84, 2007. Google Scholar
Digital Library
- QuickGraph: A 100% C# graph library with Graphviz Support, Version 1.0, 2008. http://www.codeproject.com/KB/miscctrl/quickgraph.aspx.Google Scholar
- D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, N. James, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In Proc. ICISS, pages 1--25, 2008. Google Scholar
Digital Library
- S. Thummalapenta, J. de Halleux, N. Tillmann, and S. Wadsworth. DyGen: Automatic generation of high-coverage tests via mining gigabytes of dynamic traces. In Proc. TAP, pages 77--93, 2010. Google Scholar
Digital Library
- S. Thummalapenta and T. Xie. PARSEWeb: A programmer assistant for reusing open source code on the web. In Proc. ASE, pages 204--213, 2007. Google Scholar
Digital Library
- S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. MSeqGen: Object-oriented unit-test generation via mining source code. In Proc. ESEC/FSE, pages 193--202, 2009. Google Scholar
Digital Library
- N. Tillmann and J. de Halleux. Pex - white box test generation for .NET. In Proc. TAP, pages 134--153, 2008. Google Scholar
Digital Library
- N. Tillmann and W. Schulte. Parameterized Unit Tests. In Proc. ESEC/FSE, pages 253--262, 2005. Google Scholar
Digital Library
- P. Tonella. Evolutionary testing of classes. In Proc. ISSTA, pages 119--128, 2004. Google Scholar
Digital Library
- X. Xiao, T. Xie, N. Tillmann, and J. de Halleux. Precise identification of problems for structural test generation. In Proc. ICSE, pages 611--620, 2011. Google Scholar
Digital Library
- T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Proc. TACAS, pages 365--381, 2005. Google Scholar
Digital Library
- T. Xie, N. Tillmann, P. de Halleux, and W. Schulte. Fitness-guided path exploration in dynamic symbolic execution. In Proc. DSN, pages 359--368, 2009.Google Scholar
Cross Ref
- Z3: An efficient theorem prover, 2005. http://research.microsoft.com/en-us/um/redmond/projects/z3/.Google Scholar
- W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with MUT-aware sequence recommendation. In Proc. ASE, pages 293--296, 2010. Google Scholar
Digital Library
Index Terms
Synthesizing method sequences for high-coverage testing
Recommendations
Synthesizing method sequences for high-coverage testing
OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applicationsHigh-coverage testing is challenging. Modern object-oriented programs present additional challenges for testing. One key difficulty is the generation of proper method sequences to construct desired objects as method parameters. In this paper, we cast ...
Automated coverage calculation and test case generation
SAICSIT '12: Proceedings of the South African Institute for Computer Scientists and Information Technologists ConferenceThis article describes the use of symbolic execution, a formal method of static analysis, to calculate code coverage of a program's existing JUnit test suites. Code coverage is measured with respect to a number of test adequacy criteria, including ...
Probabilistic Balancing of Fault Coverage and Test Cost in Combined Built-In Self-Test/Automated Test Equipment Testing Environment
DFT '04: Proceedings of the Defect and Fault Tolerance in VLSI Systems, 19th IEEE International SymposiumAs design and test complexities of SoCs ever intensify, the balanced utilization of combined Built-In Self-Test (BIST) and Automated Test Equipment (ATE) testing becomes desirable to meet the required minimum fault-coverage while maintaining acceptable ...







Comments