ABSTRACT
Dynamic test generation is a form of dynamic program analysis that attempts to compute test inputs to drive a program along a specific program path. Directed Automated Random Testing, or DART for short, blends dynamic test generation with model checking techniques with the goal of systematically executing all feasible program paths of a program while detecting various types of errors using run-time checking tools (like Purify, for instance). Unfortunately, systematically executing all feasible program paths does not scale to large, realistic programs.This paper addresses this major limitation and proposes to perform dynamic test generation compositionally, by adapting known techniques for interprocedural static analysis. Specifically, we introduce a new algorithm, dubbed SMART for Systematic Modular Automated Random Testing, that extends DART by testing functions in isolation, encoding test results as function summaries expressed using input preconditions and output postconditions, and then re-using those summaries when testing higher-level functions. We show that, for a fixed reasoning capability, our compositional approach to dynamic test generation (SMART) is both sound and complete compared to monolithic dynamic test generation (DART). In other words, SMART can perform dynamic test generation compositionally without any reduction in program path coverage. We also show that, given a bound on the maximum number of feasible paths in individual program functions, the number of program executions explored by SMART is linear in that bound, while the number of program executions explored by DART can be exponential in that bound. We present examples of C programs and preliminary experimental results that illustrate and validate empirically these properties.
- R. Alur, M. Benedikt, K. Etessami, P. Godefroid, T. Reps, and M. Yannakakis. Analysis of Recursive State Machines. TOPLAS, 27(4):786--818, July 2005. Google Scholar
Digital Library
- R. Alur and M. Yannakakis. Model Checking of Hierarchical State Machines. In FSE'98. Google Scholar
Digital Library
- D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Tests from Counterexamples. In ICSE'2004. Google Scholar
Digital Library
- C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In ISSTA'2002. Google Scholar
Digital Library
- W.R. Bush, J.D. Pincus, and D.J. Sielaff. A static analyzer for finding dynamic programming errors. Software Practice and Experience, 30(7):775--802, 2000. Google Scholar
Digital Library
- H. Chen, D. Dean, and D. Wagner. Model Checking One Million Lines of C Code. In NDSS'04.Google Scholar
- C. Cadar and D. Engler. Execution Generated Test Cases: How to Make Systems Code Crash Itself. In SPIN'2005. Google Scholar
Digital Library
- A. Chakrabarti and P. Godefroid. Software Partitioning for Effective Automated Unit Testing. In EMSOFT'2006. Google Scholar
Digital Library
- C. Csallner and Y. Smaragdakis. Check'n Crash: Combining Static Checking and Testing. In ICSE'2005. Google Scholar
Digital Library
- M. Das, S. Lerner, and M. Seigle. ESP: Path-Sensitive Program Verification in Polynomial Time. In PLDI'2002. Google Scholar
Digital Library
- J. Edvardsson. A Survey on Automatic Test Data Generation. In Proceedings of the 2nd Conference on Computer Science and Engineering, pages 21--28, Linkoping, October 1999.Google Scholar
- P. Godefroid and N. Klarlund. Software Model Checking: Searching for Computations in the Abstract or the Concrete (Invited Paper). In IFM'2005. Google Scholar
Digital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In PLDI'2005. Google Scholar
Digital Library
- N. Gupta, A. P. Mathur, and M. L. Soffa. Generating Test Data for Branch Coverage. In ASE'2000. Google Scholar
Digital Library
- P. Godefroid. Model Checking for Programming Languages using VeriSoft. In POPL'97. Google Scholar
Digital Library
- P. Godefroid. The Soundness of Bugs is What Matters (Position Paper). In BUGS'2005.Google Scholar
- S. Hallem, B. Chelf, Y. Xie, and D. Engler. A System and Language for Building System-Specific Static Analyses. In PLDI'2002. Google Scholar
Digital Library
- J. C. King. Symbolic Execution and Program Testing. Journal of the ACM, 19(7):385--394, 1976. Google Scholar
Digital Library
- B. Korel. A Dynamic Approach of Test Data Generation. In IEEE Conference on Software Maintenance, 1990.Google Scholar
Cross Ref
- G. J. Myers. The Art of Software Testing. Wiley, 1979. Google Scholar
Digital Library
- T. Reps, S. Horwitz, and M. Sagiv. Precise interprocedural dataflow analysis via graph reachability. In POPL'95. Google Scholar
Digital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In FSE'2005. Google Scholar
Digital Library
- W. Visser, C. Pasareanu, and S. Khurshid. Test Input Generation with Java PathFinder. In ISSTA'2004. Google Scholar
Digital Library
- T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution. In TACAS'2005. Google Scholar
Digital Library
- J. Yang, C. Sar, P. Twohey, C. Cadar, and D. Engler. Automatically Generating Malicious Disks using Symbolic Execution. In Proceedings of IEEE Security and Privacy'2006, Oakland, 2006. Google Scholar
Digital Library
Index Terms
Compositional dynamic test generation
Recommendations
Compositional dynamic test generation
Proceedings of the 2007 POPL ConferenceDynamic test generation is a form of dynamic program analysis that attempts to compute test inputs to drive a program along a specific program path. Directed Automated Random Testing, or DART for short, blends dynamic test generation with model checking ...
Grammar-based whitebox fuzzing
PLDI '08Whitebox fuzzing is a form of automatic dynamic test generation, based on symbolic execution and constraint solving, designed for security testing of large applications. Unfortunately, the current effectiveness of whitebox fuzzing is limited when ...
Grammar-based whitebox fuzzing
PLDI '08: Proceedings of the 29th ACM SIGPLAN Conference on Programming Language Design and ImplementationWhitebox fuzzing is a form of automatic dynamic test generation, based on symbolic execution and constraint solving, designed for security testing of large applications. Unfortunately, the current effectiveness of whitebox fuzzing is limited when ...






Comments