Abstract
Testing is among the most effective tools available for finding bugs. Still, we know of no automatic technique for generating test cases that expose bugs involving a combination of mutable state and callbacks, even though objects and method overriding set up exactly that combination. For such cases, a test generator must create callbacks or subclasses that aggressively exercise side-effecting operations using combinations of generated objects.
This paper presents a new algorithm for randomly testing programs that use state and callbacks. Our algorithm exploits a combination of contracts and environment bindings to guide the test-case generator toward interesting inputs. Our prototype implementation for Racket (formerly PLT Scheme) - which has a Java-like class system, but with first-class classes as well as gbeta-like augmentable methods - uncovered dozens of bugs in a well-tested and widely used text-editor library.
We describe our approach in a precise, formal notation, borrowing the techniques used to describe operational semantics and type systems. The formalism enables us to provide a compact and self-contained explanation of the core of our technique without the ambiguity usually present in pseudo-code descriptions.
- }}Mike Barnett, Manuel Fahndrich, Peli de Halleux, Francesco Logozzo, and Nikolai Tillmann. Exploiting the Synergy between Automated-Test-Generation and Programming-by-Contract. In Proc. Intl. Conf. Soft. Eng.: Companion Volume, pp. 401--402, 2009.Google Scholar
Cross Ref
- }}Cristian Cadar, Vijay Ganesh, Peter M. Pawlowski, David L. Gill, and Dawson R. Engler. EXE: Automatically Generating Inputs of Death. In Proc. ACM Conf. Computer and Communications Security, pp. 322--335, 2006. Google Scholar
Digital Library
- }}Koen Claessen and John Hughes. Testing Monadic Code with QuickCheck. In Proc. ACM SIGPLAN Haskell Wksp., pp. 47--59, 2002. Google Scholar
Digital Library
- }}Koen Claessen and John Hughes. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proc. ACM Intl. Conf. Functional Programming, pp. 268--279, 2000. Google Scholar
Digital Library
- }}Cristoph Csallner and Yannis Smaragdakis. JCrasher: An Automatic Robustness Tester for Java. Software - Practice & Experience 34(11), pp. 1025--1050, 2004. Google Scholar
Digital Library
- }}Cristoph Csallner and Yannis Smaragdakis. DSD-Crasher: A Hybrid Analysis Tool for Bug Finding. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 245--254, 2006. Google Scholar
Digital Library
- }}Cristoph Csallner and Yannis Smaragdakis. Check 'n' Crash: Combining Static Checking and Testing. In Proc. Intl. Conf. Soft. Eng., pp. 422--431, 2005. Google Scholar
Digital Library
- }}Xianghua Deng, Robby, and John Hatcliff. Kiasan/KUnit: Automatic Test Case Generation and Analysis Feedback for Open Object-Oriented Systems. In Proc. Testing: Academia & Industry Conference - Practice & Research Techniques, pp. 3--12, 2007. Google Scholar
Digital Library
- }}Erik Ernst. gbeta - a Language with Virtual Attributes, Block Structure, and Propagating, Dynamic Inheritance. PhD dissertation, Department of Computer Science, University of Aarhus, Denmark, 1999.Google Scholar
- }}Matthias Felleisen, Robert Bruce Findler, and Matthew Flatt. Semantics Engineering with PLT Redex. MIT Press, 2009. Google Scholar
Digital Library
- }}Robert Bruce Findler, John Clements, Cormac Flanagan, Matthew Flatt, Shriram Krishnamurthi, Paul Steckler, and Matthias Felleisen. DrScheme: A programming environment for Scheme. J. Functional Programming 2(12), pp. 159--182, 2002. Google Scholar
Digital Library
- }}Robert Bruce Findler and Matthias Felleisen. Contracts for Higher-Order Functions. In Proc. ACM Intl. Conf. Functional Programming, pp. 48--59, 2002. Google Scholar
Digital Library
- }}Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. Extended Static Checking for Java. In Proc. ACM Conf. Programming Language Design and Implementation, pp. 234--245, 2002. Google Scholar
Digital Library
- }}Matthew Flatt and Eli Barzilay. Keyword and Optional Arguments in PLT Scheme. In Proc. Scheme and Functional Programming, pp. 94--102, 2009.Google Scholar
- }}Matthew Flatt, Robert Bruce Findler, and Matthias Felleisen. Scheme with Classes, Mixins, and Traits (invited tutorial). In Proc. Asian Symp. Programming Languages and Systems, pp. 270--289, 2006. Google Scholar
Digital Library
- }}Matthew Flatt, Robert Bruce Findler, and PLT. Guide: Racket. 2009. http://docs.racket-lang.org/guide.Google Scholar
- }}Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns. Addison-Wesley Publishing Company, 1994.Google Scholar
- }}Patrice Godefroid, Nils Karlund, and Koushik Sen. DART: Directed Automated Random Testing. In Proc. ACM Conf. Programming Language Design and Implementation, pp. 213--223, 2005. Google Scholar
Digital Library
- }}Phillip Heidegger and Peter Thiemann. JSConTest - Contract-Driven Testing of JavaScript Code. In Proc. Intl. Conf. Objects, Models, Components, Patterns, pp. 154--172, 2010. Google Scholar
Digital Library
- }}Kobi Inkumsah and Tao Xie. Improving Structural Testing of Object-Oriented Programs via Integrated Evolutionary Testing and Symbolic Execution. In Proc. IEEE/ACM Intl. Conf. Automated Soft. Eng., pp. 297--306, 2009. Google Scholar
Digital Library
- }}Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-Directed Random Test Generation. In Proc. Intl. Conf. Soft. Eng., pp. 75--84, 2007. Google Scholar
Digital Library
- }}B.S. Rubin, A. R. Christ, and K. A. Bohrer. Java and the IBM San Franscisco project. IBM Systems Journal 37(3), pp. 365--371, 1998. Google Scholar
Digital Library
- }}Koushik Sen and Gul Agha. CUTE and jCUTE: Unit Testing and Explicit Path Model-Checking Tools. In Proc. Intl. Conf. Computer Aided Verification, pp. 419--423, 2006. Google Scholar
Digital Library
- }}T. Stephen Strickland and Matthias Felleisen. Contracts for First-Class Classes. In Proc. Dynamic Languages Symposium, 2010. To Appear. Google Scholar
Digital Library
- }}Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. MSeqGen: Object-Oriented Unit-Test Generation via Mining Source Code. In Proc. Joint Euro. Soft. Eng. Conf. and ACM Symp. Foundations of Soft. Eng., pp. 193--202, 2009. Google Scholar
Digital Library
- }}Nikolai Tillmann and Jonathan de Halleux. Pex -- White Box Test Generation for .NET. In Proc. Intl. Conf. Tests and Proofs, pp. 134--153, 2008. Google Scholar
Digital Library
- }}Paolo Tonella. Evolutionary Testing of Classes. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 119--128, 2004. Google Scholar
Digital Library
- }}Willem Visser, Corina S. Pasareanu, and Radek Pelanek. Test Input Generation for Java Containers using State Matching. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 37--48, 2006. Google Scholar
Digital Library
- }}Tai Xie, Darko Marinov, and David Notkin. Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests. In Proc. IEEE Intl. Conf. Automated Soft. Eng., pp. 196--205, 2004. Google Scholar
Digital Library
- }}Tai Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proc. Intl. Conf. Tools and Algorithms for the Construction and Analysis of Systems, pp. 365--381, 2004. Google Scholar
Digital Library
Index Terms
Random testing for higher-order, stateful programs
Recommendations
Random testing for higher-order, stateful programs
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applicationsTesting is among the most effective tools available for finding bugs. Still, we know of no automatic technique for generating test cases that expose bugs involving a combination of mutable state and callbacks, even though objects and method overriding ...
The Effectiveness of T-Way Test Data Generation
SAFECOMP '08: Proceedings of the 27th international conference on Computer Safety, Reliability, and SecurityThis paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and <em>t</em>-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...







Comments