skip to main content
research-article

Random testing for higher-order, stateful programs

Published:17 October 2010Publication History
Skip Abstract Section

Abstract

Testing is among the most effective tools available for finding bugs. Still, we know of no automatic technique for generating test cases that expose bugs involving a combination of mutable state and callbacks, even though objects and method overriding set up exactly that combination. For such cases, a test generator must create callbacks or subclasses that aggressively exercise side-effecting operations using combinations of generated objects.

This paper presents a new algorithm for randomly testing programs that use state and callbacks. Our algorithm exploits a combination of contracts and environment bindings to guide the test-case generator toward interesting inputs. Our prototype implementation for Racket (formerly PLT Scheme) - which has a Java-like class system, but with first-class classes as well as gbeta-like augmentable methods - uncovered dozens of bugs in a well-tested and widely used text-editor library.

We describe our approach in a precise, formal notation, borrowing the techniques used to describe operational semantics and type systems. The formalism enables us to provide a compact and self-contained explanation of the core of our technique without the ambiguity usually present in pseudo-code descriptions.

References

  1. }}Mike Barnett, Manuel Fahndrich, Peli de Halleux, Francesco Logozzo, and Nikolai Tillmann. Exploiting the Synergy between Automated-Test-Generation and Programming-by-Contract. In Proc. Intl. Conf. Soft. Eng.: Companion Volume, pp. 401--402, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  2. }}Cristian Cadar, Vijay Ganesh, Peter M. Pawlowski, David L. Gill, and Dawson R. Engler. EXE: Automatically Generating Inputs of Death. In Proc. ACM Conf. Computer and Communications Security, pp. 322--335, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. }}Koen Claessen and John Hughes. Testing Monadic Code with QuickCheck. In Proc. ACM SIGPLAN Haskell Wksp., pp. 47--59, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. }}Koen Claessen and John Hughes. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proc. ACM Intl. Conf. Functional Programming, pp. 268--279, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. }}Cristoph Csallner and Yannis Smaragdakis. JCrasher: An Automatic Robustness Tester for Java. Software - Practice & Experience 34(11), pp. 1025--1050, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. }}Cristoph Csallner and Yannis Smaragdakis. DSD-Crasher: A Hybrid Analysis Tool for Bug Finding. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 245--254, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. }}Cristoph Csallner and Yannis Smaragdakis. Check 'n' Crash: Combining Static Checking and Testing. In Proc. Intl. Conf. Soft. Eng., pp. 422--431, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. }}Xianghua Deng, Robby, and John Hatcliff. Kiasan/KUnit: Automatic Test Case Generation and Analysis Feedback for Open Object-Oriented Systems. In Proc. Testing: Academia & Industry Conference - Practice & Research Techniques, pp. 3--12, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. }}Erik Ernst. gbeta - a Language with Virtual Attributes, Block Structure, and Propagating, Dynamic Inheritance. PhD dissertation, Department of Computer Science, University of Aarhus, Denmark, 1999.Google ScholarGoogle Scholar
  10. }}Matthias Felleisen, Robert Bruce Findler, and Matthew Flatt. Semantics Engineering with PLT Redex. MIT Press, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. }}Robert Bruce Findler, John Clements, Cormac Flanagan, Matthew Flatt, Shriram Krishnamurthi, Paul Steckler, and Matthias Felleisen. DrScheme: A programming environment for Scheme. J. Functional Programming 2(12), pp. 159--182, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. }}Robert Bruce Findler and Matthias Felleisen. Contracts for Higher-Order Functions. In Proc. ACM Intl. Conf. Functional Programming, pp. 48--59, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. }}Cormac Flanagan, K. Rustan M. Leino, Mark Lillibridge, Greg Nelson, James B. Saxe, and Raymie Stata. Extended Static Checking for Java. In Proc. ACM Conf. Programming Language Design and Implementation, pp. 234--245, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. }}Matthew Flatt and Eli Barzilay. Keyword and Optional Arguments in PLT Scheme. In Proc. Scheme and Functional Programming, pp. 94--102, 2009.Google ScholarGoogle Scholar
  15. }}Matthew Flatt, Robert Bruce Findler, and Matthias Felleisen. Scheme with Classes, Mixins, and Traits (invited tutorial). In Proc. Asian Symp. Programming Languages and Systems, pp. 270--289, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. }}Matthew Flatt, Robert Bruce Findler, and PLT. Guide: Racket. 2009. http://docs.racket-lang.org/guide.Google ScholarGoogle Scholar
  17. }}Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns. Addison-Wesley Publishing Company, 1994.Google ScholarGoogle Scholar
  18. }}Patrice Godefroid, Nils Karlund, and Koushik Sen. DART: Directed Automated Random Testing. In Proc. ACM Conf. Programming Language Design and Implementation, pp. 213--223, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. }}Phillip Heidegger and Peter Thiemann. JSConTest - Contract-Driven Testing of JavaScript Code. In Proc. Intl. Conf. Objects, Models, Components, Patterns, pp. 154--172, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. }}Kobi Inkumsah and Tao Xie. Improving Structural Testing of Object-Oriented Programs via Integrated Evolutionary Testing and Symbolic Execution. In Proc. IEEE/ACM Intl. Conf. Automated Soft. Eng., pp. 297--306, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. }}Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-Directed Random Test Generation. In Proc. Intl. Conf. Soft. Eng., pp. 75--84, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. }}B.S. Rubin, A. R. Christ, and K. A. Bohrer. Java and the IBM San Franscisco project. IBM Systems Journal 37(3), pp. 365--371, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. }}Koushik Sen and Gul Agha. CUTE and jCUTE: Unit Testing and Explicit Path Model-Checking Tools. In Proc. Intl. Conf. Computer Aided Verification, pp. 419--423, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. }}T. Stephen Strickland and Matthias Felleisen. Contracts for First-Class Classes. In Proc. Dynamic Languages Symposium, 2010. To Appear. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. }}Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. MSeqGen: Object-Oriented Unit-Test Generation via Mining Source Code. In Proc. Joint Euro. Soft. Eng. Conf. and ACM Symp. Foundations of Soft. Eng., pp. 193--202, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. }}Nikolai Tillmann and Jonathan de Halleux. Pex -- White Box Test Generation for .NET. In Proc. Intl. Conf. Tests and Proofs, pp. 134--153, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. }}Paolo Tonella. Evolutionary Testing of Classes. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 119--128, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. }}Willem Visser, Corina S. Pasareanu, and Radek Pelanek. Test Input Generation for Java Containers using State Matching. In Proc. Intl. Symp. Soft. Testing and Analysis, pp. 37--48, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. }}Tai Xie, Darko Marinov, and David Notkin. Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests. In Proc. IEEE Intl. Conf. Automated Soft. Eng., pp. 196--205, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. }}Tai Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proc. Intl. Conf. Tools and Algorithms for the Construction and Analysis of Systems, pp. 365--381, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Random testing for higher-order, stateful programs

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!