skip to main content
research-article

Higher-order test generation

Published:04 June 2011Publication History
Skip Abstract Section

Abstract

Symbolic reasoning about large programs is bound to be imprecise. How to deal with this imprecision is a fundamental problem in program analysis. Imprecision forces approximation. Traditional static program verification builds "may" over-approximations of the program behaviors to check universal "for-all-paths" properties, while automatic test generation requires "must" under-approximations to check existential "for-some-path" properties.

In this paper, we introduce a new approach to test generation where tests are derived from validity proofs of first-order logic formulas, rather than satisfying assignments of quantifier-free first-order logic formulas as usual. Two key ingredients of this higher-order test generation are to (1) represent complex/unknown program functions/instructions causing imprecision in symbolic execution by uninterpreted functions, and (2) record uninterpreted function samples capturing input-output pairs observed at execution time for those functions. We show that higher-order test generation generalizes and is more precise than simplifying complex symbolic expressions using their concrete runtime values. We present several program examples where our approach can exercise program paths and find bugs missed by previous techniques. We discuss the implementability and applications of this approach. We also explain in what sense dynamic test generation is more powerful than static test generation.

References

  1. S. Anand, P. Godefroid, and N. Tillmann. Demand-Driven Compositional Symbolic Execution. In Proceedings of TACAS'2008, volume 4963 of Lecture Notes in Computer Science, pages 367--381, Budapest, April 2008. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. M. Paradkar, and M. D. Ernst. Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking. IEEE Trans. Software Eng., 36(4):474--494, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. L. Bachmair and H. Ganzinger. Resolution Theorem Proving. In Handbook of Automated Reasoning, pages 19--99. 2001.Google ScholarGoogle ScholarCross RefCross Ref
  4. M. Barnett, B. E. Chang, R. DeLine, B. Jacobs, and K. R. M. Leino. Boogie: A modular reusable verifier for object-oriented programs. In Proceedings of FMCO'2005, volume 4111 of Lecture Notes in Computer Science, pages 364--387. Springer-Verlag, September 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. A. Blass, Y. Gurevich, L. Nachmanson, and M. Veanes. Play to Test. In Proceedings of FATES'2005, Edinburgh, July 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: Automatically Generating Inputs of Death. In ACM CCS, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. Cadar, P. Godefroid, S. Khurshid, C.S. Pasareanu, K. Sen, N. Tillmann, and W. Visser. Symbolic Execution for Software Testing in Practice -- Preliminary Assessment. In Proceedings of ICSE'2011, Honolulu, May 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. S. Chandra, S. J. Fink, and M. Sridharan. Snugglebug: A Powerful Approach to Weakest Preconditions. In Proceedings of PLDI'2009, Dublin, June 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. L. de Moura and N. Bjorner. Z3: An Efficient SMT Solver. In Proceedings of TACAS'2008, volume 4963 of Lecture Notes in Computer Science, pages 337--340, Budapest, April 2008. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. Emmi, R. Majumdar, and K. Sen. Dynamic Test Input Generation for Database Applications. In Proceedings of ISSTA'2007, pages 151--162, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Godefroid. Compositional Dynamic Test Generation. In Proceedings of POPL'2007, pages 47--54, Nice, January 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. P. Godefroid. Software Model Checking Improving Security of a Billion Computers. In Proceedings of SPIN'2009, volume 5578 of Lecture Notes in Computer Science, page 1, Grenoble, June 2009. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. P. Godefroid, M. Huth, and R. Jagadeesan. Abstraction-based Model Checking using Modal Transition Systems. In Proceedings of CONCUR'2001, volume 2154 of Lecture Notes in Computer Science, pages 426--440, Aalborg, August 2001. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. P. Godefroid, A. Kiezun, and M. Y. Levin. Grammar-based Whitebox Fuzzing. In Proceedings of PLDI'2008, pages 206--215, Tucson, June 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In Proceedings of PLDI'2005, pages 213--223, Chicago, June 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. P. Godefroid, M.Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. In Proceedings of NDSS'2008, pages 151--166, San Diego, February 2008.Google ScholarGoogle Scholar
  17. P. Godefroid, A.V. Nori, S.K. Rajamani, and S.D. Tetali. Compositional May-Must Program Analysis: Unleashing The Power of Alternation. In Proceedings of POPL'2010, pages 43--55, Madrid, January 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. J. Hoenicke, K. R. M. Leino, A. Podelski, M. Schaf, and Th. Wies. It's doomed; we can prove it. In Proceedings of 2009 World Congress on Formal Methods, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Sarfraz Khurshid, Corina S. Păsăreanu, and Willem Visser. Generalized Symbolic Execution for Model Checking and Testing. In Proceeding of TACAS'2003, April 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. C. King. Symbolic Execution and Program Testing. Journal of the ACM, 19(7):385--394, 1976. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. B. Korel. A Dynamic Approach of Test Data Generation. In IEEE Conference on Software Maintenance, pages 311--317, San Diego, November 1990.Google ScholarGoogle Scholar
  22. D. Molnar, X. C. Li, and D. Wagner. Dynamic test generation to find integer bugs in x86 binary linux programs. In Proc. of the 18th Usenix Security Symposium, Aug 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCamant, and D. Song. A Symbolic Execution Framework for JavaScript. In IEEE Symposium on Security and Privacy, pages 513--528, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. N. Tillmann and J. de Halleux. Pex - White Box Test Generation for .NET. In Proceedings of TAP'2008, volume 4966 of Lecture Notes in Computer Science, pages 134--153. Springer-Verlag, April 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. M. Yannakakis. Testing, Optimization, and Games. In Proceedings of LICS'2004, pages 78--88, Turku, July 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Higher-order test generation

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              • Published in

                cover image ACM SIGPLAN Notices
                ACM SIGPLAN Notices  Volume 46, Issue 6
                PLDI '11
                June 2011
                652 pages
                ISSN:0362-1340
                EISSN:1558-1160
                DOI:10.1145/1993316
                Issue’s Table of Contents
                • cover image ACM Conferences
                  PLDI '11: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation
                  June 2011
                  668 pages
                  ISBN:9781450306638
                  DOI:10.1145/1993498
                  • General Chair:
                  • Mary Hall,
                  • Program Chair:
                  • David Padua

                Copyright © 2011 ACM

                Publisher

                Association for Computing Machinery

                New York, NY, United States

                Publication History

                • Published: 4 June 2011

                Check for updates

                Qualifiers

                • research-article

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!