skip to main content
research-article

SmartCheck: automatic and efficient counterexample reduction and generalization

Published:03 September 2014Publication History
Skip Abstract Section

Abstract

QuickCheck is a powerful library for automatic test-case generation. Because QuickCheck performs random testing, some of the counterexamples discovered are very large. QuickCheck provides an interface for the user to write shrink functions to attempt to reduce the size of counter examples. Hand-written implementations of shrink can be complex, inefficient, and consist of significant boilerplate code. Furthermore, shrinking is only one aspect in debugging: counterexample generalization is the process of extrapolating from individual counterexamples to a class of counterexamples, often requiring a flash of insight from the programmer. To improve counterexample reduction and generalization, we introduce SmartCheck. SmartCheck is a debugging tool that reduces algebraic data using generic search heuristics to efficiently find smaller counterexamples. In addition to shrinking, SmartCheck also automatically generalizes counterexamples to formulas representing classes of counterexamples. SmartCheck has been implemented for Haskell and is freely available.

References

  1. T. Arts, J. Hughes, J. Johansson, and U. T. Wiger. Testing telecoms software with Quviq QuickCheck. In ACM SIGPLAN Workshop on Erlang Erlang Workshop, pages 2--10. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. K. Claessen. Shrinking and showing functions: (functional pearl). In Proceedings of the Haskell symposium, pages 73--80. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. In ACM SIGPLAN International Conference on Functional Programming (ICFP), pages 268--279. ACM, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. K. Claessen and J. Hughes. Testing monadic code with QuickCheck. In ACM SIGPLAN workshop on Haskell, pages 65--77, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. Claessen, N. Smallbone, and J. Hughes. QuickSpec: Guessing formal specifications using testing. In Tests and Proofs Intl. Conference (TAP), LNCS, pages 6--21, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. J. Duregård, P. Jansson, and M. Wang. Feat: functional enumeration of algebraic types. In Proceedings of the 5th ACM SIGPLAN Symposium on Haskell, pages 61--72. ACM, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programing, 69 (1--3): 35--45, Dec. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. G. P. Huet. The zipper. Journal of Functional Programming, 7 (5): 549--554, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. J. Hughes. Software testing with QuickCheck. In Central European Functional Programming School (CEFP), volume 6299 of LNCS, pages 183--223. Springer, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. D. Jackson. Software abstractions: logic, language and analysis. MIT Press, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Johann and N. Ghani. Foundations for structured programming with GADTs. In Symposium on Principles of programming Languages (POPL), pages 297--308. ACM, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. E. Kow. GenI: natural language generation in Haskell. In Proceedings of the 2006 ACM SIGPLAN workshop on Haskell, pages 110--119. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. R. Lmmel and S. L. Peyton-Jones. Scrap your boilerplate with class: extensible generic functions. In ACM SIGPLAN International Conference on Functional Programming (ICFP), pages 204--215. ACM, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. J. P. Magalhães, A. Dijkstra, J. Jeuring, and A. Löh. A generic deriving mechanism for Haskell. In Proceedings of the 3rd ACM Haskell Symposium on Haskell, pages 37--48. ACM, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Marlow (editor). Haskell 2010 language report. Technical report, July 2010.Google ScholarGoogle Scholar
  16. G. Misherghi and Z. Su. HDD: hierarchical delta debugging. In Proceedings of the 28th international conference on Software engineering, pages 142--151. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-case reduction for C compiler bugs. SIGPLAN Notices, 47 (6), June 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. C. Runciman, M. Naylor, and F. Lindblad. SmallCheck and lazy smallcheck: automatic exhaustive testing for small values. In Proceedings of the ACM Haskell Symposium, pages 37--48. ACM, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. D. Stewart and S. Sjanssen. XMonad. In ACM SIGPLAN Workshop on Haskell, page 119. ACM, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering, 28 (2): 183--200, Feb. 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. SmartCheck: automatic and efficient counterexample reduction and generalization

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 49, Issue 12
      Haskell '14
      December 2014
      141 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/2775050
      • Editor:
      • Andy Gill
      Issue’s Table of Contents
      • cover image ACM Conferences
        Haskell '14: Proceedings of the 2014 ACM SIGPLAN symposium on Haskell
        September 2014
        154 pages
        ISBN:9781450330411
        DOI:10.1145/2633357

      Copyright © 2014 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 3 September 2014

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!