Abstract
QuickCheck is a powerful library for automatic test-case generation. Because QuickCheck performs random testing, some of the counterexamples discovered are very large. QuickCheck provides an interface for the user to write shrink functions to attempt to reduce the size of counter examples. Hand-written implementations of shrink can be complex, inefficient, and consist of significant boilerplate code. Furthermore, shrinking is only one aspect in debugging: counterexample generalization is the process of extrapolating from individual counterexamples to a class of counterexamples, often requiring a flash of insight from the programmer. To improve counterexample reduction and generalization, we introduce SmartCheck. SmartCheck is a debugging tool that reduces algebraic data using generic search heuristics to efficiently find smaller counterexamples. In addition to shrinking, SmartCheck also automatically generalizes counterexamples to formulas representing classes of counterexamples. SmartCheck has been implemented for Haskell and is freely available.
- T. Arts, J. Hughes, J. Johansson, and U. T. Wiger. Testing telecoms software with Quviq QuickCheck. In ACM SIGPLAN Workshop on Erlang Erlang Workshop, pages 2--10. ACM, 2006. Google Scholar
Digital Library
- K. Claessen. Shrinking and showing functions: (functional pearl). In Proceedings of the Haskell symposium, pages 73--80. ACM, 2012. Google Scholar
Digital Library
- K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. In ACM SIGPLAN International Conference on Functional Programming (ICFP), pages 268--279. ACM, 2000. Google Scholar
Digital Library
- K. Claessen and J. Hughes. Testing monadic code with QuickCheck. In ACM SIGPLAN workshop on Haskell, pages 65--77, 2002. Google Scholar
Digital Library
- K. Claessen, N. Smallbone, and J. Hughes. QuickSpec: Guessing formal specifications using testing. In Tests and Proofs Intl. Conference (TAP), LNCS, pages 6--21, 2010. Google Scholar
Digital Library
- J. Duregård, P. Jansson, and M. Wang. Feat: functional enumeration of algebraic types. In Proceedings of the 5th ACM SIGPLAN Symposium on Haskell, pages 61--72. ACM, 2012. Google Scholar
Digital Library
- M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programing, 69 (1--3): 35--45, Dec. 2007. Google Scholar
Digital Library
- G. P. Huet. The zipper. Journal of Functional Programming, 7 (5): 549--554, 1997. Google Scholar
Digital Library
- J. Hughes. Software testing with QuickCheck. In Central European Functional Programming School (CEFP), volume 6299 of LNCS, pages 183--223. Springer, 2010. Google Scholar
Digital Library
- D. Jackson. Software abstractions: logic, language and analysis. MIT Press, 2006. Google Scholar
Digital Library
- P. Johann and N. Ghani. Foundations for structured programming with GADTs. In Symposium on Principles of programming Languages (POPL), pages 297--308. ACM, 2008. Google Scholar
Digital Library
- E. Kow. GenI: natural language generation in Haskell. In Proceedings of the 2006 ACM SIGPLAN workshop on Haskell, pages 110--119. ACM, 2006. Google Scholar
Digital Library
- R. Lmmel and S. L. Peyton-Jones. Scrap your boilerplate with class: extensible generic functions. In ACM SIGPLAN International Conference on Functional Programming (ICFP), pages 204--215. ACM, 2005. Google Scholar
Digital Library
- J. P. Magalhães, A. Dijkstra, J. Jeuring, and A. Löh. A generic deriving mechanism for Haskell. In Proceedings of the 3rd ACM Haskell Symposium on Haskell, pages 37--48. ACM, 2010. Google Scholar
Digital Library
- S. Marlow (editor). Haskell 2010 language report. Technical report, July 2010.Google Scholar
- G. Misherghi and Z. Su. HDD: hierarchical delta debugging. In Proceedings of the 28th international conference on Software engineering, pages 142--151. ACM, 2006. Google Scholar
Digital Library
- J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-case reduction for C compiler bugs. SIGPLAN Notices, 47 (6), June 2012. Google Scholar
Digital Library
- C. Runciman, M. Naylor, and F. Lindblad. SmallCheck and lazy smallcheck: automatic exhaustive testing for small values. In Proceedings of the ACM Haskell Symposium, pages 37--48. ACM, 2008. Google Scholar
Digital Library
- D. Stewart and S. Sjanssen. XMonad. In ACM SIGPLAN Workshop on Haskell, page 119. ACM, 2007. Google Scholar
Digital Library
- A. Zeller and R. Hildebrandt. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering, 28 (2): 183--200, Feb. 2002. Google Scholar
Digital Library
Index Terms
SmartCheck: automatic and efficient counterexample reduction and generalization
Recommendations
SmartCheck: automatic and efficient counterexample reduction and generalization
Haskell '14: Proceedings of the 2014 ACM SIGPLAN symposium on HaskellQuickCheck is a powerful library for automatic test-case generation. Because QuickCheck performs random testing, some of the counterexamples discovered are very large. QuickCheck provides an interface for the user to write shrink functions to attempt to ...
A framework for testing first-order logic axioms in program verification
Program verification systems based on automated theorem provers rely on user-provided axioms in order to verify domain-specific properties of code. However, formulating axioms correctly (that is, formalizing properties of an intended mathematical ...
Towards random and enumerative testing for OCaml and WhyML properties
AbstractDeductive program verification greatly improves software quality, but proving formal specifications is difficult, and this activity can only be partially automated. It is therefore relevant to supplement deductive verification tools, such as Why3, ...







Comments