Abstract
Property-based random testing (PBRT) is widely used in the functional programming and verification communities. For testing simple properties, PBRT tools such as QuickCheck can automatically generate random inputs of a given type. But for more complex properties, effective testing often demands generators for random inputs that belong to a given type and satisfy some logical condition. QuickCheck provides a library of combinators for building such generators by hand, but this can be tedious for simple conditions and error prone for more complex ones. Fortunately, the process can often be automated. The most prominent method, narrowing, works by traversing the structure of the condition, lazily instantiating parts of the data structure as constraints involving them are met.
We show how to use ideas from narrowing to compile a large subclass of Coq's inductive relations into efficient generators, avoiding the interpretive overhead of previous implementations. More importantly, the same compilation technique allows us to produce proof terms certifying that each derived generator is good---i.e., sound and complete with respect to the inductive relation it was derived from. We implement our algorithm as an extension of QuickChick, an existing tool for property-based testing in Coq. We evaluate our method by automatically deriving good generators for the majority of the specifications in Software Foundations, a formalized textbook on programming language foundations.
Supplemental Material
- Sergio Antoy. 2000. A Needed Narrowing Strategy. In Journal of the ACM, Vol. 47. ACM Press, 776–822. https://www. informatik.uni- kiel.de/~mh/papers/JACM00.pdfGoogle Scholar
Digital Library
- Thomas Arts, Laura M. Castro, and John Hughes. 2008. Testing Erlang Data Types with QuviQ QuickCheck. In 7th ACM SIGPLAN Workshop on Erlang. ACM, 1–8. Google Scholar
Digital Library
- Lukas Bulwahn. 2012a. The New Quickcheck for Isabelle - Random, Exhaustive and Symbolic Testing under One Roof. In 2nd International Conference on Certified Programs and Proofs (CPP) (Lecture Notes in Computer Science), Vol. 7679. Springer, 92–108. https://www.irisa.fr/celtique/genet/ACF/BiblioIsabelle/quickcheckNew.pdfGoogle Scholar
- Lukas Bulwahn. 2012b. Smart Testing of Functional Programs in Isabelle. In 18th International Conference on Logic for Programming, Artificial Intelligence, and Reasoning (LPAR) (Lecture Notes in Computer Science), Vol. 7180. Springer, 153–167. Google Scholar
Digital Library
- Matthieu Carlier, Catherine Dubois, and Arnaud Gotlieb. 2013. FocalTest: A Constraint Programming Approach for Property-Based Testing. In Software and Data Technologies (Communications in Computer and Information Science), Vol. 170. Springer, 140–155. Google Scholar
Cross Ref
- Harsh Raju Chamarthi, Peter C. Dillinger, Matt Kaufmann, and Panagiotis Manolios. 2011. Integrating Testing and Interactive Theorem Proving. In 10th International Workshop on the ACL2 Theorem Prover and its Applications (EPTCS), Vol. 70. 4–19. http://arxiv.org/abs/1105.4394Google Scholar
- Jan Christiansen and Sebastian Fischer. 2008. EasyCheck – Test Data for Free. In 9th International Symposium on Functional and Logic Programming (FLOPS) (Lecture Notes in Computer Science), Vol. 4989. Springer, 322–336. http://www- ps. informatik.uni- kiel.de/~sebf/data/pub/flops08.pdfGoogle Scholar
Cross Ref
- Koen Claessen, Jonas Duregård, and Michal H. Palka. 2014. Generating Constrained Random Data with Uniform Distribution. In Functional and Logic Programming (Lecture Notes in Computer Science), Vol. 8475. Springer, 18–34. Google Scholar
Cross Ref
- Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In 5th ACM SIGPLAN International Conference on Functional Programming (ICFP). ACM, 268–279. http://www.eecs.northwestern.edu/ ~robby/courses/395- 495- 2009- fall/quick.pdfGoogle Scholar
Digital Library
- David Delahaye, Catherine Dubois, and Jean-Frédéric Étienne. 2007. Extracting Purely Functional Contents from Logical Inductive Types. In 20th International Conference on Theorem Proving in Higher Order Logics (TPHOLs) (Lecture Notes in Computer Science), Vol. 4732. Springer, 70–85. http://cedric.cnam.fr/~delahaye/papers/pred- exec%20(TPHOLs’ 07).pdfGoogle Scholar
Cross Ref
- Maxime Dénès, Cătălin Hriţcu, Leonidas Lampropoulos, Zoe Paraskevopoulou, and Benjamin C. Pierce. 2014. QuickChick: Property-based testing for Coq. The Coq Workshop. (July 2014). http://prosecco.gforge.inria.fr/personal/hritcu/talks/ coq6_submission_4.pdfGoogle Scholar
- Peter Dybjer, Qiao Haiyan, and Makoto Takeyama. 2003. Combining Testing and Proving in Dependent Type Theory. In 16th International Conference on Theorem Proving in Higher Order Logics (TPHOLs) (Lecture Notes in Computer Science), Vol. 2758. Springer, 188–203. http://www.cse.chalmers.se/~peterd/papers/Testing_Proving.pdf Google Scholar
Cross Ref
- Peter Dybjer, Qiao Haiyan, and Makoto Takeyama. 2004. Verifying Haskell programs by combining testing, model checking and interactive theorem proving. Information & Software Technology 46, 15 (2004), 1011–1025. http://www.cse.chalmers. se/~peterd/papers/TestingModelChecking.pdf Google Scholar
Cross Ref
- Burke Fetscher, Koen Claessen, Michal H. Palka, John Hughes, and Robert Bruce Findler. 2015. Making Random Judgments: Automatically Generating Well-Typed Terms from the Definition of a Type-System. In 24th European Symposium on Programming (Lecture Notes in Computer Science), Vol. 9032. Springer, 383–405. http://users.eecs.northwestern.edu/ ~baf111/random- judgments/Google Scholar
Cross Ref
- Sebastian Fischer and Herbert Kuchen. 2007. Systematic generation of glass-box test cases for functional logic programs. In 9th International ACM SIGPLAN Conference on Principles and Practice of Declarative Programming (PPDP). ACM, 63–74. http://www- ps.informatik.uni- kiel.de/~sebf/pub/ppdp07.htmlGoogle Scholar
Digital Library
- Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test generation through programming in UDITA. In 32nd ACM/IEEE International Conference on Software Engineering. ACM, 225–234. Google Scholar
Digital Library
- Arnaud Gotlieb. 2009. Euclide: A Constraint-Based Testing Framework for Critical C Programs. In ICST 2009, Second International Conference on Software Testing Verification and Validation, 1-4 April 2009, Denver, Colorado, USA. 151–160. Google Scholar
Digital Library
- M. Hanus, H. Kuchen, and J.J. Moreno-Navarro. 1995. Curry: A Truly Functional Logic Language. In Proc. ILPS’95 Workshop on Visions for the Future of Logic Programming. 95–107. http://www.math.rug.nl/~piter/KR/hanus95curry.pdfGoogle Scholar
- Cătălin Hriţcu, John Hughes, Benjamin C. Pierce, Antal Spector-Zabusky, Dimitrios Vytiniotis, Arthur Azevedo de Amorim, and Leonidas Lampropoulos. 2013. Testing Noninterference, Quickly. In 18th ACM SIGPLAN International Conference on Functional Programming (ICFP). ACM, 455–468. http://prosecco.gforge.inria.fr/personal/hritcu/publications/ testing- noninterference- icfp2013.pdfGoogle Scholar
Digital Library
- Cătălin Hriţcu, Leonidas Lampropoulos, Antal Spector-Zabusky, Arthur Azevedo de Amorim, Maxime Dénès, John Hughes, Benjamin C. Pierce, and Dimitrios Vytiniotis. 2016. Testing Noninterference, Quickly. Journal of Functional Programming (JFP); Special issue for ICFP 2013 26 (April 2016), e4 (62 pages). Google Scholar
Cross Ref
- John Hughes. 2007. QuickCheck Testing for Fun and Profit. In 9th International Symposium on Practical Aspects of Declarative Languages (PADL) (Lecture Notes in Computer Science), Vol. 4354. Springer, 1–32. http://people.inf.elte.hu/center/fulltext. pdfGoogle Scholar
- Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. http://crest.cs.ucl.ac.uk/fileadmin/crest/sebasepaper/JiaH10.pdf Google Scholar
Digital Library
- Ivan Kuraj and Viktor Kuncak. 2014. SciFe: Scala framework for efficient enumeration of data structures with invariants. In Proceedings of the Fifth Annual Scala Workshop. ACM, 45–49. Google Scholar
Digital Library
- Ivan Kuraj, Viktor Kuncak, and Daniel Jackson. 2015. Programming with Enumerable Sets of Structures. In OOPSLA. http://lara.epfl.ch/~kuncak/papers/KurajETAL15ProgrammingEnumerableSetsStructures.pdfGoogle Scholar
- Leonidas Lampropoulos, Diane Gallois-Wong, Catalin Hritcu, John Hughes, Benjamin C. Pierce, and Li-yao Xia. 2017. Beginner’s Luck: a language for property-based generators. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, Paris, France, January 18-20, 2017. 114–129. http://dl.acm.org/citation. cfm?id=3009868Google Scholar
Digital Library
- Fredrik Lindblad. 2007. Property Directed Generation of First-Order Test Data. In 8th Symposium on Trends in Functional Programming (Trends in Functional Programming), Vol. 8. Intellect, 105–123.Google Scholar
- Carlos Pacheco and Michael D. Ernst. 2007. Randoop: feedback-directed random testing for Java. In 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems And Applications (OOPSLA). ACM, 815–816. Google Scholar
Digital Library
- Michal H. Palka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an Optimising Compiler by Generating Random Lambda Terms. In Proceedings of the 6th International Workshop on Automation of Software Test (AST ’11). ACM, New York, NY, USA, 91–97. Google Scholar
Digital Library
- Manolis Papadakis and Konstantinos F. Sagonas. 2011. A PropEr integration of types and function specifications with property-based testing. In Proceedings of the 10th ACM SIGPLAN workshop on Erlang, Tokyo, Japan, September 23, 2011. 39–50. Google Scholar
Digital Library
- Zoe Paraskevopoulou, Cătălin Hriţcu, Maxime Dénès, Leonidas Lampropoulos, and Benjamin C. Pierce. 2015. Foundational Property-Based Testing. In 6th International Conference on Interactive Theorem Proving (ITP) (Lecture Notes in Computer Science), Christian Urban and Xingyuan Zhang (Eds.), Vol. 9236. Springer, 325–343. http://prosecco.gforge.inria.fr/ personal/hritcu/publications/foundational- pbt.pdfGoogle Scholar
- Benjamin C. Pierce, Arthur Azevedo de Amorim, Chris Casinghino, Marco Gaboardi, Michael Greenberg, Cătălin Hriţcu, Vilhelm Sjöberg, and Brent Yorgey. 2016. Software Foundations. Electronic textbook, Version 4.0 beta. https://www.cis. upenn.edu/~bcpierce/sf/sf- 4.0/index.htmlGoogle Scholar
- Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation Validation. In Tools and Algorithms for Construction and Analysis of Systems, 4th International Conference, TACAS ’98, Held as Part of the European Joint Conferences on the Theory and Practice of Software, ETAPS’98, Lisbon, Portugal, March 28 - April 4, 1998, Proceedings (Lecture Notes in Computer Science), Bernhard Steffen (Ed.), Vol. 1384. Springer, 151–166. Google Scholar
Cross Ref
- Colin Runciman, Matthew Naylor, and Fredrik Lindblad. 2008. SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. In 1st ACM SIGPLAN Symposium on Haskell. ACM, 37–48. http://www.cs.york.ac.uk/fp/smallcheck/ smallcheck.pdfGoogle Scholar
Digital Library
- Eric L. Seidel, Niki Vazou, and Ranjit Jhala. 2015. Type Targeted Testing. In Programming Languages and Systems - 24th European Symposium on Programming, ESOP 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015. Proceedings. 812–836. Google Scholar
Digital Library
- Matthieu Sozeau and Nicolas Oury. 2008. First-Class Type Classes. In Proceedings of the 21st International Conference on Theorem Proving in Higher Order Logics (TPHOLs ’08). Springer-Verlag, Berlin, Heidelberg, 278–293. Google Scholar
Digital Library
- Pierre-Nicolas Tollitte, David Delahaye, and Catherine Dubois. 2012. Producing Certified Functional Code from Inductive Specifications. In Second International Conference on Certified Programs and Proofs (CPP) (Lecture Notes in Computer Science), Vol. 7679. Springer. http://cedric.cnam.fr/~delahaye/papers/relext- coq%20%28CPP%2712%29.pdf Google Scholar
Digital Library
- P. Wadler and S. Blott. 1989. How to Make Ad-hoc Polymorphism Less Ad Hoc. In Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’89). ACM, New York, NY, USA, 60–76. Google Scholar
Digital Library
- Li-yao Xia. 2017. generic-random: Generic, Customizable Arbitrary Instances. http://hackage.haskell.org/package/ generic- random . (10 April 2017).Google Scholar
Index Terms
Generating good generators for inductive relations
Recommendations
Beginner's luck: a language for property-based generators
POPL '17: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming LanguagesProperty-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific ...
Coverage guided, property based testing
Property-based random testing, exemplified by frameworks such as Haskell's QuickCheck, works by testing an executable predicate (a property) on a stream of randomly generated inputs. Property testing works very well in many cases, but not always. Some ...
Beginner's luck: a language for property-based generators
POPL '17Property-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific ...






Comments