Abstract
Property-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight annotations to control both the distribution of generated values and the amount of constraint solving that happens before each variable is instantiated. This language, called Luck, makes generators easier to write, read, and maintain.
We give Luck a formal semantics and prove several fundamental properties, including the soundness and completeness of random generation with respect to a standard predicate semantics. We evaluate Luck on common examples from the property-based testing literature and on two significant case studies, showing that it can be used in complex domains with comparable bug-finding effectiveness and a significant reduction in testing code size compared to handwritten generators.
- S. Antoy. A needed narrowing strategy. JACM. 2000. Google Scholar
Digital Library
- T. Arts, L. M. Castro, and J. Hughes. Testing Erlang data types with QuviQ QuickCheck. In 7th ACM SIGPLAN Workshop on Erlang. 2008. Google Scholar
Digital Library
- T. Avgerinos, A. Rebert, S. K. Cha, and D. Brumley. Enhancing symbolic execution with Veritesting. ICSE, 2014. Google Scholar
Digital Library
- T. Ball, V. Levin, and S. K. Rajamani. A decade of software model checking with SLAM. Commun. ACM, 54(7):68–76, 2011. Google Scholar
Digital Library
- J. C. Blanchette and T. Nipkow. Nitpick: A counterexample generator for higher-order logic based on a relational model finder. ITP. 2010. Google Scholar
Digital Library
- L. Bulwahn. The new Quickcheck for Isabelle - random, exhaustive and symbolic testing under one roof. CPP. 2012. Google Scholar
Digital Library
- L. Bulwahn. Smart testing of functional programs in Isabelle. LPAR. 2012. Google Scholar
Digital Library
- J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. ASE, 2008. Google Scholar
Digital Library
- C. Cadar, D. Dunbar, and D. Engler. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. OSDI. 2008. Google Scholar
Digital Library
- C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: automatically generating inputs of death. CCS. 2006. Google Scholar
Digital Library
- C. Cadar, P. Godefroid, S. Khurshid, C. S. Păsăreanu, K. Sen, N. Tillmann, and W. Visser. Symbolic execution for software testing in practice: preliminary assessment. ICSE. 2011. Google Scholar
Digital Library
- M. Carlier, C. Dubois, and A. Gotlieb. Constraint reasoning in Focal-Test. ICSOFT. 2010.Google Scholar
- A. T. Chaganty, A. V. Nori, and S. K. Rajamani. Efficiently sampling probabilistic programs via program analysis. AISTATS, 2013.Google Scholar
- S. Chakraborty, K. S. Meel, and M. Y. Vardi. Balancing scalability and uniformity in SAT witness generator. DAC. 2014. Google Scholar
Digital Library
- H. R. Chamarthi, P. C. Dillinger, M. Kaufmann, and P. Manolios. Integrating testing and interactive theorem proving. ACL2, 2011.Google Scholar
- J. Christiansen and S. Fischer. EasyCheck – test data for free. FLOPS. 2008.Google Scholar
- K. Claessen, J. Dureg˚ard, and M. H. Pałka. Generating constrained random data with uniform distribution. FLOPS. 2014.Google Scholar
- K. Claessen, J. Dureg˚ard, and M. H. Palka. Generating constrained random data with uniform distribution. J. Funct. Program., 25, 2015.Google Scholar
- K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. ICFP. 2000. Google Scholar
Digital Library
- P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. Boltzmann samplers for the random generation of combinatorial structures. Combinatorics, Probability & Computing, 13(4-5):577–625, 2004. Google Scholar
Digital Library
- J. Dureg˚ard, P. Jansson, and M. Wang. Feat: Functional enumeration of algebraic types. Haskell Symposium. 2012. Google Scholar
Digital Library
- P. Dybjer, Q. Haiyan, and M. Takeyama. Combining testing and proving in dependent type theory. TPHOLs. 2003.Google Scholar
- B. Fetscher, K. Claessen, M. H. Palka, J. Hughes, and R. B. Findler. Making random judgments: Automatically generating well-typed terms from the definition of a type-system. ESOP. 2015.Google Scholar
- S. Fischer and H. Kuchen. Systematic generation of glass-box test cases for functional logic programs. PPDP. 2007. Google Scholar
Digital Library
- J. Fowler and G. Huttom. Towards a theory of reach. TFP. 2015. Google Scholar
Digital Library
- D. Gallois-Wong. Formalising Luck: Improved probabilistic semantics for property-based generators. Inria Internship Report, 2016.Google Scholar
- M. Gligoric, T. Gvero, V. Jagannath, S. Khurshid, V. Kuncak, and D. Marinov. Test generation through programming in UDITA. ICSE. 2010. Google Scholar
Digital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. PLDI. 2005. Google Scholar
Digital Library
- P. Godefroid, M. Y. Levin, and D. A. Molnar. SAGE: whitebox fuzzing for security testing. ACM Queue, 10(1):20, 2012. Google Scholar
Digital Library
- N. D. Goodman, V. K. Mansinghka, D. M. Roy, K. Bonawitz, and J. B. Tenenbaum. Church: a language for generative models. UAI, 2008. Google Scholar
Digital Library
- A. D. Gordon, T. A. Henzinger, A. V. Nori, and S. K. Rajamani. Probabilistic programming. FOSE. 2014. Google Scholar
Digital Library
- A. Gotlieb. Euclide: A constraint-based testing framework for critical C programs. ICST, 2009. Google Scholar
Digital Library
- A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr. Swarm testing. ISSTA. 2012. Google Scholar
Digital Library
- M. Hanus, H. Kuchen, and J. Moreno-Navarro. Curry: A truly functional logic language. ILPS, 1995.Google Scholar
- M. Hanus. A unified computation model for functional and logic programming. POPL. 1997. Google Scholar
Digital Library
- R. Herbrich, T. Minka, and T. Graepel. Trueskilltm: A bayesian skill rating system. NIPS, 2006. Google Scholar
Digital Library
- C. Hrit¸cu, J. Hughes, B. C. Pierce, A. Spector-Zabusky, D. Vytiniotis, A. Azevedo de Amorim, and L. Lampropoulos. Testing noninterference, quickly. ICFP. 2013.Google Scholar
- C. Hrit¸cu, L. Lampropoulos, A. Spector-Zabusky, A. Azevedo de Amorim, M. Dénès, J. Hughes, B. C. Pierce, and D. Vytiniotis. Testing noninterference, quickly. JFP, 26:e4 (62 pages), 2016.Google Scholar
- Technical Report available as arXiv:1409.0393.Google Scholar
- J. Hughes. QuickCheck testing for fun and profit. PADL. 2007. Google Scholar
Digital Library
- D. Jackson. Software Abstractions: Logic, Language, and Anlysis. The MIT Press, 2011. Google Scholar
Digital Library
- T. P. Jensen. Disjunctive program analysis for algebraic data types. ACM Trans. Program. Lang. Syst., 19(5):751–803, 1997. Google Scholar
Digital Library
- R. Jhala and R. Majumdar. Software model checking. ACM Comput. Surv., 41(4), 2009. Google Scholar
Digital Library
- G. Kaki and S. Jagannathan. A relational framework for higher-order shape analysis. ICFP. 2014. Google Scholar
Digital Library
- A. S. Köksal, V. Kuncak, and P. Suter. Scala to the power of Z3: integrating SMT and programming. CADE. 2011.Google Scholar
- K. Łatuszy´nski, G. O. Roberts, and J. S. Rosenthal. Adaptive gibbs samplers and related mcmc methods. The Annals of Applied Probability, 23(1):66–98, 2013.Google Scholar
Cross Ref
- F. Lindblad. Property directed generation of first-order test data. TFP. 2007.Google Scholar
- A. K. Mackworth. Consistency in networks of relations. Artif. Intell., 8(1):99–118, 1977. Google Scholar
Digital Library
- R. Majumdar and K. Sen. Hybrid concolic testing. ICSE. 2007. Google Scholar
Digital Library
- V. K. Mansinghka, D. M. Roy, E. Jonas, and J. B. Tenenbaum. Exact and approximate sampling by systematic stochastic search. AISTATS, 2009.Google Scholar
- B. Milch, B. Marthi, S. J. Russell, D. Sontag, D. L. Ong, and A. Kolobov. BLOG: probabilistic models with unknown objects. IJCAI, 2005. Google Scholar
Digital Library
- F. Nielson and H. R. Nielson. Tensor products generalize the relational data flow analysis method. In 4th Hungarian Computer Science Conference, 1985.Google Scholar
- T. Nipkow, M. Wenzel, and L. C. Paulson. Isabelle/HOL: A Proof Assistant for Higher-order Logic. Springer-Verlag, Berlin, Heidelberg, 2002. Google Scholar
Digital Library
- A. V. Nori, C.-K. Hur, S. K. Rajamani, and S. Samuel. R2: An efficient mcmc sampler for probabilistic programs. AAAI. 2014. Google Scholar
Digital Library
- C. Okasaki. Red-black trees in a functional setting. JFP, 9(4):471– 477, 1999. Google Scholar
Digital Library
- S. Owre. Random testing in PVS. In Workshop on Automated Formal Methods, 2006.Google Scholar
- C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for Java. OOPSLA. 2007. Google Scholar
Digital Library
- M. H. Pałka. Testing an optimising compiler by generating random lambda terms. http://www.cse.chalmers.se/~palka/ testingcompiler/.Google Scholar
- M. H. Pałka, K. Claessen, A. Russo, and J. Hughes. Testing an optimising compiler by generating random lambda terms. AST. 2011.Google Scholar
- Z. Paraskevopoulou, C. Hrit¸cu, M. Dénès, L. Lampropoulos, and B. C. Pierce. Foundational property-based testing. ITP. 2015.Google Scholar
- J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Testcase reduction for C compiler bugs. PLDI, 2012. Google Scholar
Digital Library
- J. S. Reich, M. Naylor, and C. Runciman. Lazy generation of canonical test programs. IFL. 2011. Google Scholar
Digital Library
- A. Rodriguez Yakushev and J. Jeuring. Enumerating well-typed terms generically. In U. Schmid, E. Kitzelmann, and R. Plasmeijer, editors, Approaches and Applications of Inductive Programming, volume 5812 of Lecture Notes in Computer Science, pages 93–116. Springer Berlin Heidelberg, 2010.Google Scholar
- C. Runciman, M. Naylor, and F. Lindblad. SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. Haskell Symposium. 2008. Google Scholar
Digital Library
- E. L. Seidel, N. Vazou, and R. Jhala. Type targeted testing. ESOP, 2015. Google Scholar
Digital Library
- K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. ESEC/FSE. 2005. Google Scholar
Digital Library
- P. Tarau. On type-directed generation of lambda terms. ICLP, 2015.Google Scholar
- A. P. Tolmach and S. Antoy. A monadic semantics for core Curry. Electr. Notes Theor. Comput. Sci., 86(3):16–34, 2003.Google Scholar
Cross Ref
- E. Torlak and R. Bod´ık. A lightweight symbolic virtual machine for solver-aided host languages. PLDI. 2014. Google Scholar
Digital Library
Index Terms
Beginner's luck: a language for property-based generators
Recommendations
Beginner's luck: a language for property-based generators
POPL '17: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming LanguagesProperty-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific ...
Coverage guided, property based testing
Property-based random testing, exemplified by frameworks such as Haskell's QuickCheck, works by testing an executable predicate (a property) on a stream of randomly generated inputs. Property testing works very well in many cases, but not always. Some ...
Generating good generators for inductive relations
Property-based random testing (PBRT) is widely used in the functional programming and verification communities. For testing simple properties, PBRT tools such as QuickCheck can automatically generate random inputs of a given type. But for more complex ...







Comments