skip to main content
research-article

Beginner's luck: a language for property-based generators

Published:01 January 2017Publication History
Skip Abstract Section

Abstract

Property-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific language in which generators are conveniently expressed by decorating predicates with lightweight annotations to control both the distribution of generated values and the amount of constraint solving that happens before each variable is instantiated. This language, called Luck, makes generators easier to write, read, and maintain.

We give Luck a formal semantics and prove several fundamental properties, including the soundness and completeness of random generation with respect to a standard predicate semantics. We evaluate Luck on common examples from the property-based testing literature and on two significant case studies, showing that it can be used in complex domains with comparable bug-finding effectiveness and a significant reduction in testing code size compared to handwritten generators.

References

  1. S. Antoy. A needed narrowing strategy. JACM. 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. T. Arts, L. M. Castro, and J. Hughes. Testing Erlang data types with QuviQ QuickCheck. In 7th ACM SIGPLAN Workshop on Erlang. 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. T. Avgerinos, A. Rebert, S. K. Cha, and D. Brumley. Enhancing symbolic execution with Veritesting. ICSE, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. T. Ball, V. Levin, and S. K. Rajamani. A decade of software model checking with SLAM. Commun. ACM, 54(7):68–76, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. J. C. Blanchette and T. Nipkow. Nitpick: A counterexample generator for higher-order logic based on a relational model finder. ITP. 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. L. Bulwahn. The new Quickcheck for Isabelle - random, exhaustive and symbolic testing under one roof. CPP. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. L. Bulwahn. Smart testing of functional programs in Isabelle. LPAR. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. ASE, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. C. Cadar, D. Dunbar, and D. Engler. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. OSDI. 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE: automatically generating inputs of death. CCS. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. C. Cadar, P. Godefroid, S. Khurshid, C. S. Păsăreanu, K. Sen, N. Tillmann, and W. Visser. Symbolic execution for software testing in practice: preliminary assessment. ICSE. 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. M. Carlier, C. Dubois, and A. Gotlieb. Constraint reasoning in Focal-Test. ICSOFT. 2010.Google ScholarGoogle Scholar
  13. A. T. Chaganty, A. V. Nori, and S. K. Rajamani. Efficiently sampling probabilistic programs via program analysis. AISTATS, 2013.Google ScholarGoogle Scholar
  14. S. Chakraborty, K. S. Meel, and M. Y. Vardi. Balancing scalability and uniformity in SAT witness generator. DAC. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. H. R. Chamarthi, P. C. Dillinger, M. Kaufmann, and P. Manolios. Integrating testing and interactive theorem proving. ACL2, 2011.Google ScholarGoogle Scholar
  16. J. Christiansen and S. Fischer. EasyCheck – test data for free. FLOPS. 2008.Google ScholarGoogle Scholar
  17. K. Claessen, J. Dureg˚ard, and M. H. Pałka. Generating constrained random data with uniform distribution. FLOPS. 2014.Google ScholarGoogle Scholar
  18. K. Claessen, J. Dureg˚ard, and M. H. Palka. Generating constrained random data with uniform distribution. J. Funct. Program., 25, 2015.Google ScholarGoogle Scholar
  19. K. Claessen and J. Hughes. QuickCheck: a lightweight tool for random testing of Haskell programs. ICFP. 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. Boltzmann samplers for the random generation of combinatorial structures. Combinatorics, Probability & Computing, 13(4-5):577–625, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. J. Dureg˚ard, P. Jansson, and M. Wang. Feat: Functional enumeration of algebraic types. Haskell Symposium. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. P. Dybjer, Q. Haiyan, and M. Takeyama. Combining testing and proving in dependent type theory. TPHOLs. 2003.Google ScholarGoogle Scholar
  23. B. Fetscher, K. Claessen, M. H. Palka, J. Hughes, and R. B. Findler. Making random judgments: Automatically generating well-typed terms from the definition of a type-system. ESOP. 2015.Google ScholarGoogle Scholar
  24. S. Fischer and H. Kuchen. Systematic generation of glass-box test cases for functional logic programs. PPDP. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. J. Fowler and G. Huttom. Towards a theory of reach. TFP. 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. D. Gallois-Wong. Formalising Luck: Improved probabilistic semantics for property-based generators. Inria Internship Report, 2016.Google ScholarGoogle Scholar
  27. M. Gligoric, T. Gvero, V. Jagannath, S. Khurshid, V. Kuncak, and D. Marinov. Test generation through programming in UDITA. ICSE. 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. PLDI. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. P. Godefroid, M. Y. Levin, and D. A. Molnar. SAGE: whitebox fuzzing for security testing. ACM Queue, 10(1):20, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. N. D. Goodman, V. K. Mansinghka, D. M. Roy, K. Bonawitz, and J. B. Tenenbaum. Church: a language for generative models. UAI, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. A. D. Gordon, T. A. Henzinger, A. V. Nori, and S. K. Rajamani. Probabilistic programming. FOSE. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. A. Gotlieb. Euclide: A constraint-based testing framework for critical C programs. ICST, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr. Swarm testing. ISSTA. 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. M. Hanus, H. Kuchen, and J. Moreno-Navarro. Curry: A truly functional logic language. ILPS, 1995.Google ScholarGoogle Scholar
  35. M. Hanus. A unified computation model for functional and logic programming. POPL. 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. R. Herbrich, T. Minka, and T. Graepel. Trueskilltm: A bayesian skill rating system. NIPS, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. C. Hrit¸cu, J. Hughes, B. C. Pierce, A. Spector-Zabusky, D. Vytiniotis, A. Azevedo de Amorim, and L. Lampropoulos. Testing noninterference, quickly. ICFP. 2013.Google ScholarGoogle Scholar
  38. C. Hrit¸cu, L. Lampropoulos, A. Spector-Zabusky, A. Azevedo de Amorim, M. Dénès, J. Hughes, B. C. Pierce, and D. Vytiniotis. Testing noninterference, quickly. JFP, 26:e4 (62 pages), 2016.Google ScholarGoogle Scholar
  39. Technical Report available as arXiv:1409.0393.Google ScholarGoogle Scholar
  40. J. Hughes. QuickCheck testing for fun and profit. PADL. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. D. Jackson. Software Abstractions: Logic, Language, and Anlysis. The MIT Press, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. T. P. Jensen. Disjunctive program analysis for algebraic data types. ACM Trans. Program. Lang. Syst., 19(5):751–803, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. R. Jhala and R. Majumdar. Software model checking. ACM Comput. Surv., 41(4), 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. G. Kaki and S. Jagannathan. A relational framework for higher-order shape analysis. ICFP. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. A. S. Köksal, V. Kuncak, and P. Suter. Scala to the power of Z3: integrating SMT and programming. CADE. 2011.Google ScholarGoogle Scholar
  46. K. Łatuszy´nski, G. O. Roberts, and J. S. Rosenthal. Adaptive gibbs samplers and related mcmc methods. The Annals of Applied Probability, 23(1):66–98, 2013.Google ScholarGoogle ScholarCross RefCross Ref
  47. F. Lindblad. Property directed generation of first-order test data. TFP. 2007.Google ScholarGoogle Scholar
  48. A. K. Mackworth. Consistency in networks of relations. Artif. Intell., 8(1):99–118, 1977. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. R. Majumdar and K. Sen. Hybrid concolic testing. ICSE. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. V. K. Mansinghka, D. M. Roy, E. Jonas, and J. B. Tenenbaum. Exact and approximate sampling by systematic stochastic search. AISTATS, 2009.Google ScholarGoogle Scholar
  51. B. Milch, B. Marthi, S. J. Russell, D. Sontag, D. L. Ong, and A. Kolobov. BLOG: probabilistic models with unknown objects. IJCAI, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. F. Nielson and H. R. Nielson. Tensor products generalize the relational data flow analysis method. In 4th Hungarian Computer Science Conference, 1985.Google ScholarGoogle Scholar
  53. T. Nipkow, M. Wenzel, and L. C. Paulson. Isabelle/HOL: A Proof Assistant for Higher-order Logic. Springer-Verlag, Berlin, Heidelberg, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. A. V. Nori, C.-K. Hur, S. K. Rajamani, and S. Samuel. R2: An efficient mcmc sampler for probabilistic programs. AAAI. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. C. Okasaki. Red-black trees in a functional setting. JFP, 9(4):471– 477, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. S. Owre. Random testing in PVS. In Workshop on Automated Formal Methods, 2006.Google ScholarGoogle Scholar
  57. C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for Java. OOPSLA. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. M. H. Pałka. Testing an optimising compiler by generating random lambda terms. http://www.cse.chalmers.se/~palka/ testingcompiler/.Google ScholarGoogle Scholar
  59. M. H. Pałka, K. Claessen, A. Russo, and J. Hughes. Testing an optimising compiler by generating random lambda terms. AST. 2011.Google ScholarGoogle Scholar
  60. Z. Paraskevopoulou, C. Hrit¸cu, M. Dénès, L. Lampropoulos, and B. C. Pierce. Foundational property-based testing. ITP. 2015.Google ScholarGoogle Scholar
  61. J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Testcase reduction for C compiler bugs. PLDI, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. J. S. Reich, M. Naylor, and C. Runciman. Lazy generation of canonical test programs. IFL. 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. A. Rodriguez Yakushev and J. Jeuring. Enumerating well-typed terms generically. In U. Schmid, E. Kitzelmann, and R. Plasmeijer, editors, Approaches and Applications of Inductive Programming, volume 5812 of Lecture Notes in Computer Science, pages 93–116. Springer Berlin Heidelberg, 2010.Google ScholarGoogle Scholar
  64. C. Runciman, M. Naylor, and F. Lindblad. SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. Haskell Symposium. 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. E. L. Seidel, N. Vazou, and R. Jhala. Type targeted testing. ESOP, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. ESEC/FSE. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. P. Tarau. On type-directed generation of lambda terms. ICLP, 2015.Google ScholarGoogle Scholar
  68. A. P. Tolmach and S. Antoy. A monadic semantics for core Curry. Electr. Notes Theor. Comput. Sci., 86(3):16–34, 2003.Google ScholarGoogle ScholarCross RefCross Ref
  69. E. Torlak and R. Bod´ık. A lightweight symbolic virtual machine for solver-aided host languages. PLDI. 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Beginner's luck: a language for property-based generators

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!