skip to main content
article

Branching processes for QuickCheck generators

Published:17 September 2018Publication History
Skip Abstract Section

Abstract

In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the expected number of generated constructors, even in the presence of mutually recursive or composite ADTs. Using our probabilistic formulas, we design heuristics capable of automatically adjusting probabilities in order to synthesize generators which distributions are aligned with users' demands. We provide a Haskell implementation of our mechanism in a tool called DRaGeN and perform case studies with real-world applications. When generating random values, our synthesized QuickCheck generators show improvements in code coverage when compared with those automatically derived by state-of-the-art tools.

References

  1. A. Arcuri and L. Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proc. of the International Symposium on Software Testing and Analysis (ISSTA ’11). ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. B. Arkin, S. Stender, and G. McGraw. 2005. Software penetration testing. IEEE Security Privacy (2005). Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. T. Arts, J. Hughes, U. Norell, and H. Svensson. 2015. Testing AUTOSAR software with QuickCheck. In In Proc. of IEEE International Conference on Software Testing, Verification and Validation, ICST Workshops.Google ScholarGoogle Scholar
  4. M. Bendkowski, K. Grygiel, and P. Tarau. 2017. Boltzmann Samplers for Closed Simply-Typed Lambda Terms. In In Proc. of International Symposium on Practical Aspects of Declarative Languages. ACM.Google ScholarGoogle Scholar
  5. F.T. Chan, T.Y. Chen, I.K. Mak, and Y.T. Yu. 1996. Proportional sampling strategy: guidelines for software testing practitioners. Information and Software Technology 38, 12 (1996), 775 – 782.Google ScholarGoogle ScholarCross RefCross Ref
  6. T. Y. Chen, H. Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, Michael J. Maher (Ed.). Springer Berlin Heidelberg. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In Proc. of International Conference on Software Engineering. ACM/IEEE. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. K. Claessen, J. Duregård, and M. H. Palka. 2014. Generating Constrained Random Data with Uniform Distribution. In Proc. of the Functional and Logic Programming FLOPS.Google ScholarGoogle Scholar
  9. K. Claessen and J. Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proc. of the ACM SIGPLAN International Conference on Functional Programming (ICFP). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. 2004. Boltzmann Samplers for the Random Generation of Combinatorial Structures. Combinatorics, Probability and Computing. 13 (2004). Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J. Duregård, P. Jansson, and M. Wang. 2012. Feat: Functional enumeration of algebraic types. In Proc. of the ACM SIGPLAN Symposium on Haskell. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. R. Feldt and S. Poulding. 2013. Finding test data with specific properties via metaheuristic search. In Proc. of International Symp. on Software Reliability Engineering (ISSRE). IEEE.Google ScholarGoogle Scholar
  13. G. Grieco, M. Ceresa, and P. Buiras. 2016. QuickFuzz: An automatic random fuzzer for common file formats. In Proc. of the International Symposium on Haskell. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. G. Grieco, M. Ceresa, A. Mista, and P. Buiras. 2017. QuickFuzz testing for fun and profit. Journal of Systems and Software 134, Supp. C (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. P. Haccou, P. Jagers, and V. Vatutin. 2005. Branching processes. Variation, growth, and extinction of populations. Cambridge University Press.Google ScholarGoogle Scholar
  16. J. Hughes, C. Pierce B, T. Arts, and U. Norell. 2016. Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service. In Proc. of the Int. Conference on Software Testing, Verification and Validation, ICST.Google ScholarGoogle Scholar
  17. J. Hughes, U. Norell, N. Smallbone, and T. Arts. 2016. Find more bugs with QuickCheck!. In Proc. of the International Workshop on Automation of Software Test, [email protected]. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. L. Lampropoulos, D. Gallois-Wong, C. Hritcu, J. Hughes, B. C. Pierce, and L. Xia. 2017. Beginner’s luck: a language for property-based generators. In Proc. of the ACM SIGPLAN Symposium on Principles of Programming Languages, POPL. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. L. Lampropoulos, Z. Paraskevopoulou, and B. C. Pierce. 2017. Generating Good Generators for Inductive Relations. In Proc. ACM on Programming Languages 2, POPL, Article 45 (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. M. Zalewski. 2010. American Fuzzy Lop: a security-oriented fuzzer. http://lcamtuf.coredump.cx/afl/ . (2010).Google ScholarGoogle Scholar
  21. C. McBride and R. Paterson. 2008. Applicative Programming with Effects. Journal of Functional Programming 18, 1 (Jan. 2008). Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. Midtgaard, M. N. Justesen, P. Kasting, F. Nielson, and H. R. Nielson. 2017. Effect-driven QuickChecking of compilers. In Proceedings of the ACM on Programming Languages, Volume 1 ICFP (2017). Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. A. Mista, A. Russo, and J. Hughes. 2018. Branching Processes for QuickCheck Generators (extended version). https://bitbucket.org/ agustinmista/dragen/downloads/full-paper.pdf . (2018).Google ScholarGoogle Scholar
  24. N. Mitchell. 2007. Deriving Generic Functions by Example. In Proc. of the 1st York Doctoral Syposium. Tech. Report YCS-2007-421, Department of Computer Science, University of York, UK, 55–62.Google ScholarGoogle Scholar
  25. M. Pałka, K. Claessen, A. Russo, and J. Hughes. 2011. Testing and Optimising Compiler by Generating Random Lambda Terms. In The IEEE/ACM International Workshop on Automation of Software Test (AST 2011). Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. S. M. Poulding and R. Feldt. 2017. Automated Random Testing in Multiple Dispatch Languages. IEEE International Conference on Software Testing, Verification and Validation (ICST) (2017).Google ScholarGoogle Scholar
  27. C. Runciman, M. Naylor, and F. Lindblad. 2008. Smallcheck and Lazy Smallcheck: automatic exhaustive testing for small values. In Proc. of the ACM SIGPLAN Symposium on Haskell. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. Sheard and Simon L. Peyton Jones. 2002. Template meta-programming for Haskell. SIGPLAN Notices 37, 12 (2002), 60–75. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. H. W. Watson and F. Galton. 1875. On the probability of the extinction of families. The Journal of the Anthropological Institute of Great Britain and Ireland (1875).Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Branching processes for QuickCheck generators

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 53, Issue 7
      Haskell '18
      July 2018
      185 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/3299711
      Issue’s Table of Contents
      • cover image ACM Conferences
        Haskell 2018: Proceedings of the 11th ACM SIGPLAN International Symposium on Haskell
        September 2018
        185 pages
        ISBN:9781450358354
        DOI:10.1145/3242744

      Copyright © 2018 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 17 September 2018

      Check for updates

      Qualifiers

      • article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!