Abstract
In QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics known as branching processes, and show how they help to analytically predict (at compile-time) the expected number of generated constructors, even in the presence of mutually recursive or composite ADTs. Using our probabilistic formulas, we design heuristics capable of automatically adjusting probabilities in order to synthesize generators which distributions are aligned with users' demands. We provide a Haskell implementation of our mechanism in a tool called DRaGeN and perform case studies with real-world applications. When generating random values, our synthesized QuickCheck generators show improvements in code coverage when compared with those automatically derived by state-of-the-art tools.
- A. Arcuri and L. Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proc. of the International Symposium on Software Testing and Analysis (ISSTA ’11). ACM. Google Scholar
Digital Library
- B. Arkin, S. Stender, and G. McGraw. 2005. Software penetration testing. IEEE Security Privacy (2005). Google Scholar
Digital Library
- T. Arts, J. Hughes, U. Norell, and H. Svensson. 2015. Testing AUTOSAR software with QuickCheck. In In Proc. of IEEE International Conference on Software Testing, Verification and Validation, ICST Workshops.Google Scholar
- M. Bendkowski, K. Grygiel, and P. Tarau. 2017. Boltzmann Samplers for Closed Simply-Typed Lambda Terms. In In Proc. of International Symposium on Practical Aspects of Declarative Languages. ACM.Google Scholar
- F.T. Chan, T.Y. Chen, I.K. Mak, and Y.T. Yu. 1996. Proportional sampling strategy: guidelines for software testing practitioners. Information and Software Technology 38, 12 (1996), 775 – 782.Google Scholar
Cross Ref
- T. Y. Chen, H. Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, Michael J. Maher (Ed.). Springer Berlin Heidelberg. Google Scholar
Digital Library
- I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. 2008. ARTOO: adaptive random testing for object-oriented software. In Proc. of International Conference on Software Engineering. ACM/IEEE. Google Scholar
Digital Library
- K. Claessen, J. Duregård, and M. H. Palka. 2014. Generating Constrained Random Data with Uniform Distribution. In Proc. of the Functional and Logic Programming FLOPS.Google Scholar
- K. Claessen and J. Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In Proc. of the ACM SIGPLAN International Conference on Functional Programming (ICFP). Google Scholar
Digital Library
- P. Duchon, P. Flajolet, G. Louchard, and G. Schaeffer. 2004. Boltzmann Samplers for the Random Generation of Combinatorial Structures. Combinatorics, Probability and Computing. 13 (2004). Google Scholar
Digital Library
- J. Duregård, P. Jansson, and M. Wang. 2012. Feat: Functional enumeration of algebraic types. In Proc. of the ACM SIGPLAN Symposium on Haskell. Google Scholar
Digital Library
- R. Feldt and S. Poulding. 2013. Finding test data with specific properties via metaheuristic search. In Proc. of International Symp. on Software Reliability Engineering (ISSRE). IEEE.Google Scholar
- G. Grieco, M. Ceresa, and P. Buiras. 2016. QuickFuzz: An automatic random fuzzer for common file formats. In Proc. of the International Symposium on Haskell. ACM. Google Scholar
Digital Library
- G. Grieco, M. Ceresa, A. Mista, and P. Buiras. 2017. QuickFuzz testing for fun and profit. Journal of Systems and Software 134, Supp. C (2017). Google Scholar
Digital Library
- P. Haccou, P. Jagers, and V. Vatutin. 2005. Branching processes. Variation, growth, and extinction of populations. Cambridge University Press.Google Scholar
- J. Hughes, C. Pierce B, T. Arts, and U. Norell. 2016. Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service. In Proc. of the Int. Conference on Software Testing, Verification and Validation, ICST.Google Scholar
- J. Hughes, U. Norell, N. Smallbone, and T. Arts. 2016. Find more bugs with QuickCheck!. In Proc. of the International Workshop on Automation of Software Test, [email protected]. Google Scholar
Digital Library
- L. Lampropoulos, D. Gallois-Wong, C. Hritcu, J. Hughes, B. C. Pierce, and L. Xia. 2017. Beginner’s luck: a language for property-based generators. In Proc. of the ACM SIGPLAN Symposium on Principles of Programming Languages, POPL. Google Scholar
Digital Library
- L. Lampropoulos, Z. Paraskevopoulou, and B. C. Pierce. 2017. Generating Good Generators for Inductive Relations. In Proc. ACM on Programming Languages 2, POPL, Article 45 (2017). Google Scholar
Digital Library
- M. Zalewski. 2010. American Fuzzy Lop: a security-oriented fuzzer. http://lcamtuf.coredump.cx/afl/ . (2010).Google Scholar
- C. McBride and R. Paterson. 2008. Applicative Programming with Effects. Journal of Functional Programming 18, 1 (Jan. 2008). Google Scholar
Digital Library
- J. Midtgaard, M. N. Justesen, P. Kasting, F. Nielson, and H. R. Nielson. 2017. Effect-driven QuickChecking of compilers. In Proceedings of the ACM on Programming Languages, Volume 1 ICFP (2017). Google Scholar
Digital Library
- A. Mista, A. Russo, and J. Hughes. 2018. Branching Processes for QuickCheck Generators (extended version). https://bitbucket.org/ agustinmista/dragen/downloads/full-paper.pdf . (2018).Google Scholar
- N. Mitchell. 2007. Deriving Generic Functions by Example. In Proc. of the 1st York Doctoral Syposium. Tech. Report YCS-2007-421, Department of Computer Science, University of York, UK, 55–62.Google Scholar
- M. Pałka, K. Claessen, A. Russo, and J. Hughes. 2011. Testing and Optimising Compiler by Generating Random Lambda Terms. In The IEEE/ACM International Workshop on Automation of Software Test (AST 2011). Google Scholar
Digital Library
- S. M. Poulding and R. Feldt. 2017. Automated Random Testing in Multiple Dispatch Languages. IEEE International Conference on Software Testing, Verification and Validation (ICST) (2017).Google Scholar
- C. Runciman, M. Naylor, and F. Lindblad. 2008. Smallcheck and Lazy Smallcheck: automatic exhaustive testing for small values. In Proc. of the ACM SIGPLAN Symposium on Haskell. Google Scholar
Digital Library
- T. Sheard and Simon L. Peyton Jones. 2002. Template meta-programming for Haskell. SIGPLAN Notices 37, 12 (2002), 60–75. Google Scholar
Digital Library
- H. W. Watson and F. Galton. 1875. On the probability of the extinction of families. The Journal of the Anthropological Institute of Great Britain and Ireland (1875).Google Scholar
Cross Ref
Index Terms
Branching processes for QuickCheck generators
Recommendations
Generating good generators for inductive relations
Property-based random testing (PBRT) is widely used in the functional programming and verification communities. For testing simple properties, PBRT tools such as QuickCheck can automatically generate random inputs of a given type. But for more complex ...
Branching processes for QuickCheck generators
Haskell 2018: Proceedings of the 11th ACM SIGPLAN International Symposium on HaskellIn QuickCheck (or, more generally, random testing), it is challenging to control random data generators' distributions---specially when it comes to user-defined algebraic data types (ADT). In this paper, we adapt results from an area of mathematics ...
Testing Erlang data types with quviq quickcheck
ERLANG '08: Proceedings of the 7th ACM SIGPLAN workshop on ERLANGWhen creating software, data types are the basic bricks. Most of the time a programmer will use data types defined in library modules, therefore being tested by many users over many years. But sometimes, the appropriate data type is unavailable in the ...







Comments