Abstract
Two fundamental goals in programming are correctness (producing the right results) and efficiency (using as few resources as possible). Property-based testing tools such as QuickCheck provide a lightweight means to check the correctness of Haskell programs, but what about their efficiency? In this article, we show how QuickCheck can be combined with the Criterion benchmarking library to give a lightweight means to compare the time performance of Haskell programs. We present the design and implementation of the AutoBench system, demonstrate its utility with a number of case studies, and find that many QuickCheck correctness properties are also efficiency improvements.
Supplemental Material
Available for Download
In this supplementary material we give larger versions of each of the time performance figures from the main article.
- T. Arts, J. Hughes, J. Johansson, and U.T. Wiger. 2006. Testing Telecoms Software with Quviq QuickCheck. Erlang Workshop. Google Scholar
Digital Library
- D. Beardsley. 2018. Fake. https://hackage.haskell.org/package/fake .Google Scholar
- J. Bernardy, P. Jansson, and K. Claessen. 2010. Testing Polymorphic Properties. ESOP. Google Scholar
Digital Library
- N. Brown. 2010. Progression. https://hackage.haskell.org/package/ progression .Google Scholar
- J. Christiansen and S. Fischer. 2008. EasyCheck - Test Data for Free. FLOPS . Google Scholar
Digital Library
- K. Claessen, J. Duregård, and M.H. Palka. 2015. Generating Constrained Random Data with Uniform Distribution. JFP.Google Scholar
- K. Claessen and J. Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. ICFP. Google Scholar
Digital Library
- K. Claessen and J. Hughes. 2002. Testing Monadic Code with QuickCheck. Haskell Workshop. Google Scholar
Digital Library
- K. Claessen, N. Smallbone, and J. Hughes. 2010. QuickSpec: Guessing Formal Specifications Using Testing. TAP. Google Scholar
Digital Library
- E. Coppa, C. Demetrescu, and I. Finocchi. 2012. Input-Sensitive Profiling. PLDI. Google Scholar
Digital Library
- E. Coppa, C. Demetrescu, I. Finocchi, and R. Marotta. 2014. Estimating the Empirical Cost Function of Routines with Dynamic Workloads. CGO . Google Scholar
Digital Library
- T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein. 2009. Introduction to Algorithms . MIT Press. Google Scholar
Digital Library
- T. Docker. 2006. Chart. http://hackage.haskell.org/package/Chart .Google Scholar
- V. Estivill-Castro and D.Wood. 1992. A Survey of Adaptive Sorting Algorithms. ACM Computing Surveys. Google Scholar
Digital Library
- C.McGeoch et al. 2002. Using Finite Experiments to Study Asymptotic Performance. In Experimental Algorithmics. Springer. Google Scholar
Digital Library
- C. Klein et al. 2012. Run Your Research: On the Effectiveness of Lightweight Mechanization. POPL. Google Scholar
Digital Library
- J. Fox. 1997. Applied Regression Analysis, Linear Models, and Related Methods . Sage Publications.Google Scholar
- S.F. Goldsmith, A.S.Aiken, and D.S. Wilkerson. 2007. Measuring Empirical Computational Complexity. ESEC-FSE. Google Scholar
Digital Library
- D. Gorin. 2015. Hint. https://hackage.haskell.org/package/hint .Google Scholar
- M.A.T. Handley. 2018. GitHub Repository for AutoBench. https: //github.com/mathandley/AutoBench .Google Scholar
- A.E. Hoerl and R.W. Kennard. 1970. Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics. Google Scholar
Digital Library
- J. Hughes. 2007. QuickCheck Testing for Fun and Profit. PADL. Google Scholar
Digital Library
- G. Hutton. 2016. Programming in Haskell. Cambridge University Press. Google Scholar
Digital Library
- D.G. Hyams. 2010. CurveExpert. http://www.curveexpert.net .Google Scholar
- M.P. Jones. 1999. Hugs. https://www.haskell.org/hugs .Google Scholar
- L.G.L.T. Meertens. 2004. Calculating the Sieve of Eratosthenes. JFP. Google Scholar
Digital Library
- A.K. Moran and D. Sands. 1999. Improvement in a Lazy Context: An Operational Theory for Call-By-Need. POPL. Google Scholar
Digital Library
- G.E. Moss. 2000. Benchmarking Purely Functional Data Structures. Ph.D. Dissertation. University of York.Google Scholar
- The University of Glasgow. 2001. Base: Data.List. http://hackage. haskell.org/package/base-4.11.1.0/docs/src/Data.OldList.html#sort .Google Scholar
- The University of Glasgow. 2014. Process. https://hackage.haskell. org/package/process .Google Scholar
- The University of Glasgow. 2015. Glasgow Haskell Compiler User’s Guide: Profiling. http://downloads.haskell.org/~ghc/latest/docs/html/ users_guide/profiling.html .Google Scholar
- R. O’Keefe. 1982. A Smooth Applicative Merge Sort. Department of Artificial Intelligence, University of Edinburgh.Google Scholar
- M.E. O’Neill. 2009. The Genuine Sieve of Eratosthenes. JFP. Google Scholar
Digital Library
- OriginLab. 2000. Origin. https://www.originlab.com .Google Scholar
- B. O’Sullivan. 2009. Criterion: Robust Reliable Performance Measurement and Analysis. http://www.serpentine.com/criterion .Google Scholar
- C. Runciman, M. Naylor, and F. Lindblad. 2008. Smallcheck and Lazy Smallcheck: Automatic Exhaustive Testing for Small Values. Haskell Symposium . Google Scholar
Digital Library
- Systat Software. 2002. TableCurve 2D. http://www.sigmaplot.co.uk .Google Scholar
- The GHC Team. 2017. GHC. https://hackage.haskell.org/package/ghc .Google Scholar
- P. Wadler. 1987. The Concatenate Vanishes. University of Glasgow.Google Scholar
- Q. Xu and Y. Liang. 2001. Monte Carlo Cross Validation. Chemometrics and Intelligent Laboratory Systems .Google Scholar
- D. Zaparanuks and M. Hauswirth. 2012. Algorithmic Profiling. PLDI. Google Scholar
Digital Library
Index Terms
AutoBench: comparing the time performance of Haskell programs
Recommendations
AutoBench: comparing the time performance of Haskell programs
Haskell 2018: Proceedings of the 11th ACM SIGPLAN International Symposium on HaskellTwo fundamental goals in programming are correctness (producing the right results) and efficiency (using as few resources as possible). Property-based testing tools such as QuickCheck provide a lightweight means to check the correctness of Haskell ...
Strength Induction in a Haskell Program Verifier
Haskell employs a melange of strict and non-strict evaluation semantics, hence a Haskell verifier should be capable of checking assumptions that program variables may or may not denote well-defined values. The paper introduces a new strategy, called ...
Call-pattern specialisation for Haskell programs
Proceedings of the ICFP '07 conferenceUser-defined data types, pattern-matching, and recursion are ubiquitous features of Haskell programs. Sometimes a function is called with arguments that are statically known to be in constructor form, so that the work of pattern-matching is wasted. Even ...







Comments