Abstract
We describe our experience of using property-based testing---an approach for automatically generating random inputs to check executable program specifications---in a development of a higher-order smart contract language that powers a state-of-the-art blockchain with thousands of active daily users.
We outline the process of integrating QuickChick---a framework for property-based testing built on top of the Coq proof assistant---into a real-world language implementation in OCaml. We discuss the challenges we have encountered when generating well-typed programs for a realistic higher-order smart contract language, which mixes purely functional and imperative computations and features runtime resource accounting. We describe the set of the language implementation properties that we tested, as well as the semantic harness required to enable their validation. The properties range from the standard type safety to the soundness of a control- and type-flow analysis used by the optimizing compiler. Finally, we present the list of bugs discovered and rediscovered with the help of QuickChick and discuss their severity and possible ramifications.
- Sam Blackshear, Evan Cheng, David L. Dill, Victor Gao, Ben Maurer, Todd Nowacki, Alistair Pott, Shaz Qadeer, Rain, Dario Russi, Stephane Sezer, Tim Zakian, and Runtian Zhou. 2019. Move: A Language With Programmable Resources. Available at https://developers.diem.com/papers/diem-move-a-language-with-programmable-resources/2019-06-18.pdf
Google Scholar
- Ethan Cecchetti, Siqiu Yao, Haobin Ni, and Andrew C. Myers. 2021. Compositional Security for Reentrant Applications. In 42nd IEEE Symposium on Security and Privacy. IEEE, 1249–1267. https://doi.org/10.1109/SP40001.2021.00084
Google Scholar
Cross Ref
- Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In ICFP. ACM, 268–279. https://doi.org/10.1145/351240.351266
Google Scholar
Digital Library
- Guillaume Claret. 2021. Coq-of-OCaml, A Compiler from OCaml to Coq. Available at https://formal.land/docs/coq-of-ocaml/introduction
Google Scholar
- Michael Coblenz. 2017. Obsidian: A Safer Blockchain Programming Language. In ICSE (Companion). IEEE Press, 97–99. https://doi.org/10.1109/ICSE-C.2017.150
Google Scholar
Digital Library
- David Darais, Nicholas Labich, Phuc C. Nguyen, and David Van Horn. 2017. Abstracting Definitional Interpreters (Functional Pearl). PACMPL, 1, ICFP (2017), 12:1–12:25. https://doi.org/10.1145/3110256
Google Scholar
Digital Library
- Ankush Das, Stephanie Balzer, Jan Hoffmann, Frank Pfenning, and Ishani Santurkar. 2021. Resource-Aware Session Types for Digital Contracts. In CSF. IEEE, 1–16. https://doi.org/10.1109/CSF51468.2021.00004
Google Scholar
Cross Ref
- Ethereum Foundation. 2018. ERC20 Token Standard. https://en.bitcoinwiki.org/wiki/ERC20 Online
Google Scholar
- Andrzej Filinski. 1994. Representing Monads. In POPL. ACM Press, 446–457. https://doi.org/10.1145/174675.178047
Google Scholar
Digital Library
- Matthew Fluet. 2012. A Type- and Control-Flow Analysis for System F. In IFL (LNCS, Vol. 8241). Springer, 122–139. https://doi.org/10.1007/978-3-642-41582-1_8
Google Scholar
Cross Ref
- Jean-Yves Girard. 1972. Interprétation fonctionnelle et élimination des coupures de l’arithmétique d’ordre supérieur. Université de Paris VII. Paris, France.
Google Scholar
- Harrison Goldstein, John Hughes, Leonidas Lampropoulos, and Benjamin C. Pierce. 2021. Do Judge a Test by its Cover - Combining Combinatorial and Property-Based Testing. In ESOP (LNCS, Vol. 12648). Springer, 264–291. https://doi.org/10.1007/978-3-030-72019-3_10
Google Scholar
Digital Library
- Emin Gün Sirer. 2016. Reentrancy Woes in Smart Contracts. http://hackingdistributed.com/2016/07/13/reentrancy-woes/
Google Scholar
- Andreas Haas, Andreas Rossberg, Derek L. Schuff, Ben L. Titzer, Michael Holman, Dan Gohman, Luke Wagner, Alon Zakai, and J. F. Bastien. 2017. Bringing the web up to speed with WebAssembly. In PLDI. ACM, 185–200. https://doi.org/10.1145/3062341.3062363
Google Scholar
Digital Library
- Tram Hoang, Anton Trunov, Leonidas Lampropoulos, and Ilya Sergey. 2022. Random Testing of a Higher-Order Blockchain Language (ICFP 2022 Artifact): Code and Commentary. https://doi.org/10.5281/zenodo.6610599
Google Scholar
Cross Ref
- Catalin Hritcu, Leonidas Lampropoulos, Antal Spector-Zabusky, Arthur Azevedo de Amorim, Maxime Dénès, John Hughes, Benjamin C. Pierce, and Dimitrios Vytiniotis. 2016. Testing Noninterference, Quickly. J. Funct. Program., 26 (2016), e4. https://doi.org/10.1017/S0956796816000058
Google Scholar
Cross Ref
- IOHK Foundation. 2019. Plutus: A Functional Contract Platform. https://testnet.iohkdev.io/en/plutus/ Online; accessed 23 February 2022
Google Scholar
- Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In ISSTA. ACM, 239–250. https://doi.org/10.1145/3293882.3330553
Google Scholar
Digital Library
- Ramana Kumar, Magnus O. Myreen, Michael Norrish, and Scott Owens. 2014. CakeML: a verified implementation of ML. In POPL. ACM, 179–192. https://doi.org/10.1145/2535838.2535841
Google Scholar
Digital Library
- Leonidas Lampropoulos. 2018. Random Testing for Language Design. Ph. D. Dissertation. University of Pennsylvania.
Google Scholar
- Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. 2019. Coverage guided, property based testing. Proc. ACM Program. Lang., 3, OOPSLA, 181:1–181:29. https://doi.org/10.1145/3360607
Google Scholar
Digital Library
- Leonidas Lampropoulos, Zoe Paraskevopoulou, and Benjamin C. Pierce. 2018. Generating good generators for inductive relations. PACMPL, 2, POPL (2018), 45:1–45:30. https://doi.org/10.1145/3158133
Google Scholar
Digital Library
- Leonidas Lampropoulos and Benjamin C. Pierce. 2018. QuickChick: Property-Based Testing In Coq. Software Foundations series, Volume 4. Electronic textbook. https://softwarefoundations.cis.upenn.edu/
Google Scholar
- Xavier Leroy. 2006. Formal certification of a compiler back-end or: programming a compiler with a proof assistant. In POPL. ACM, 42–54. https://doi.org/10.1145/1111037.1111042
Google Scholar
Digital Library
- Andreas Löscher and Konstantinos Sagonas. 2017. Targeted Property-Based Testing. In ISSTA. ACM, 46–56. https://doi.org/10.1145/3092703.3092711
Google Scholar
Digital Library
- Magnus Madsen and Ondrej Lhoták. 2018. Safe and sound program analysis with Flix. In ISSTA. ACM, 38–48. https://doi.org/10.1145/3213846.3213847
Google Scholar
Digital Library
- Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, and Hanne Riis Nielson. 2017. Effect-driven QuickChecking of compilers. Proc. ACM Program. Lang., 1, ICFP (2017), 15:1–15:23. https://doi.org/10.1145/3110259
Google Scholar
Digital Library
- Jan Midtgaard and Anders Møller. 2015. QuickChecking Static Analysis Properties. In ICST. IEEE Computer Society, 1–10. https://doi.org/10.1109/ICST.2015.7102603
Google Scholar
Cross Ref
- Matthew Might. 2010. Abstract Interpreters for Free. In SAS (LNCS, Vol. 6337). Springer, 407–421. https://doi.org/10.1007/978-3-642-15769-1_25
Google Scholar
Cross Ref
- Vaivaswatha Nagaraj, Jacob Johannsen, Anton Trunov, George Pîrlea, Amrit Kumar, and Ilya Sergey. 2020. Compiling a Higher-Order Smart Contract Language to LLVM. CoRR, abs/2008.05555 (2020), arxiv:2008.05555
Google Scholar
- Flemming Nielson, Hanne Riis Nielson, and Chris Hankin. 1999. Principles of Program Analysis. Springer. https://doi.org/10.1007/978-3-662-03811-6
Google Scholar
Cross Ref
- Michał H. Pał ka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an optimising compiler by generating random lambda terms. In AST. ACM, 91–97. https://doi.org/10.1145/1982595.1982615
Google Scholar
Digital Library
- Daniel Perez and Benjamin Livshits. 2020. Broken Metre: Attacking Resource Metering in EVM. In NDSS. The Internet Society.
Google Scholar
- George Pîrlea, Amrit Kumar, and Ilya Sergey. 2021. Practical Smart Contract Sharding with Ownership and Commutativity Analysis. In PLDI. ACM, 1327–1341. https://doi.org/10.1145/3453483.3454112
Google Scholar
Digital Library
- John C. Reynolds. 1974. Towards a theory of type structure. In Programming Symposium (LNCS, Vol. 19). Springer, 408–423. https://doi.org/10.1007/3-540-06859-7_148
Google Scholar
Cross Ref
- Franklin Schrans. 2018. Writing Safe Smart Contracts in Flint. Master’s thesis. Imperial College London.
Google Scholar
- Ilya Sergey, Dominique Devriese, Matthew Might, Jan Midtgaard, David Darais, Dave Clarke, and Frank Piessens. 2013. Monadic Abstract Interpreters. In PLDI. ACM, 399–410. https://doi.org/10.1145/2491956.2491979
Google Scholar
Digital Library
- Ilya Sergey, Vaivaswatha Nagaraj, Jacob Johannsen, Amrit Kumar, Anton Trunov, and Ken Chan Guan Hao. 2019. Safer smart contract programming with Scilla. PACMPL, 3, OOPSLA (2019), 185:1–185:30. https://doi.org/10.1145/3360611
Google Scholar
Digital Library
- Olin Shivers. 1991. Control-Flow Analysis of Higher-Order Languages or Taming Lambda. Ph. D. Dissertation. School of Computer Science, Carnegie Mellon University.
Google Scholar
- Jubi Taneja, Zhengyang Liu, and John Regehr. 2020. Testing static analyses for precision and soundness. In CGO. ACM, 81–93. https://doi.org/10.1145/3368826.3377927
Google Scholar
Digital Library
- Tezos Foundation. 2018. Michelson: the language of Smart Contracts in Tezos. https://tezos.gitlab.io/whitedoc/michelson.html Online; accessed 23 February 2022
Google Scholar
- Stephen Weeks. 2006. Whole-Program Compilation in MLton. In Proceedings of the 2006 Workshop on ML. ACM. https://doi.org/10.1145/1159876.1159877
Google Scholar
Digital Library
- Gavin Wood. 2014. Ethereum: A Secure Decentralized Generalised Transaction Ledger.
Google Scholar
- Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In PLDI. ACM, 283–294. https://doi.org/10.1145/1993498.1993532
Google Scholar
Digital Library
Index Terms
Random testing of a higher-order blockchain language (experience report)
Recommendations
Generating good generators for inductive relations
Property-based random testing (PBRT) is widely used in the functional programming and verification communities. For testing simple properties, PBRT tools such as QuickCheck can automatically generate random inputs of a given type. But for more complex ...
Towards random and enumerative testing for OCaml and WhyML properties
AbstractDeductive program verification greatly improves software quality, but proving formal specifications is difficult, and this activity can only be partially automated. It is therefore relevant to supplement deductive verification tools, such as Why3, ...
Beginner's luck: a language for property-based generators
POPL '17: Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming LanguagesProperty-based random testing à la QuickCheck requires building efficient generators for well-distributed random data satisfying complex logical predicates, but writing these generators can be difficult and error prone. We propose a domain-specific ...






Comments