skip to main content

Random testing of a higher-order blockchain language (experience report)

Published:31 August 2022Publication History
Skip Abstract Section

Abstract

We describe our experience of using property-based testing---an approach for automatically generating random inputs to check executable program specifications---in a development of a higher-order smart contract language that powers a state-of-the-art blockchain with thousands of active daily users.

We outline the process of integrating QuickChick---a framework for property-based testing built on top of the Coq proof assistant---into a real-world language implementation in OCaml. We discuss the challenges we have encountered when generating well-typed programs for a realistic higher-order smart contract language, which mixes purely functional and imperative computations and features runtime resource accounting. We describe the set of the language implementation properties that we tested, as well as the semantic harness required to enable their validation. The properties range from the standard type safety to the soundness of a control- and type-flow analysis used by the optimizing compiler. Finally, we present the list of bugs discovered and rediscovered with the help of QuickChick and discuss their severity and possible ramifications.

References

  1. Sam Blackshear, Evan Cheng, David L. Dill, Victor Gao, Ben Maurer, Todd Nowacki, Alistair Pott, Shaz Qadeer, Rain, Dario Russi, Stephane Sezer, Tim Zakian, and Runtian Zhou. 2019. Move: A Language With Programmable Resources. Available at https://developers.diem.com/papers/diem-move-a-language-with-programmable-resources/2019-06-18.pdf Google ScholarGoogle Scholar
  2. Ethan Cecchetti, Siqiu Yao, Haobin Ni, and Andrew C. Myers. 2021. Compositional Security for Reentrant Applications. In 42nd IEEE Symposium on Security and Privacy. IEEE, 1249–1267. https://doi.org/10.1109/SP40001.2021.00084 Google ScholarGoogle ScholarCross RefCross Ref
  3. Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In ICFP. ACM, 268–279. https://doi.org/10.1145/351240.351266 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Guillaume Claret. 2021. Coq-of-OCaml, A Compiler from OCaml to Coq. Available at https://formal.land/docs/coq-of-ocaml/introduction Google ScholarGoogle Scholar
  5. Michael Coblenz. 2017. Obsidian: A Safer Blockchain Programming Language. In ICSE (Companion). IEEE Press, 97–99. https://doi.org/10.1109/ICSE-C.2017.150 Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. David Darais, Nicholas Labich, Phuc C. Nguyen, and David Van Horn. 2017. Abstracting Definitional Interpreters (Functional Pearl). PACMPL, 1, ICFP (2017), 12:1–12:25. https://doi.org/10.1145/3110256 Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Ankush Das, Stephanie Balzer, Jan Hoffmann, Frank Pfenning, and Ishani Santurkar. 2021. Resource-Aware Session Types for Digital Contracts. In CSF. IEEE, 1–16. https://doi.org/10.1109/CSF51468.2021.00004 Google ScholarGoogle ScholarCross RefCross Ref
  8. Ethereum Foundation. 2018. ERC20 Token Standard. https://en.bitcoinwiki.org/wiki/ERC20 Online Google ScholarGoogle Scholar
  9. Andrzej Filinski. 1994. Representing Monads. In POPL. ACM Press, 446–457. https://doi.org/10.1145/174675.178047 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Matthew Fluet. 2012. A Type- and Control-Flow Analysis for System F. In IFL (LNCS, Vol. 8241). Springer, 122–139. https://doi.org/10.1007/978-3-642-41582-1_8 Google ScholarGoogle ScholarCross RefCross Ref
  11. Jean-Yves Girard. 1972. Interprétation fonctionnelle et élimination des coupures de l’arithmétique d’ordre supérieur. Université de Paris VII. Paris, France. Google ScholarGoogle Scholar
  12. Harrison Goldstein, John Hughes, Leonidas Lampropoulos, and Benjamin C. Pierce. 2021. Do Judge a Test by its Cover - Combining Combinatorial and Property-Based Testing. In ESOP (LNCS, Vol. 12648). Springer, 264–291. https://doi.org/10.1007/978-3-030-72019-3_10 Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Emin Gün Sirer. 2016. Reentrancy Woes in Smart Contracts. http://hackingdistributed.com/2016/07/13/reentrancy-woes/ Google ScholarGoogle Scholar
  14. Andreas Haas, Andreas Rossberg, Derek L. Schuff, Ben L. Titzer, Michael Holman, Dan Gohman, Luke Wagner, Alon Zakai, and J. F. Bastien. 2017. Bringing the web up to speed with WebAssembly. In PLDI. ACM, 185–200. https://doi.org/10.1145/3062341.3062363 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Tram Hoang, Anton Trunov, Leonidas Lampropoulos, and Ilya Sergey. 2022. Random Testing of a Higher-Order Blockchain Language (ICFP 2022 Artifact): Code and Commentary. https://doi.org/10.5281/zenodo.6610599 Google ScholarGoogle ScholarCross RefCross Ref
  16. Catalin Hritcu, Leonidas Lampropoulos, Antal Spector-Zabusky, Arthur Azevedo de Amorim, Maxime Dénès, John Hughes, Benjamin C. Pierce, and Dimitrios Vytiniotis. 2016. Testing Noninterference, Quickly. J. Funct. Program., 26 (2016), e4. https://doi.org/10.1017/S0956796816000058 Google ScholarGoogle ScholarCross RefCross Ref
  17. IOHK Foundation. 2019. Plutus: A Functional Contract Platform. https://testnet.iohkdev.io/en/plutus/ Online; accessed 23 February 2022 Google ScholarGoogle Scholar
  18. Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In ISSTA. ACM, 239–250. https://doi.org/10.1145/3293882.3330553 Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Ramana Kumar, Magnus O. Myreen, Michael Norrish, and Scott Owens. 2014. CakeML: a verified implementation of ML. In POPL. ACM, 179–192. https://doi.org/10.1145/2535838.2535841 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Leonidas Lampropoulos. 2018. Random Testing for Language Design. Ph. D. Dissertation. University of Pennsylvania. Google ScholarGoogle Scholar
  21. Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. 2019. Coverage guided, property based testing. Proc. ACM Program. Lang., 3, OOPSLA, 181:1–181:29. https://doi.org/10.1145/3360607 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Leonidas Lampropoulos, Zoe Paraskevopoulou, and Benjamin C. Pierce. 2018. Generating good generators for inductive relations. PACMPL, 2, POPL (2018), 45:1–45:30. https://doi.org/10.1145/3158133 Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Leonidas Lampropoulos and Benjamin C. Pierce. 2018. QuickChick: Property-Based Testing In Coq. Software Foundations series, Volume 4. Electronic textbook. https://softwarefoundations.cis.upenn.edu/ Google ScholarGoogle Scholar
  24. Xavier Leroy. 2006. Formal certification of a compiler back-end or: programming a compiler with a proof assistant. In POPL. ACM, 42–54. https://doi.org/10.1145/1111037.1111042 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Andreas Löscher and Konstantinos Sagonas. 2017. Targeted Property-Based Testing. In ISSTA. ACM, 46–56. https://doi.org/10.1145/3092703.3092711 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Magnus Madsen and Ondrej Lhoták. 2018. Safe and sound program analysis with Flix. In ISSTA. ACM, 38–48. https://doi.org/10.1145/3213846.3213847 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jan Midtgaard, Mathias Nygaard Justesen, Patrick Kasting, Flemming Nielson, and Hanne Riis Nielson. 2017. Effect-driven QuickChecking of compilers. Proc. ACM Program. Lang., 1, ICFP (2017), 15:1–15:23. https://doi.org/10.1145/3110259 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jan Midtgaard and Anders Møller. 2015. QuickChecking Static Analysis Properties. In ICST. IEEE Computer Society, 1–10. https://doi.org/10.1109/ICST.2015.7102603 Google ScholarGoogle ScholarCross RefCross Ref
  29. Matthew Might. 2010. Abstract Interpreters for Free. In SAS (LNCS, Vol. 6337). Springer, 407–421. https://doi.org/10.1007/978-3-642-15769-1_25 Google ScholarGoogle ScholarCross RefCross Ref
  30. Vaivaswatha Nagaraj, Jacob Johannsen, Anton Trunov, George Pîrlea, Amrit Kumar, and Ilya Sergey. 2020. Compiling a Higher-Order Smart Contract Language to LLVM. CoRR, abs/2008.05555 (2020), arxiv:2008.05555 Google ScholarGoogle Scholar
  31. Flemming Nielson, Hanne Riis Nielson, and Chris Hankin. 1999. Principles of Program Analysis. Springer. https://doi.org/10.1007/978-3-662-03811-6 Google ScholarGoogle ScholarCross RefCross Ref
  32. Michał H. Pał ka, Koen Claessen, Alejandro Russo, and John Hughes. 2011. Testing an optimising compiler by generating random lambda terms. In AST. ACM, 91–97. https://doi.org/10.1145/1982595.1982615 Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Daniel Perez and Benjamin Livshits. 2020. Broken Metre: Attacking Resource Metering in EVM. In NDSS. The Internet Society. Google ScholarGoogle Scholar
  34. George Pîrlea, Amrit Kumar, and Ilya Sergey. 2021. Practical Smart Contract Sharding with Ownership and Commutativity Analysis. In PLDI. ACM, 1327–1341. https://doi.org/10.1145/3453483.3454112 Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. John C. Reynolds. 1974. Towards a theory of type structure. In Programming Symposium (LNCS, Vol. 19). Springer, 408–423. https://doi.org/10.1007/3-540-06859-7_148 Google ScholarGoogle ScholarCross RefCross Ref
  36. Franklin Schrans. 2018. Writing Safe Smart Contracts in Flint. Master’s thesis. Imperial College London. Google ScholarGoogle Scholar
  37. Ilya Sergey, Dominique Devriese, Matthew Might, Jan Midtgaard, David Darais, Dave Clarke, and Frank Piessens. 2013. Monadic Abstract Interpreters. In PLDI. ACM, 399–410. https://doi.org/10.1145/2491956.2491979 Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ilya Sergey, Vaivaswatha Nagaraj, Jacob Johannsen, Amrit Kumar, Anton Trunov, and Ken Chan Guan Hao. 2019. Safer smart contract programming with Scilla. PACMPL, 3, OOPSLA (2019), 185:1–185:30. https://doi.org/10.1145/3360611 Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Olin Shivers. 1991. Control-Flow Analysis of Higher-Order Languages or Taming Lambda. Ph. D. Dissertation. School of Computer Science, Carnegie Mellon University. Google ScholarGoogle Scholar
  40. Jubi Taneja, Zhengyang Liu, and John Regehr. 2020. Testing static analyses for precision and soundness. In CGO. ACM, 81–93. https://doi.org/10.1145/3368826.3377927 Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Tezos Foundation. 2018. Michelson: the language of Smart Contracts in Tezos. https://tezos.gitlab.io/whitedoc/michelson.html Online; accessed 23 February 2022 Google ScholarGoogle Scholar
  42. Stephen Weeks. 2006. Whole-Program Compilation in MLton. In Proceedings of the 2006 Workshop on ML. ACM. https://doi.org/10.1145/1159876.1159877 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Gavin Wood. 2014. Ethereum: A Secure Decentralized Generalised Transaction Ledger. Google ScholarGoogle Scholar
  44. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In PLDI. ACM, 283–294. https://doi.org/10.1145/1993498.1993532 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Random testing of a higher-order blockchain language (experience report)

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Article Metrics

      • Downloads (Last 12 months)196
      • Downloads (Last 6 weeks)8

      Other Metrics

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!