skip to main content

Data-driven lemma synthesis for interactive proofs

Published:31 October 2022Publication History
Skip Abstract Section

Abstract

Interactive proofs of theorems often require auxiliary helper lemmas to prove the desired theorem. Existing approaches for automatically synthesizing helper lemmas fall into two broad categories. Some approaches are goal-directed, producing lemmas specifically to help a user make progress from a given proof state, but they have limited expressiveness in terms of the lemmas that can be produced. Other approaches are highly expressive, able to generate arbitrary lemmas from a given grammar, but they are completely undirected and hence not amenable to interactive usage.

In this paper, we develop an approach to lemma synthesis that is both goal-directed and expressive. The key novelty is a technique for reducing lemma synthesis to a data-driven program synthesis problem, whereby examples for synthesis are generated from the current proof state. We also describe a technique to systematically introduce new variables for lemma synthesis, as well as techniques for filtering and ranking candidate lemmas for presentation to the user. We implement these ideas in a tool called lfind, which can be run as a Coq tactic. In an evaluation on four benchmark suites, lfind produces useful lemmas in 68% of the cases where a human prover used a lemma to make progress. In these cases lfind synthesizes a lemma that either enables a fully automated proof of the original goal or that matches the human-provided lemma.

References

  1. 1995. Circuits. https://github.com/coq-contribs/circuits Google ScholarGoogle Scholar
  2. 2003. Coq-of-Ocaml. https://github.com/foobar-land/coq-of-ocaml Google ScholarGoogle Scholar
  3. Aws Albarghouthi, Sumit Gulwani, and Zachary Kincaid. 2013. Recursive program synthesis. In International Conference on Computer Aided Verification. 934–950. Google ScholarGoogle ScholarCross RefCross Ref
  4. Angello Astorga, P Madhusudan, Shambwaditya Saha, Shiyu Wang, and Tao Xie. 2019. Learning stateful preconditions modulo a test generator. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation. 775–787. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. R Aubin. 1976. Mechanising Structural Induction. Google ScholarGoogle Scholar
  6. Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, and Stewart Wilcox. 2019. HOList: An Environment for Machine Learning of Higher-Order Theorem Proving (extended version). CoRR, abs/1904.03241 (2019), arxiv:1904.03241. arxiv:1904.03241 Google ScholarGoogle Scholar
  7. Jasmin Christian Blanchette, Sascha Böhme, and Lawrence C. Paulson. 2011. Extending Sledgehammer with SMT Solvers. In Automated Deduction – CADE-23, Nikolaj Bjørner and Viorica Sofronie-Stokkermans (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 116–130. isbn:978-3-642-22438-6 Google ScholarGoogle Scholar
  8. Robert S Boyer and J Strother Moore. 1979. A Computational Logic. ACM Monograph Series. Google ScholarGoogle Scholar
  9. Alan Bundy, Andrew Stevens, Frank Van Harmelen, Andrew Ireland, and Alan Smaill. 1993. Rippling: A heuristic for guiding inductive proofs. Artificial intelligence, 62, 2 (1993), 185–253. Google ScholarGoogle Scholar
  10. Jacqueline Castaing. 1985. How to Facilitate the Proof of Theorems by Using the Induction-matching, and by Generalization. In IJCAI. Google ScholarGoogle Scholar
  11. Harsh Raju Chamarthi, Peter C. Dillinger, Matt Kaufmann, and Panagiotis Manolios. 2011. Integrating Testing and Interactive Theorem Proving. In Proceedings 10th International Workshop on the ACL2 Theorem Prover and its Applications, ACL2 2011, Austin, Texas, USA, November 3-4, 2011, David Hardin and Julien Schmaltz (Eds.) (EPTCS, Vol. 70). 4–19. arxiv:1110.4473 Google ScholarGoogle ScholarCross RefCross Ref
  12. Adam Chlipala. 2013. Certified Programming with Dependent Types: A Pragmatic Introduction to the Coq Proof Assistant. MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In 5th ACM SIGPLAN International Conference on Functional Programming (ICFP) (ICFP). ACM, 268–279. http://www.eecs.northwestern.edu/~robby/courses/395-495-2009-fall/quick.pdf Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Koen Claessen, Moa Johansson, Dan Rosén, and Nicholas Smallbone. 2013. Automating Inductive Proofs Using Theory Exploration. In Automated Deduction - CADE-24 - 24th International Conference on Automated Deduction, Lake Placid, NY, USA, June 9-14, 2013. Proceedings, Maria Paola Bonacina (Ed.) (Lecture Notes in Computer Science, Vol. 7898). Springer, 392–406. isbn:978-3-642-38573-5 Google ScholarGoogle Scholar
  15. Koen Claessen, Nicholas Smallbone, and John Hughes. 2010. QuickSpec: Guessing Formal Specifications Using Testing. In [email protected], Gordon Fraser 0001 and Angelo Gargantini (Eds.) (Lecture Notes in Computer Science, Vol. 6143). Springer, 6–21. isbn:978-3-642-13976-5 Google ScholarGoogle Scholar
  16. Ł ukasz Czajka and Cezary Kaliszyk. 2018. Hammer for Coq: Automation for Dependent Type Theory. Journal of Automated Reasoning, 61, 1 (2018), 01 Jun, 423–453. issn:1573-0670 https://doi.org/10.1007/s10817-018-9458-4 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. 2015. The Lean Theorem Prover (System Description). In Automated Deduction - CADE-25, Amy P. Felty and Aart Middeldorp (Eds.). Springer International Publishing, Cham. 378–388. isbn:978-3-319-21401-6 Google ScholarGoogle Scholar
  18. Lucas Dixon and Jacques Fleuriot. 2003. IsaPlanner: A prototype proof planner in Isabelle. In International Conference on Automated Deduction. 279–283. Google ScholarGoogle ScholarCross RefCross Ref
  19. Michael D. Ernst. 2000. Dynamically Discovering Likely Program Invariants. University of Washington Department of Computer Science and Engineering. Seattle, Washington. Google ScholarGoogle Scholar
  20. P Ezudheen, Daniel Neider, Deepak D’Souza, Pranav Garg, and P Madhusudan. 2018. Horn-ICE learning for synthesizing invariants and contracts. Proceedings of the ACM on Programming Languages, 2, OOPSLA (2018), 1–25. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. John K. Feser, Swarat Chaudhuri, and Isil Dillig. 2015. Synthesizing Data Structure Transformations from Input-Output Examples. SIGPLAN Not., 50, 6 (2015), jun, 229–239. issn:0362-1340 https://doi.org/10.1145/2813885.2737977 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Jean-Christophe Filliâtre, Hugo Herbelin, Bruno Barras, Bruno Barras, Samuel Boutin, Eduardo Giménez, Samuel Boutin, Gérard Huet, César Muñoz, Cristina Cornes, Cristina Cornes, Judicaël Courant, Judicael Courant, Chetan Murthy, Chetan Murthy, Catherine Parent, Catherine Parent, Christine Paulin-mohring, Christine Paulin-mohring, Amokrane Saibi, Amokrane Saibi, Benjamin Werner, and Benjamin Werner. 1997. The Coq Proof Assistant - Reference Manual Version 6.1. Google ScholarGoogle Scholar
  23. Emily First, Yuriy Brun, and Arjun Guha. 2020. TacTok: Semantics-Aware Proof Synthesis. In Object-oriented Programming, Systems, Languages, and Applications. Google ScholarGoogle Scholar
  24. Jonathan Frankle, Peter-Michael Osera, David Walker, and Steve Zdancewic. 2016. Example-Directed Synthesis: A Type-Theoretic Interpretation. SIGPLAN Not., 51, 1 (2016), jan, 802–815. issn:0362-1340 https://doi.org/10.1145/2914770.2837629 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Emilio Jesús Gallego Arias, Karl Palmskog, and Vasily Pestun. 2020. SerAPI:Machine-Friendly, Data-Centric Serialization for Coq. https://github.com/ejgallego/coq-serapi Google ScholarGoogle Scholar
  26. Pranav Garg, Christof Löding, P Madhusudan, and Daniel Neider. 2014. ICE: A robust framework for learning invariants. In International Conference on Computer Aided Verification. 69–87. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Pranav Garg, Daniel Neider, P. Madhusudan, and Dan Roth. 2016. Learning invariants using decision trees and implication counterexamples. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Petersburg, FL, USA, January 20 - 22, 2016, Rastislav Bodík and Rupak Majumdar (Eds.). ACM, 499–512. isbn:978-1-4503-3549-2 http://dl.acm.org/citation.cfm?id=2837614 Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Thibault Gauthier, Cezary Kaliszyk, and Josef Urban. 2017. TacticToe: Learning to Reason with HOL4 Tactics. In LPAR-21. 21st International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Thomas Eiter and David Sands (Eds.) (EPiC Series in Computing, Vol. 46). EasyChair, 125–143. issn:2398-7340 https://doi.org/10.29007/ntlb Google ScholarGoogle ScholarCross RefCross Ref
  29. Jónathan Heras, Ekaterina Komendantskaya, Moa Johansson, and Ewen Maclean. 2013. Proof-Pattern Recognition and Lemma Discovery in ACL2. In Logic for Programming, Artificial Intelligence, and Reasoning - 19th International Conference, LPAR-19, Stellenbosch, South Africa, December 14-19, 2013. Proceedings, Kenneth L. McMillan, Aart Middeldorp, and Andrei Voronkov (Eds.) (Lecture Notes in Computer Science, Vol. 8312). Springer, 389–406. isbn:978-3-642-45220-8 Google ScholarGoogle Scholar
  30. Jane Thurmann Hesketh. 1992. Using Middle-Out Reasoning to Guide Inductive Theorem Proving. Ph.D. Dissertation. University of Edinburgh. Google ScholarGoogle Scholar
  31. Daniel Huang, Prafulla Dhariwal, Dawn Song, and Ilya Sutskever. 2019. GamePad: A Learning Environment for Theorem Proving. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=r1xwKoR9Y7 Google ScholarGoogle Scholar
  32. B Hummel. 1990. Generation of induction axioms and generalisation. Google ScholarGoogle Scholar
  33. Andrew Ireland and Alan Bundy. 1996. Productive use of failure in inductive proof. In Automated Mathematical Induction. Springer, 79–111. Google ScholarGoogle Scholar
  34. Moa Johansson. 2019. Lemma Discovery for Induction - A Survey. In Intelligent Computer Mathematics - 12th International Conference, CICM 2019, Prague, Czech Republic, July 8-12, 2019, Proceedings, Cezary Kaliszyk, Edwin C. Brady, Andrea Kohlhase, and Claudio Sacerdoti Coen (Eds.) (Lecture Notes in Computer Science, Vol. 11617). Springer, 125–139. isbn:978-3-030-23249-8; 978-3-030-23250-4 Google ScholarGoogle Scholar
  35. Moa Johansson, Lucas Dixon, and Alan Bundy. 2010. Dynamic Rippling, Middle-Out Reasoning and Lemma Discovery. In Verification, Induction, Termination Analysis - Festschrift for Christoph Walther on the Occasion of His 60th Birthday, Simon Siegler and Nathan Wasser (Eds.) (Lecture Notes in Computer Science, Vol. 6463). Springer, 102–116. isbn:978-3-642-17171-0 Google ScholarGoogle Scholar
  36. Moa Johansson, Lucas Dixon, and Alan Bundy. 2011. Conjecture Synthesis for Inductive Theories. J. Autom. Reason, 47, 3 (2011), 251–289. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Moa Johansson, Dan Rosén, Nicholas Smallbone, and Koen Claessen. 2014. Hipster: Integrating Theory Exploration in a Proof Assistant. In Intelligent Computer Mathematics - International Conference, CICM 2014, Coimbra, Portugal, July 7-11, 2014. Proceedings, Stephen M. Watt, James H. Davenport, Alan P. Sexton, Petr Sojka, and Josef Urban (Eds.) (Lecture Notes in Computer Science, Vol. 8543). Springer, 108–122. isbn:978-3-319-08433-6 Google ScholarGoogle Scholar
  38. Cezary Kaliszyk and Josef Urban. 2015. HOL(y)Hammer: Online ATP Service for HOL Light. Mathematics in Computer Science, 9, 1 (2015), 01 Mar, 5–22. issn:1661-8289 https://doi.org/10.1007/s11786-014-0182-0 Google ScholarGoogle ScholarCross RefCross Ref
  39. Cezary Kaliszyk and Josef Urban. 2015. MizAR 40 for Mizar 40. Journal of Automated Reasoning, 55, 3 (2015), 01 Oct, 245–256. issn:1573-0670 https://doi.org/10.1007/s10817-015-9330-8 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Deepak Kapur and Mahadevan Subramaniam. 1996. Lemma Discovery in Automated Induction. In Automated Deduction - CADE-13, 13th International Conference on Automated Deduction, New Brunswick, NJ, USA, July 30 - August 3, 1996, Proceedings, Michael A. McRobbie and John K. Slaney (Eds.) (Lecture Notes in Computer Science, Vol. 1104). Springer, 538–552. isbn:3-540-61511-3 Google ScholarGoogle Scholar
  41. Matt Kaufmann and J S. Moore. 1997. An Industrial Strength Theorem Prover for a Logic Based on Common Lisp. IEEE Transactions on Software Engineering, 23, 4 (1997), April, 203–213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Justin Lubin, Nick Collins, Cyrus Omar, and Ravi Chugh. 2020. Program Sketching with Live Bidirectional Evaluation. Proc. ACM Program. Lang., 4, ICFP (2020), Article 109, aug, 29 pages. https://doi.org/10.1145/3408991 Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Anders Miltner, Adrian Trejo Nuñez, Ana Brendel, Swarat Chaudhuri, and Isil Dillig. 2022. Bottom-up Synthesis of Recursive Functional Programs Using Angelic Execution. Proc. ACM Program. Lang., 6, POPL (2022), Article 21, jan, 29 pages. https://doi.org/10.1145/3498682 Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Anders Miltner, Saswat Padhi, David Walker, and Todd Millstein. 2020. Data-driven inference of representation invariants. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Omar Montaño-Rivas, Roy L. McCasland, Lucas Dixon, and Alan Bundy. 2012. Scheme-based theorem discovery and concept invention. Expert Syst. Appl, 39, 2 (2012), 1637–1646. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Peter-Michael Osera and Steve Zdancewic. 2015. Type-and-example-directed program synthesis. ACM SIGPLAN Notices, 50, 6 (2015), 619–630. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Saswat Padhi, Rahul Sharma, and Todd Millstein. 2016. Data-driven precondition inference with learned features. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation. 42–56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Aditya Paliwal, Sarah M. Loos, Markus N. Rabe, Kshitij Bansal, and Christian Szegedy. 2020. Graph Representations for Higher-Order Logic and Theorem Proving. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 2967–2974. https://ojs.aaai.org/index.php/AAAI/article/view/5689 Google ScholarGoogle ScholarCross RefCross Ref
  49. Zoe Paraskevopoulou, Aaron Eline, and Leonidas Lampropoulos. 2022. Computing correctly with inductive relations. In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation. 966–980. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Zoe Paraskevopoulou, Cătălin Hriţcu, Maxime Dénès, Leonidas Lampropoulos, and Benjamin Pierce. 2015. Foundational Property-Based Testing. 9236, https://doi.org/10.1007/978-3-319-22102-1_22 Google ScholarGoogle ScholarCross RefCross Ref
  51. Lawrence C. Paulson. 1993. Natural Deduction as Higher-Order Resolution. CoRR, cs.LO/9301104 (1993), arxiv:cs.LO/9301104 Google ScholarGoogle Scholar
  52. Andrew Reynolds and Viktor Kuncak. 2015. Induction for SMT Solvers. In VMCAI, Deepak D’Souza, Akash Lal, and Kim Guldstrand Larsen (Eds.) (Lecture Notes in Computer Science, Vol. 8931). Springer, 80–98. isbn:978-3-662-46080-1 Google ScholarGoogle Scholar
  53. Alex Sanchez-Stern, Yousef Alhessi, Lawrence K. Saul, and Sorin Lerner. 2020. Generating correctness proofs with neural networks. In Proceedings of the 4th ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, [email protected] 2020, London, UK, June 15, 2020, Koushik Sen and Mayur Naik (Eds.). ACM, 1–10. https://doi.org/10.1145/3394450.3397466 Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Taro Sekiyama, Akifumi Imanishi, and Kohei Suenaga. 2017. Towards Proof Synthesis Guided by Neural Machine Translation for Intuitionistic Propositional Logic. CoRR, abs/1706.06462 (2017), arxiv:1706.06462. arxiv:1706.06462 Google ScholarGoogle Scholar
  55. Armando Solar-Lezama. 2009. The Sketching Approach to Program Synthesis. In Programming Languages and Systems, 7th Asian Symposium, APLAS 2009, Seoul, Korea, December 14-16, 2009. Proceedings, Zhenjiang Hu (Ed.) (Lecture Notes in Computer Science, Vol. 5904). Springer, 4–13. isbn:978-3-642-10671-2 Google ScholarGoogle Scholar
  56. William Sonnex, Sophia Drossopoulou, and Susan Eisenbach. 2012. Zeno: An automated prover for properties of recursive data structures. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 407–421. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Daniel Whalen. 2016. Holophrasm: a neural Automated Theorem Prover for higher-order logic. arxiv:1608.02644. Google ScholarGoogle Scholar
  58. Kaiyu Yang and Jia Deng. 2019. Learning to Prove Theorems via Interacting with Proof Assistants. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, 6984–6994. http://proceedings.mlr.press/v97/yang19a.html Google ScholarGoogle Scholar
  59. Weikun Yang, Grigory Fedyukovich, and Aarti Gupta. 2019. Lemma synthesis for automating induction over algebraic data types. In International Conference on Principles and Practice of Constraint Programming. 600–617. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. He Zhu, Stephen Magill, and Suresh Jagannathan. 2018. A data-driven CHC solver. ACM SIGPLAN Notices, 53, 4 (2018), 707–721. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Data-driven lemma synthesis for interactive proofs

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Article Metrics

        • Downloads (Last 12 months)233
        • Downloads (Last 6 weeks)21

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!