research-article

A formal proof of PAC learnability for decision stumps

Authors:

Joseph Tassarotti,

Koundinya Vajjha,

Anindya Banerjee,

Jean-Baptiste TristanAuthors Info & Claims

CPP 2021: Proceedings of the 10th ACM SIGPLAN International Conference on Certified Programs and Proofs

Pages 5 - 17

https://doi.org/10.1145/3437992.3439917

Published: 20 January 2021 Publication History

Abstract

We present a formal proof in Lean of probably approximately correct (PAC) learnability of the concept class of decision stumps. This classic result in machine learning theory derives a bound on error probabilities for a simple type of classifier. Though such a proof appears simple on paper, analytic and measure-theoretic subtleties arise when carrying it out fully formally. Our proof is structured so as to separate reasoning about deterministic properties of a learning function from proofs of measurability and analysis of probabilities.

References

[1]

Reynald Afeldt, Cyril Cohen, Marie Kerjean, Assia Mahboubi, Damien Rouhling, Kazuhiko Sakaguchi, and Pierre-Yves Strub. 2020. mathcomp Analysis Library. htps://github.com/math-comp/analysis.

[2]

Reynald Afeldt, Manabu Hagiwara, and Jonas Sénizergues. 2014. Formalization of Shannon's Theorems. J. Autom. Reason. 53, 1 ( 2014 ), 63-103.

[3]

Philippe Audebaud and Christine Paulin-Mohring. 2009. Proofs of randomized algorithms in Coq. Sci. Comput. Program. 74, 8 ( 2009 ), 568-589.

[4]

Robert J. Aumann. 1961. Borel structures for function spaces. Illinois J. Math. 5, 4 ( 12 1961 ), 614-630.

[5]

Jeremy Avigad, Johannes Hölzl, and Luke Serafin. 2017. A Formally Verified Proof of the Central Limit Theorem. J. Autom. Reason. 59, 4 ( 2017 ), 389-423.

[6]

Alexander Bagnall and Gordon Stewart. 2019. Certifying the True Error: Machine Learning in Coq with Verified Generalization Guarantees. In AAAI'19: The Thirty-Third AAAI Conference on Artificial Intelligence. 2662-2669.

[7]

Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, and Stewart Wilcox. 2019. HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving. In Thirty-sixth International Conference on Machine Learning (ICML). 454-463.

[8]

Gilles Barthe, François Dupressoir, Benjamin Grégoire, César Kunz, Benedikt Schmidt, and Pierre-Yves Strub. 2013. EasyCrypt: A Tutorial. In Foundations of Security Analysis and Design VII-FOSAD 2012 /2013 Tutorial Lectures. 146-166.

[9]

Gilles Barthe, Benjamin Grégoire, and Santiago Zanella Béguelin. 2009. Formal certification of code-based cryptographic proofs. In POPL. 90-101.

Digital Library

[10]

Alexander Bentkamp, Jasmin Christian Blanchette, and Dietrich Klakow. 2019. A formal proof of the expressiveness of deep learning. Journal of Automated Reasoning 63, 2 ( 2019 ), 347-368.

Digital Library

[11]

Martin E. Bidlingmaier, Florian Faissole, and Bas Spitters. 2019. Synthetic topology in Homotopy Type Theory for probabilistic programming. CoRR abs/1912.07339 ( 2019 ). arXiv: 1912.07339 htp: //arxiv.org/abs/ 1912.07339

[12]

Bruno Blanchet. 2006. A Computationally Sound Mechanized Prover for Security Protocols. In 2006 IEEE Symposium on Security and Privacy. 140-154.

Digital Library

[13]

Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. 1989. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM (JACM) 36, 4 ( 1989 ), 929-965.

Digital Library

[14]

Nadav Cohen, Or Sharir, and Amnon Shashua. 2016. On the Expressive Power of Deep Learning: A Tensor Analysis. In Proceedings of the 29th Conference on Learning Theory, COLT 2016. 698-728.

[15]

Leonardo Mendonça de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. 2015. The Lean Theorem Prover (System Description). In CADE-25-25th International Conference on Automated Deduction. 378-388.

[16]

R. M. Dudley. 2014. Uniform Central Limit Theorems (2nd ed.). Cambridge University Press.

[17]

Manuel Eberl, Max W. Haslbeck, and Tobias Nipkow. 2018. Verified Analysis of Random Trees. In ITP. 196-214.

[18]

Michèle Giry. 1982. A Categorical Approach to Probability Theory. In Categorical Aspects of Topology and Analysis (Lecture Notes in Mathematics, Vol. 915 ), B. Banaschewski (Ed.). 68-85.

[19]

Kiran Gopinathan and Ilya Sergey. 2020. Certifying Certainty and Uncertainty in Approximate Membership Query Structures. In CAV, Shuvendu K. Lahiri and Chao Wang (Eds.). 279-303.

[20]

Johannes Hölzl. 2013. Construction and stochastic applications of measure spaces in higher-order logic. Ph.D. Dissertation. Technical University Munich.

[21]

Johannes Hölzl and Armin Heller. 2011. Three Chapters of Measure Theory in Isabelle/HOL. In ITP. 135-151.

[22]

Daniel Huang, Prafulla Dhariwal, Dawn Song, and Ilya Sutskever. 2019. GamePad: A Learning Environment for Theorem Proving. In 7th International Conference on Learning Representations, ICLR 2019.

[23]

Joe Hurd. 2003. Formal Verification of Probabilistic Algorithms. Ph.D. Dissertation. Cambridge University.

[24]

Jan Jakubuv and Josef Urban. 2019. Hammering Mizar by Learning Clause Guidance (Short Paper). In ITP. 34 : 1-34 : 8.

[25]

Cezary Kaliszyk, François Chollet, and Christian Szegedy. 2017. HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving. In 5th International Conference on Learning Representations, ICLR 2017.

[26]

Cezary Kaliszyk, Josef Urban, Henryk Michalewski, and Miroslav Olsák. 2018. Reinforcement Learning of Theorem Proving. In NeurIPS. 8836-8847.

[27]

Robert Kam. 2008. coq-markov Library. htps://github.com/coqcontribs/markov.

[28]

Michael J Kearns and Umesh Virkumar Vazirani. 1994. An Introduction to Computational Learning Theory. MIT press.

Digital Library

[29]

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of Machine Learning. MIT press.

Digital Library

[30]

Adam Petcher and Greg Morrisett. 2015. The Foundational Cryptography Framework. In POST. 53-72.

[31]

Simon Peyton Jones, Mark Jones, and Erik Meijer. 1997. Type classes: an exploration of the design space. In Haskell Workshop.

[32]

Ronald L. Rivest. 1991. Cryptography and Machine Learning. In Advances in Cryptology-ASIACRYPT '91. 427-439.

[33]

Daniel Selsam and Nikolaj Bjørner. 2019. Guiding High-Performance SAT Solvers with Unsat-Core Predictions. In Theory and Applications of Satisfiability Testing-SAT 2019. 336-353.

[34]

Daniel Selsam, Percy Liang, and David Dill. 2017. Developing Bug-Free Machine Learning Systems With Formal Mathematics. In International Conference on Machine Learning (ICML). 3047-3056.

[35]

Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.

Digital Library

[36]

Joseph Tassarotti. 2020. coq-proba Probability Library. htps://github. com/jtassaroti/coq-proba.

[37]

Joseph Tassarotti and Robert Harper. 2018. Verified Tail Bounds for Randomized Programs. In ITP. 560-578.

[38]

The mathlib Community. 2020. The Lean Mathematical Library. In CPP. 367-381.

[39]

Leslie G. Valiant. 1984. A Theory of the Learnable. Commun. ACM 27, 11 ( 1984 ), 1134-1142.

[40]

Eelis van der Weegen and James McKinna. 2008. A Machine-Checked Proof of the Average-Case Complexity of Quicksort in Coq. In TYPES. 256-271.

[41]

Vladimir Naumovich Vapnik. 2000. The Nature of Statistical Learning Theory, Second Edition. Springer.

[42]

Martin Zinkevich. 2020. htps://github.com/google/formal-ml

Cited By

Li CWang ZHe WWu YXu SWen Z(2025)Formalization of Convergence Rates of Four First-order Algorithms for Convex OptimizationJournal of Automated Reasoning10.1007/s10817-025-09741-w69:4Online publication date: 15-Oct-2025
https://doi.org/10.1007/s10817-025-09741-w
Affeldt RCohen CSaito A(2025)Semantics of Probabilistic Programs Using s-Finite Kernels in Dependent Type TheoryACM Transactions on Probabilistic Machine Learning10.1145/37322911:3(1-34)Online publication date: 29-Aug-2025
https://dl.acm.org/doi/10.1145/3732291
Affeldt RIshiguro YStone Z(2025)A Formal Foundation for Equational Reasoning on Probabilistic ProgramsProgramming Languages and Systems10.1007/978-981-95-3585-9_3(44-64)Online publication date: 27-Jun-2025
https://dl.acm.org/doi/10.1007/978-981-95-3585-9_3
Show More Cited By

Index Terms

A formal proof of PAC learnability for decision stumps
1. General and reference
  1. Cross-computing tools and techniques
    1. Verification
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Sample complexity and generalization bounds

Recommendations

A formal proof of the independence of the continuum hypothesis
CPP 2020: Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs

We describe a formal proof of the independence of the continuum hypothesis (CH) in the Lean theorem prover. We use Boolean-valued models to give forcing arguments for both directions, using Cohen forcing for the consistency of ¬ CH and a σ-closed ...
PAC learnability under non-atomic measures: A problem by Vidyasagar

In response to a 1997 problem of M. Vidyasagar, we state a criterion for PAC learnability of a concept class C under the family of all non-atomic (diffuse) measures on the domain @W. The uniform Glivenko-Cantelli property with respect to non-atomic ...
On the Fine-Structure of Regular Algebra

Regular algebra is the algebra of regular expressions as induced by regular language identity. We use Isabelle/HOL for a detailed systematic study of the regular algebra axioms given by Boffa, Conway, Kozen and Salomaa. We investigate the relationships ...

Comments

View Table of Contents