skip to main content
research-article
Free Access

Automatic differentiation in PCF

Published:04 January 2021Publication History
Skip Abstract Section

Abstract

We study the correctness of automatic differentiation (AD) in the context of a higher-order, Turing-complete language (PCF with real numbers), both in forward and reverse mode. Our main result is that, under mild hypotheses on the primitive functions included in the language, AD is almost everywhere correct, that is, it computes the derivative or gradient of the program under consideration except for a set of Lebesgue measure zero. Stated otherwise, there are inputs on which AD is incorrect, but the probability of randomly choosing one such input is zero. Our result is in fact more precise, in that the set of failure points admits a more explicit description: for example, in case the primitive functions are just constants, addition and multiplication, the set of points where AD fails is contained in a countable union of zero sets of polynomials.

References

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jefrey Dean, Matthieu Devin, Sanjay Ghemawat, Geofrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of OSDI. USENIX Association, 265-283.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Martín Abadi and Gordon D. Plotkin. 2020. A simple diferentiable programming language. Proc. ACM Program. Lang. 4, POPL ( 2020 ), 38 : 1-38 : 28.Google ScholarGoogle Scholar
  3. Roberto Amadio and Pierre-Louis Curien. 1998. Domains and Lambda-Calculi. Vol. 46. Cambridge University Press.Google ScholarGoogle Scholar
  4. Henk P. Barendregt. 1985. The Lambda Calculus, Its Syntax and Semantics. North Holland.Google ScholarGoogle Scholar
  5. Gilles Barthe, Raphaëlle Crubillé, Ugo Dal Lago, and Francesco Gavazzo. 2020. On the Versatility of Open Logical Relations-Continuity, Automatic Diferentiation, and a Containment Theorem. In Proceedings of ESOP. 56-83.Google ScholarGoogle Scholar
  6. Atilim Baydin, Barak Pearlmutter, Alexey Radul, and Jefrey Siskind. 2018. Automatic diferentiation in machine learning: A survey. Journal of Machine Learning Research 18 ( 2018 ), 1-43.Google ScholarGoogle Scholar
  7. Thomas Beck and Herbert Fischer. 1994. The if-problem in automatic diferentiation. J. Comput. Appl. Math. 50, 1 ( 1994 ), 119-131.Google ScholarGoogle Scholar
  8. Aloïs Brunel, Damiano Mazza, and Michele Pagani. 2020. Backpropagation in the simply typed lambda-calculus with linear negation. PACMPL 4, POPL ( 2020 ), 64 : 1-64 : 27.Google ScholarGoogle Scholar
  9. Pietro Di Gianantonio and Abbas Edalat. 2013. A Language for Diferentiable Functions. In Proceedings of FOSSACS. 337-352.Google ScholarGoogle Scholar
  10. Thomas Ehrhard and Laurent Regnier. 2006. Böhm Trees, Krivine's Machine and the Taylor Expansion of Lambda-Terms. In Proceedings of CiE. 186-197.Google ScholarGoogle Scholar
  11. Thomas Ehrhard and Laurent Regnier. 2008. Uniformity and the Taylor expansion of ordinary lambda-terms. Theor. Comput. Sci. 403, 2-3 ( 2008 ), 347-372.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Conal Elliott. 2018. The simple essence of automatic diferentiation. PACMPL 2, ICFP ( 2018 ), 70 : 1-70 : 29.Google ScholarGoogle Scholar
  13. Martín Hötzel Escardó. 1996. PCF Extended with Real Numbers. Theor. Comput. Sci. 162, 1 ( 1996 ), 79-115.Google ScholarGoogle Scholar
  14. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook. org Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives-principles and techniques of algorithmic diferentiation, Second Edition. SIAM.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Mathieu Huot, Sam Staton, and Matthijs Vákár. 2020. Correctness of Automatic Diferentiation via Difeologies and Categorical Gluing. In Proceedings of FOSSACS. 319-338.Google ScholarGoogle Scholar
  16. Johan Joss. 1976. Algorthmisches Diferenzieren. Ph.D. Dissertation. ETH Zurich.Google ScholarGoogle Scholar
  17. Yann LeCun. 2018. Deep Learning est mort. Vive Diferentiable Programming! ( 2018 ). https://www.facebook.com/yann. lecun/posts/10155003011462143Google ScholarGoogle Scholar
  18. Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. 2020. On Correctness of Automatic Diferentiation for Non-Diferentiable Functions. CoRR abs/ 2006.06903 ( 2020 ).Google ScholarGoogle Scholar
  19. Carol Mak, C.-H. Luke Ong, Hugo Paquet, and Dominik Wagner. 2020. Densities of almost-surely terminating probabilistic programs are diferentiable almost everywhere. CoRR abs/ 2004.03924 ( 2020 ).Google ScholarGoogle Scholar
  20. Damiano Mazza. 2017. Polyadic Approximations in Logic and Computation. Habilitation thesis. Université Paris 13.Google ScholarGoogle Scholar
  21. Damiano Mazza, Luc Pellissier, and Pierre Vial. 2018. Polyadic approximations, fibrations and intersection types. Proc. ACM Program. Lang. 2, POPL ( 2018 ), 6 : 1-6 : 28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Boris Mityagin. 2015. The Zero Set of a Real Analytic Function. arXiv: 1512.07276 [math.CA] ( 2015 ).Google ScholarGoogle Scholar
  23. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic diferentiation in PyTorch. ( 2017 ).Google ScholarGoogle Scholar
  24. Barak A. Pearlmutter and Jefrey Mark Siskind. 2008. Reverse-mode AD in a Functional Framework: Lambda the Ultimate Backpropagator. ACM Trans. Program. Lang. Syst. 30, 2 ( 2008 ), 7 : 1-7 : 36.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Gordon Plotkin. 1977. LCF Considered as a Programming Language. Theoretical Computer Science 5, 3 ( 1977 ), 223-255.Google ScholarGoogle Scholar
  26. Amir Shaikhha, Andrew Fitzgibbon, Dimitrios Vytiniotis, and Simon Peyton Jones. 2019. Eficient diferentiable programming in a functional array-processing language. PACMPL 3, ICFP ( 2019 ), 97 : 1-97 : 30.Google ScholarGoogle Scholar
  27. Bert Speelpenning. 1980. Compiling Fast Partial Derivatives of Functions Given by Algorithms. Ph.D. Dissertation. University of Illinois at Urbana-Champaign.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ágnes Szendrei. 1986. Clones in Universal Algebra. Presses de l'Université de Montréal.Google ScholarGoogle Scholar
  29. Fei Wang, Daniel Zheng, James M. Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2019. Demystifying diferentiable programming: shift/reset the penultimate backpropagator. PACMPL 3, ICFP ( 2019 ), 96 : 1-96 : 31.Google ScholarGoogle Scholar
  30. Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, and Frank Wood. 2019. LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Diferentiable Models. In Proceedings of AISTATS. 148-157.Google ScholarGoogle Scholar

Index Terms

  1. Automatic differentiation in PCF

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image Proceedings of the ACM on Programming Languages
        Proceedings of the ACM on Programming Languages  Volume 5, Issue POPL
        January 2021
        1789 pages
        EISSN:2475-1421
        DOI:10.1145/3445980
        Issue’s Table of Contents

        Copyright © 2021 Owner/Author

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 4 January 2021
        Published in pacmpl Volume 5, Issue POPL

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!