skip to main content
research-article
Open Access

Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation

Published:12 January 2022Publication History
Skip Abstract Section

Abstract

In this paper, we give a simple and efficient implementation of reverse-mode automatic differentiation, which both extends easily to higher-order functions, and has run time and memory consumption linear in the run time of the original program. In addition to a formal description of the translation, we also describe an implementation of this algorithm, and prove its correctness by means of a logical relations argument.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

This is a short presentation about our paper "Provably Correct, Asymptotically Efficient, Higher-Order Reverse-Mode Automatic Differentiation" for POPL 2022 attendees. In this paper, we give a simple and efficient implementation of reverse-mode automatic differentiation, which both extends easily to higher-order functions, and has run time and memory consumption linear in the run time of the original program. In addition to a formal description of the translation, we also describe an implementation of this algorithm, and prove its correctness by means of a logical relations argument.

References

  1. Aloïs Brunel, Damiano Mazza, and Michele Pagani. 2020. Backpropagation in the simply typed lambda-calculus with linear negation. Proc. ACM Program. Lang., 4, POPL (2020), 64:1–64:27. https://doi.org/10.1145/3371132 Google ScholarGoogle Scholar
  2. George Corliss, Christèle Faure, Andreas Griewank, and Uwe Naumann. 2002. Automatic Differentiation of Algorithms: From Simulation to Optimization. Springer. isbn:978-1-4612-6543-6 https://doi.org/10.1007/978-1-4613-0075-5 Google ScholarGoogle Scholar
  3. Jeff Egger, Rasmus Ejlers Møgelberg, and Alex Simpson. 2012. The enriched effect calculus: syntax and semantics. Journal of Logic and Computation, 24, 3 (2012), 06, 615–654. issn:0955-792X https://doi.org/10.1093/logcom/exs025 arxiv:https://academic.oup.com/logcom/article-pdf/24/3/615/2785623/exs025.pdf. Google ScholarGoogle Scholar
  4. Conal Elliott. 2018. The simple essence of automatic differentiation. Proc. ACM Program. Lang., 2, ICFP (2018), 70:1–70:29. https://doi.org/10.1145/3236765 Google ScholarGoogle Scholar
  5. Conal M. Elliott. 2009. Beautiful Differentiation. In Proceedings of the 14th ACM SIGPLAN International Conference on Functional Programming (ICFP ’09). Association for Computing Machinery, New York, NY, USA. 191–202. isbn:9781605583327 https://doi.org/10.1145/1596550.1596579 Google ScholarGoogle Scholar
  6. Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics, USA. isbn:0898716594 https://doi.org/10.1137/1.9780898717761 Google ScholarGoogle Scholar
  7. Mathieu Huot, Sam Staton, and Matthijs Vákár. 2020. Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing. In Foundations of Software Science and Computation Structures - 23rd International Conference, FOSSACS 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings, Jean Goubault-Larrecq and Barbara König (Eds.) (Lecture Notes in Computer Science, Vol. 12077). Springer, 319–338. https://doi.org/10.1007/978-3-030-45231-5_17 Google ScholarGoogle Scholar
  8. Achim Jung and Jerzy Tiuryn. 1993. A new characterization of lambda definability. In Typed Lambda Calculi and Applications, Marc Bezem and Jan Friso Groote (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 245–257. isbn:978-3-540-47586-6 https://doi.org/10.1007/BFb0037110 Google ScholarGoogle Scholar
  9. Jerzy Karczmarczuk. 1998. Functional Differentiation of Computer Programs. In Proceedings of the Third ACM SIGPLAN International Conference on Functional Programming (ICFP ’98). Association for Computing Machinery, New York, NY, USA. 195–203. isbn:1581130244 https://doi.org/10.1145/289423.289442 Google ScholarGoogle Scholar
  10. Edward Kmett, Barak Pearlmutter, and Jeffrey Mark Siskind. 2010–2021. ad: Automatic Differentiation. Haskell package at. https://hackage.haskell.org/package/ad Accessed: 2021-11-10.Google ScholarGoogle Scholar
  11. Carol Mak and Luke Ong. 2020. A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation. CoRR, abs/2002.08241 (2020), arxiv:2002.08241. arxiv:2002.08241Google ScholarGoogle Scholar
  12. Damiano Mazza and Michele Pagani. 2021. Automatic differentiation in PCF. Proc. ACM Program. Lang., 5, POPL (2021), 1–27. https://doi.org/10.1145/3434309 Google ScholarGoogle Scholar
  13. Eugenio Moggi. 1991. Notions of Computation and Monads. Inf. Comput., 93, 1 (1991), 55–92. https://doi.org/10.1016/0890-5401(91)90052-4 Google ScholarGoogle Scholar
  14. Barak A. Pearlmutter and Jeffrey Mark Siskind. 2008. Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Trans. Program. Lang. Syst., 30, 2 (2008), 7:1–7:36. https://doi.org/10.1145/1330017.1330018 Google ScholarGoogle Scholar
  15. Dan Piponi. 2004. Automatic Differentiation, C++ Templates, and Photogrammetry. J. Graphics, GPU, & Game Tools, 9, 4 (2004), 41–55. https://doi.org/10.1080/10867651.2004.10504901 Google ScholarGoogle Scholar
  16. Gordon Plotkin and Martin Abadi. 2020. A simple differentiable programming language. Proc. ACM Program. Lang., https://doi.org/10.1145/3371106 Google ScholarGoogle Scholar
  17. John C. Reynolds. 1998. Definitional Interpreters for Higher-Order Programming Languages. High. Order Symb. Comput., 11, 4 (1998), 363–397. https://doi.org/10.1023/A:1010027404223 Google ScholarGoogle Scholar
  18. Benjamin Sherman, Jesse Michel, and Michael Carbin. 2021. 𝜆ₛ: computable semantics for differentiable programming with higher-order functions and datatypes. Proc. ACM Program. Lang., 5, POPL (2021), 1–31. https://doi.org/10.1145/3434284 Google ScholarGoogle Scholar
  19. Matthijs Vákár. 2021. Reverse AD at Higher Types: Pure, Principled and Denotationally Correct. In Programming Languages and Systems - 30th European Symposium on Programming, ESOP 2021, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021, Luxembourg City, Luxembourg, March 27 - April 1, 2021, Proceedings, Nobuko Yoshida (Ed.) (Lecture Notes in Computer Science, Vol. 12648). Springer, 607–634. https://doi.org/10.1007/978-3-030-72019-3_22 Google ScholarGoogle Scholar
  20. Fei Wang, Daniel Zheng, James M. Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2019. Demystifying differentiable programming: shift/reset the penultimate backpropagator. Proc. ACM Program. Lang., 3, ICFP (2019), 96:1–96:31. https://doi.org/10.1145/3341700 Google ScholarGoogle Scholar
  21. RE Wengert. 1964. A simple automatic derivative evaluation program. Communication of the ACM, 7 (1964), Aug., 463–464. https://doi.org/10.1145/355586.364791 Google ScholarGoogle Scholar

Index Terms

  1. Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image Proceedings of the ACM on Programming Languages
              Proceedings of the ACM on Programming Languages  Volume 6, Issue POPL
              January 2022
              1886 pages
              EISSN:2475-1421
              DOI:10.1145/3511309
              Issue’s Table of Contents

              Copyright © 2022 Owner/Author

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 12 January 2022
              Published in pacmpl Volume 6, Issue POPL

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!