Abstract
In this paper, we give a simple and efficient implementation of reverse-mode automatic differentiation, which both extends easily to higher-order functions, and has run time and memory consumption linear in the run time of the original program. In addition to a formal description of the translation, we also describe an implementation of this algorithm, and prove its correctness by means of a logical relations argument.
Supplemental Material
- Aloïs Brunel, Damiano Mazza, and Michele Pagani. 2020. Backpropagation in the simply typed lambda-calculus with linear negation. Proc. ACM Program. Lang., 4, POPL (2020), 64:1–64:27. https://doi.org/10.1145/3371132 Google Scholar
- George Corliss, Christèle Faure, Andreas Griewank, and Uwe Naumann. 2002. Automatic Differentiation of Algorithms: From Simulation to Optimization. Springer. isbn:978-1-4612-6543-6 https://doi.org/10.1007/978-1-4613-0075-5 Google Scholar
- Jeff Egger, Rasmus Ejlers Møgelberg, and Alex Simpson. 2012. The enriched effect calculus: syntax and semantics. Journal of Logic and Computation, 24, 3 (2012), 06, 615–654. issn:0955-792X https://doi.org/10.1093/logcom/exs025 arxiv:https://academic.oup.com/logcom/article-pdf/24/3/615/2785623/exs025.pdf. Google Scholar
- Conal Elliott. 2018. The simple essence of automatic differentiation. Proc. ACM Program. Lang., 2, ICFP (2018), 70:1–70:29. https://doi.org/10.1145/3236765 Google Scholar
- Conal M. Elliott. 2009. Beautiful Differentiation. In Proceedings of the 14th ACM SIGPLAN International Conference on Functional Programming (ICFP ’09). Association for Computing Machinery, New York, NY, USA. 191–202. isbn:9781605583327 https://doi.org/10.1145/1596550.1596579 Google Scholar
- Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics, USA. isbn:0898716594 https://doi.org/10.1137/1.9780898717761 Google Scholar
- Mathieu Huot, Sam Staton, and Matthijs Vákár. 2020. Correctness of Automatic Differentiation via Diffeologies and Categorical Gluing. In Foundations of Software Science and Computation Structures - 23rd International Conference, FOSSACS 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25-30, 2020, Proceedings, Jean Goubault-Larrecq and Barbara König (Eds.) (Lecture Notes in Computer Science, Vol. 12077). Springer, 319–338. https://doi.org/10.1007/978-3-030-45231-5_17 Google Scholar
- Achim Jung and Jerzy Tiuryn. 1993. A new characterization of lambda definability. In Typed Lambda Calculi and Applications, Marc Bezem and Jan Friso Groote (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 245–257. isbn:978-3-540-47586-6 https://doi.org/10.1007/BFb0037110 Google Scholar
- Jerzy Karczmarczuk. 1998. Functional Differentiation of Computer Programs. In Proceedings of the Third ACM SIGPLAN International Conference on Functional Programming (ICFP ’98). Association for Computing Machinery, New York, NY, USA. 195–203. isbn:1581130244 https://doi.org/10.1145/289423.289442 Google Scholar
- Edward Kmett, Barak Pearlmutter, and Jeffrey Mark Siskind. 2010–2021. ad: Automatic Differentiation. Haskell package at. https://hackage.haskell.org/package/ad Accessed: 2021-11-10.Google Scholar
- Carol Mak and Luke Ong. 2020. A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation. CoRR, abs/2002.08241 (2020), arxiv:2002.08241. arxiv:2002.08241Google Scholar
- Damiano Mazza and Michele Pagani. 2021. Automatic differentiation in PCF. Proc. ACM Program. Lang., 5, POPL (2021), 1–27. https://doi.org/10.1145/3434309 Google Scholar
- Eugenio Moggi. 1991. Notions of Computation and Monads. Inf. Comput., 93, 1 (1991), 55–92. https://doi.org/10.1016/0890-5401(91)90052-4 Google Scholar
- Barak A. Pearlmutter and Jeffrey Mark Siskind. 2008. Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Trans. Program. Lang. Syst., 30, 2 (2008), 7:1–7:36. https://doi.org/10.1145/1330017.1330018 Google Scholar
- Dan Piponi. 2004. Automatic Differentiation, C++ Templates, and Photogrammetry. J. Graphics, GPU, & Game Tools, 9, 4 (2004), 41–55. https://doi.org/10.1080/10867651.2004.10504901 Google Scholar
- Gordon Plotkin and Martin Abadi. 2020. A simple differentiable programming language. Proc. ACM Program. Lang., https://doi.org/10.1145/3371106 Google Scholar
- John C. Reynolds. 1998. Definitional Interpreters for Higher-Order Programming Languages. High. Order Symb. Comput., 11, 4 (1998), 363–397. https://doi.org/10.1023/A:1010027404223 Google Scholar
- Benjamin Sherman, Jesse Michel, and Michael Carbin. 2021. 𝜆ₛ: computable semantics for differentiable programming with higher-order functions and datatypes. Proc. ACM Program. Lang., 5, POPL (2021), 1–31. https://doi.org/10.1145/3434284 Google Scholar
- Matthijs Vákár. 2021. Reverse AD at Higher Types: Pure, Principled and Denotationally Correct. In Programming Languages and Systems - 30th European Symposium on Programming, ESOP 2021, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021, Luxembourg City, Luxembourg, March 27 - April 1, 2021, Proceedings, Nobuko Yoshida (Ed.) (Lecture Notes in Computer Science, Vol. 12648). Springer, 607–634. https://doi.org/10.1007/978-3-030-72019-3_22 Google Scholar
- Fei Wang, Daniel Zheng, James M. Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2019. Demystifying differentiable programming: shift/reset the penultimate backpropagator. Proc. ACM Program. Lang., 3, ICFP (2019), 96:1–96:31. https://doi.org/10.1145/3341700 Google Scholar
- RE Wengert. 1964. A simple automatic derivative evaluation program. Communication of the ACM, 7 (1964), Aug., 463–464. https://doi.org/10.1145/355586.364791 Google Scholar
Index Terms
Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation
Recommendations
Local higher-order fixpoint iteration
AbstractLocal fixpoint iteration describes a technique that restricts fixpoint iteration in function spaces to needed arguments only. It has been studied well for first-order functions in abstract interpretation and also in model checking. ...
Intensional computation with higher-order functions
AbstractIntensional computations are those that query the internal structure of their arguments. In a higher-order setting, such queries perform program analysis. This is beyond the expressive power of traditional term rewriting systems, such ...
Mining the use of higher-order functions:: An exploratory study on Scala programs
AbstractA higher-order function takes one or more functions as inputs or outputs to support the generality of function definitions. In modern programming languages, higher-order functions are designed as a feature to enhance usability and scalability. ...






Comments