Abstract
Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The higher-dimensional case has also been tackled, though with extra complexity. This paper develops an implementation of higher-dimensional, higher-order, forward-mode AD in the extremely general and elegant setting of calculus on manifolds and derives that implementation from a simple and precise specification. In order to motivate and discover the implementation, the paper poses the question "What does AD mean, independently of implementation?" An answer arises in the form of naturality of sampling a function and its derivative. Automatic differentiation flows out of this naturality condition, together with the chain rule. Graduating from first-order to higher-order AD corresponds to sampling all derivatives instead of just one. Next, the setting is expanded to arbitrary vector spaces, in which derivative values are linear maps. The specification of AD adapts to this elegant and very general setting, which even simplifies the development.
Supplemental Material
- Manuel M. T. Chakravarty, Gabriele Keller, and Simon Peyton-Jones. Associated type synonyms. In ICFP '05: Proceedings of the tenth ACM SIGPLAN international conference on Functional programming, pages 241--253. ACM Press, 2005. Google Scholar
Digital Library
- Conal Elliott. Beautiful differentiation (extended version). Technical Report 2009-02, LambdaPix, March 2009. URL http://conal.net/papers/beautiful-differentiation.Google Scholar
- Conal Elliott. Denotational design with type class morphisms. Technical Report 2009-01, LambdaPix, March 2009. URL http://conal.net/papers/type-class-morphisms.Google Scholar
- Conal Elliott. Push-pull functional reactive programming. In Proceedings of the Haskell Symposium, 2009. Google Scholar
Digital Library
- Jason Foutz. Higher order multivariate automatic differentiation in haskell. Blog post, February 2008. URL http://metavar.blogspot.com/2008/02/higher-order- multivariate-automatic.html.Google Scholar
- Ralf Hinze. Generalizing generalized tries. Journal of Functional Programming, 10 (04): 327--351, 2000. Google Scholar
Digital Library
- Graham Hutton and Jeremy Gibbons. The generic approximation lemma. Information Processing Letters, 79 (4): 197--201, 2001. Google Scholar
Digital Library
- Jerzy Karczmarczuk. Functional coding of differential forms. In Scottish Workshop on Functional Programming, 1999.Google Scholar
- Jerzy Karczmarczuk. Functional differentiation of computer programs. Higher-Order and Symbolic Computation, 14 (1), 2001. Google Scholar
Digital Library
- Conor McBride and Ross Paterson. Applicative programming with effects. Journal of Functional Programming, 18 (1): 1--13, 2008. Google Scholar
Digital Library
- M. Douglas McIlroy. The music of streams. Information Processing Letters, 77 (2-4): 189--195, 2001. Google Scholar
Digital Library
- Henrik Nilsson. Functional automatic differentiation with Dirac impulses. In Proceedings of the Eighth ACM SIGPLAN International Conference on Functional Programming, pages 153--164, Uppsala, Sweden, August 2003. ACM Press. Google Scholar
Digital Library
- Barak A. Pearlmutter and Jeffrey Mark Siskind. Lazy multivariate higher-order forward-mode AD. In Proceedings of the 2007 Symposium on Principles of Programming Languages, pages 155--60, Nice, France, January 2007. Google Scholar
Digital Library
- Simon Peyton Jones, Andrew Tolmach, and Tony Hoare. Playing by the rules: rewriting as a practical optimisation technique in ghc. In In Haskell Workshop. ACM SIGPLAN, 2001.Google Scholar
- Jeffrey Mark Siskind and Barak A. Pearlmutter. Nesting forward-mode AD in a functional framework. Higher Order Symbolic Computation, 21 (4): 361--376, 2008. Google Scholar
Digital Library
- Jeffrey Mark Siskind and Barak A. Pearlmutter. Perturbation confusion and referential transparency: Correct functional implementation of forward-mode AD. In Implementation and Application of Functional Languages, IFL'05, September 2005.Google Scholar
- Michael Spivak. Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus. HarperCollins Publishers, 1971.Google Scholar
Index Terms
Beautiful differentiation
Recommendations
Beautiful differentiation
ICFP '09: Proceedings of the 14th ACM SIGPLAN international conference on Functional programmingAutomatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The ...
Automatic differentiation of explicit Runge-Kutta methods for optimal control
This paper considers the numerical solution of optimal control problems based on ODEs. We assume that an explicit Runge-Kutta method is applied to integrate the state equation in the context of a recursive discretization approach. To compute the ...
Symplectic Runge--Kutta Schemes for Adjoint Equations, Automatic Differentiation, Optimal Control, and More
The study of the sensitivity of the solution of a system of differential equations with respect to changes in the initial conditions leads to the introduction of an adjoint system, whose discretization is related to reverse accumulation in automatic ...







Comments