skip to main content
research-article

Beautiful differentiation

Published:31 August 2009Publication History
Skip Abstract Section

Abstract

Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The higher-dimensional case has also been tackled, though with extra complexity. This paper develops an implementation of higher-dimensional, higher-order, forward-mode AD in the extremely general and elegant setting of calculus on manifolds and derives that implementation from a simple and precise specification. In order to motivate and discover the implementation, the paper poses the question "What does AD mean, independently of implementation?" An answer arises in the form of naturality of sampling a function and its derivative. Automatic differentiation flows out of this naturality condition, together with the chain rule. Graduating from first-order to higher-order AD corresponds to sampling all derivatives instead of just one. Next, the setting is expanded to arbitrary vector spaces, in which derivative values are linear maps. The specification of AD adapts to this elegant and very general setting, which even simplifies the development.

Skip Supplemental Material Section

Supplemental Material

beautifuldierentiationonvimeo.mp4

References

  1. Manuel M. T. Chakravarty, Gabriele Keller, and Simon Peyton-Jones. Associated type synonyms. In ICFP '05: Proceedings of the tenth ACM SIGPLAN international conference on Functional programming, pages 241--253. ACM Press, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Conal Elliott. Beautiful differentiation (extended version). Technical Report 2009-02, LambdaPix, March 2009. URL http://conal.net/papers/beautiful-differentiation.Google ScholarGoogle Scholar
  3. Conal Elliott. Denotational design with type class morphisms. Technical Report 2009-01, LambdaPix, March 2009. URL http://conal.net/papers/type-class-morphisms.Google ScholarGoogle Scholar
  4. Conal Elliott. Push-pull functional reactive programming. In Proceedings of the Haskell Symposium, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Jason Foutz. Higher order multivariate automatic differentiation in haskell. Blog post, February 2008. URL http://metavar.blogspot.com/2008/02/higher-order- multivariate-automatic.html.Google ScholarGoogle Scholar
  6. Ralf Hinze. Generalizing generalized tries. Journal of Functional Programming, 10 (04): 327--351, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Graham Hutton and Jeremy Gibbons. The generic approximation lemma. Information Processing Letters, 79 (4): 197--201, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Jerzy Karczmarczuk. Functional coding of differential forms. In Scottish Workshop on Functional Programming, 1999.Google ScholarGoogle Scholar
  9. Jerzy Karczmarczuk. Functional differentiation of computer programs. Higher-Order and Symbolic Computation, 14 (1), 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Conor McBride and Ross Paterson. Applicative programming with effects. Journal of Functional Programming, 18 (1): 1--13, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. Douglas McIlroy. The music of streams. Information Processing Letters, 77 (2-4): 189--195, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Henrik Nilsson. Functional automatic differentiation with Dirac impulses. In Proceedings of the Eighth ACM SIGPLAN International Conference on Functional Programming, pages 153--164, Uppsala, Sweden, August 2003. ACM Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Barak A. Pearlmutter and Jeffrey Mark Siskind. Lazy multivariate higher-order forward-mode AD. In Proceedings of the 2007 Symposium on Principles of Programming Languages, pages 155--60, Nice, France, January 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Simon Peyton Jones, Andrew Tolmach, and Tony Hoare. Playing by the rules: rewriting as a practical optimisation technique in ghc. In In Haskell Workshop. ACM SIGPLAN, 2001.Google ScholarGoogle Scholar
  15. Jeffrey Mark Siskind and Barak A. Pearlmutter. Nesting forward-mode AD in a functional framework. Higher Order Symbolic Computation, 21 (4): 361--376, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Jeffrey Mark Siskind and Barak A. Pearlmutter. Perturbation confusion and referential transparency: Correct functional implementation of forward-mode AD. In Implementation and Application of Functional Languages, IFL'05, September 2005.Google ScholarGoogle Scholar
  17. Michael Spivak. Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus. HarperCollins Publishers, 1971.Google ScholarGoogle Scholar

Index Terms

  1. Beautiful differentiation

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 44, Issue 9
        ICFP '09
        September 2009
        343 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/1631687
        Issue’s Table of Contents
        • cover image ACM Conferences
          ICFP '09: Proceedings of the 14th ACM SIGPLAN international conference on Functional programming
          August 2009
          364 pages
          ISBN:9781605583327
          DOI:10.1145/1596550

        Copyright © 2009 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 31 August 2009

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!