skip to main content

A general construction for abstract interpretation of higher-order automatic differentiation

Published:31 October 2022Publication History
Skip Abstract Section

Abstract

We present a novel, general construction to abstractly interpret higher-order automatic differentiation (AD). Our construction allows one to instantiate an abstract interpreter for computing derivatives up to a chosen order. Furthermore, since our construction reduces the problem of abstractly reasoning about derivatives to abstractly reasoning about real-valued straight-line programs, it can be instantiated with almost any numerical abstract domain, both relational and non-relational. We formally establish the soundness of this construction.

We implement our technique by instantiating our construction with both the non-relational interval domain and the relational zonotope domain to compute both first and higher-order derivatives. In the latter case, we are the first to apply a relational domain to automatic differentiation for abstracting higher-order derivatives, and hence we are also the first abstract interpretation work to track correlations across not only different variables, but different orders of derivatives.

We evaluate these instantiations on multiple case studies, namely robustly explaining a neural network and more precisely computing a neural network’s Lipschitz constant. For robust interpretation, first and second derivatives computed via zonotope AD are up to 4.76× and 6.98× more precise, respectively, compared to interval AD. For Lipschitz certification, we obtain bounds that are up to 11,850× more precise with zonotopes, compared to the state-of-the-art interval-based tool.

References

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX symposium on operating systems design and implementation. Google ScholarGoogle Scholar
  2. Aws Albarghouthi. 2021. Introduction to Neural Network Verification. Foundations and Trends® in Programming Languages. Google ScholarGoogle Scholar
  3. David Alvarez-Melis and Tommi S Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In 6th International Conference on Learning Representations (ICLR). Google ScholarGoogle Scholar
  5. Claus Bendtsen and Ole Stauning. 1996. FADBAD, a flexible C++ package for automatic differentiation. Google ScholarGoogle Scholar
  6. Jesse Bettencourt, Matthew J Johnson, and David Duvenaud. 2019. Taylor-mode automatic differentiation for higher-order derivatives in JAX. Google ScholarGoogle Scholar
  7. Patrick Cousot and Radhia Cousot. 1977. Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Patrick Cousot, Roberto Giacobazzi, and Francesco Ranzato. 2019. A^2I: abstract^2 interpretation. Proceedings of the ACM on Programming Languages. Google ScholarGoogle Scholar
  9. Jens Deussen. 2021. Global Derivatives. Ph. D. Dissertation. Google ScholarGoogle Scholar
  10. Fr. Faa Di Bruno. 1857. Note sur une nouvelle formule de calcul différentiel. Quarterly J. Pure Appl. Math. Google ScholarGoogle Scholar
  11. Pietro Di Gianantonio and Abbas Edalat. 2013. A language for differentiable functions. In International Conference on Foundations of Software Science and Computational Structures. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Thomas Fel, Mélanie Ducoffe, David Vigouroux, Rémi Cadène, Mikael Capelle, Claire Nicodème, and Thomas Serre. 2022. Don’t Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. arXiv preprint arXiv:2202.07728. Google ScholarGoogle Scholar
  13. Jeffrey Fike and Juan Alonso. 2011. The Development of Hyper-Dual Numbers for Exact Second-Derivative Calculations. AIAA. Google ScholarGoogle Scholar
  14. Cormac Flanagan, Amr Sabry, Bruce F Duba, and Matthias Felleisen. 1993. The essence of compiling with continuations. In Proceedings of the ACM SIGPLAN 1993 conference on Programming language design and implementation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Oleg Fryazinov, Alexander Pasko, and Peter Comninos. 2010. Technical Section: Fast Reliable Interrogation of Procedurally Defined Implicit Surfaces Using Extended Revised Affine Arithmetic. Comput. Graph., 34, 6 (2010). Google ScholarGoogle Scholar
  16. Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The zonotope abstract domain taylor1+. In International Conference on Computer Aided Verification. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Andreas Griewank, Jean Utke, and Andrea Walther. 2000. Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Mathematics of computation. Google ScholarGoogle Scholar
  18. Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM. Google ScholarGoogle Scholar
  19. Horace He. 2019. The State of Machine Learning Frameworks in 2019. https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/ The Gradient. Google ScholarGoogle Scholar
  20. Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations. Google ScholarGoogle Scholar
  21. Jan Hückelheim, Ziqing Luo, Sri Hari Krishna Narayanan, Stephen Siegel, and Paul D Hovland. 2018. Verifying Properties of Differentiable Programs. In International Static Analysis Symposium. Google ScholarGoogle Scholar
  22. Mathieu Huot, Sam Staton, and Matthijs Vákár. 2021. Higher Order Automatic Differentiation of Higher Order Functions. arXiv preprint arXiv:2101.06757. Google ScholarGoogle Scholar
  23. Fabian Immler. 2018. A Verified ODE Solver and Smale’s 14th Problem. Ph. D. Dissertation. Technische Universität München. Google ScholarGoogle Scholar
  24. Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2021. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22 (2021). Google ScholarGoogle Scholar
  25. Bertrand Jeannet and Antoine Miné. 2009. Apron: A library of numerical abstract domains for static analysis. In International Conference on Computer Aided Verification. 661–667. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Matt Jordan and Alex Dimakis. 2021. Provable Lipschitz certification for generative models. In International Conference on Machine Learning. 5118–5126. Google ScholarGoogle Scholar
  27. Jerzy Karczmarczuk. 2001. Functional differentiation of computer programs. Higher-order and symbolic computation. Google ScholarGoogle Scholar
  28. Faustyna Krawiec, Neel Krishnaswami, Simon Peyton Jones, Tom Ellis, Andrew Fitzgibbon, and R Eisenberg. 2022. Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation. Proceedings of the ACM on Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Jacob Laurel, Rem Yang, Gagandeep Singh, and Sasa Misailovic. 2022. A Dual Number Abstraction for Static Analysis of Clarke Jacobians. Proceedings of the ACM on Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Appendix to A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://jsl1994.github.io/papers/OOPSLA2022_appendix.pdf Google ScholarGoogle Scholar
  31. Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Artifact for A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://doi.org/10.1145/3554329 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yann LeCun. 1998. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/. Google ScholarGoogle Scholar
  33. Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics (TOG), 37, 4 (2018). Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Ravi Mangal, Kartik Sarangmath, Aditya V Nori, and Alessandro Orso. 2020. Probabilistic Lipschitz Analysis of Neural Networks. In International Static Analysis Symposium. Google ScholarGoogle Scholar
  35. Antoine Miné. 2006. Symbolic methods to enhance the precision of numerical abstract domains. In International Workshop on Verification, Model Checking, and Abstract Interpretation. Google ScholarGoogle Scholar
  36. Adam Paszke, Daniel D Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew J Johnson, Jonathan Ragan-Kelley, and Dougal Maclaurin. 2021. Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming. Proceedings of the ACM on Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Colin Paterson, Haoze Wu, John Grese, Radu Calinescu, Corina S Păsăreanu, and Clark Barrett. 2021. Deepcert: Verification of contextually relevant robustness for neural network image classifiers. In International Conference on Computer Safety, Reliability, and Security. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Barak A Pearlmutter and Jeffrey Mark Siskind. 2007. Lazy Multivariate Higher-Order Forward-Mode AD. In Symposium on Principles of Programming Languages. Google ScholarGoogle Scholar
  39. Thomas Reps and Aditya Thakur. 2016. Automating abstract interpretation. In International Conference on Verification, Model Checking, and Abstract Interpretation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. 2021. Scalable polyhedral verification of recurrent neural networks. In International Conference on Computer Aided Verification. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Benjamin Sherman, Jesse Michel, and Michael Carbin. 2021. λ _S: Computable Semantics for Differentiable Programming with Higher-Order Functions and Datatypes. Proceedings of the ACM on Programming Languages. Google ScholarGoogle Scholar
  42. Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. 2019. Robustness Verification for Transformers. In International Conference on Learning Representations. Google ScholarGoogle Scholar
  43. Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin T Vechev. 2018. Fast and Effective Robustness Certification.. NeurIPS. Google ScholarGoogle Scholar
  44. Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017. Fast polyhedra abstract domain. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. A practical construction for decomposing numerical abstract domains. Proceedings of the ACM on Programming Languages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Matthew Sotoudeh and Aditya V Thakur. 2020. Abstract Neural Networks. In International Static Analysis Symposium. Google ScholarGoogle Scholar
  48. Ole Stauning. 1997. Automatic validation of numerical solutions. Google ScholarGoogle Scholar
  49. Jorge Stolfi and Luiz Henrique de Figueiredo. 2003. An introduction to affine arithmetic. Trends in Computational and Applied Mathematics, 4 (2003). Google ScholarGoogle Scholar
  50. Vassilis Vassiliadis, Jan Riehme, Jens Deussen, Konstantinos Parasyris, Christos D Antonopoulos, Nikolaos Bellas, Spyros Lalis, and Uwe Naumann. 2016. Towards automatic significance analysis for approximate computing. In 2016 IEEE/ACM International Symposium on Code Generation and Optimization. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Andrea Walther and Andreas Griewank. 2012. Getting Started with ADOL-C. Combinatorial Scientific Computing. Google ScholarGoogle Scholar
  52. Rem Yang, Jacob Laurel, Sasa Misailovic, and Gagandeep Singh. 2022. Provable Defense Against Geometric Transformations. arXiv preprint arXiv:2207.11177. Google ScholarGoogle Scholar

Index Terms

  1. A general construction for abstract interpretation of higher-order automatic differentiation

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!