Abstract
We present a novel, general construction to abstractly interpret higher-order automatic differentiation (AD). Our construction allows one to instantiate an abstract interpreter for computing derivatives up to a chosen order. Furthermore, since our construction reduces the problem of abstractly reasoning about derivatives to abstractly reasoning about real-valued straight-line programs, it can be instantiated with almost any numerical abstract domain, both relational and non-relational. We formally establish the soundness of this construction.
We implement our technique by instantiating our construction with both the non-relational interval domain and the relational zonotope domain to compute both first and higher-order derivatives. In the latter case, we are the first to apply a relational domain to automatic differentiation for abstracting higher-order derivatives, and hence we are also the first abstract interpretation work to track correlations across not only different variables, but different orders of derivatives.
We evaluate these instantiations on multiple case studies, namely robustly explaining a neural network and more precisely computing a neural network’s Lipschitz constant. For robust interpretation, first and second derivatives computed via zonotope AD are up to 4.76× and 6.98× more precise, respectively, compared to interval AD. For Lipschitz certification, we obtain bounds that are up to 11,850× more precise with zonotopes, compared to the state-of-the-art interval-based tool.
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX symposium on operating systems design and implementation.
Google Scholar
- Aws Albarghouthi. 2021. Introduction to Neural Network Verification. Foundations and Trends® in Programming Languages.
Google Scholar
- David Alvarez-Melis and Tommi S Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.
Google Scholar
Digital Library
- Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In 6th International Conference on Learning Representations (ICLR).
Google Scholar
- Claus Bendtsen and Ole Stauning. 1996. FADBAD, a flexible C++ package for automatic differentiation.
Google Scholar
- Jesse Bettencourt, Matthew J Johnson, and David Duvenaud. 2019. Taylor-mode automatic differentiation for higher-order derivatives in JAX.
Google Scholar
- Patrick Cousot and Radhia Cousot. 1977. Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages.
Google Scholar
Digital Library
- Patrick Cousot, Roberto Giacobazzi, and Francesco Ranzato. 2019. A^2I: abstract^2 interpretation. Proceedings of the ACM on Programming Languages.
Google Scholar
- Jens Deussen. 2021. Global Derivatives. Ph. D. Dissertation.
Google Scholar
- Fr. Faa Di Bruno. 1857. Note sur une nouvelle formule de calcul différentiel. Quarterly J. Pure Appl. Math.
Google Scholar
- Pietro Di Gianantonio and Abbas Edalat. 2013. A language for differentiable functions. In International Conference on Foundations of Software Science and Computational Structures.
Google Scholar
Digital Library
- Thomas Fel, Mélanie Ducoffe, David Vigouroux, Rémi Cadène, Mikael Capelle, Claire Nicodème, and Thomas Serre. 2022. Don’t Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. arXiv preprint arXiv:2202.07728.
Google Scholar
- Jeffrey Fike and Juan Alonso. 2011. The Development of Hyper-Dual Numbers for Exact Second-Derivative Calculations. AIAA.
Google Scholar
- Cormac Flanagan, Amr Sabry, Bruce F Duba, and Matthias Felleisen. 1993. The essence of compiling with continuations. In Proceedings of the ACM SIGPLAN 1993 conference on Programming language design and implementation.
Google Scholar
Digital Library
- Oleg Fryazinov, Alexander Pasko, and Peter Comninos. 2010. Technical Section: Fast Reliable Interrogation of Procedurally Defined Implicit Surfaces Using Extended Revised Affine Arithmetic. Comput. Graph., 34, 6 (2010).
Google Scholar
- Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The zonotope abstract domain taylor1+. In International Conference on Computer Aided Verification.
Google Scholar
Digital Library
- Andreas Griewank, Jean Utke, and Andrea Walther. 2000. Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Mathematics of computation.
Google Scholar
- Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM.
Google Scholar
- Horace He. 2019. The State of Machine Learning Frameworks in 2019. https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/ The Gradient.
Google Scholar
- Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations.
Google Scholar
- Jan Hückelheim, Ziqing Luo, Sri Hari Krishna Narayanan, Stephen Siegel, and Paul D Hovland. 2018. Verifying Properties of Differentiable Programs. In International Static Analysis Symposium.
Google Scholar
- Mathieu Huot, Sam Staton, and Matthijs Vákár. 2021. Higher Order Automatic Differentiation of Higher Order Functions. arXiv preprint arXiv:2101.06757.
Google Scholar
- Fabian Immler. 2018. A Verified ODE Solver and Smale’s 14th Problem. Ph. D. Dissertation. Technische Universität München.
Google Scholar
- Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2021. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22 (2021).
Google Scholar
- Bertrand Jeannet and Antoine Miné. 2009. Apron: A library of numerical abstract domains for static analysis. In International Conference on Computer Aided Verification. 661–667.
Google Scholar
Digital Library
- Matt Jordan and Alex Dimakis. 2021. Provable Lipschitz certification for generative models. In International Conference on Machine Learning. 5118–5126.
Google Scholar
- Jerzy Karczmarczuk. 2001. Functional differentiation of computer programs. Higher-order and symbolic computation.
Google Scholar
- Faustyna Krawiec, Neel Krishnaswami, Simon Peyton Jones, Tom Ellis, Andrew Fitzgibbon, and R Eisenberg. 2022. Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation. Proceedings of the ACM on Programming Languages.
Google Scholar
Digital Library
- Jacob Laurel, Rem Yang, Gagandeep Singh, and Sasa Misailovic. 2022. A Dual Number Abstraction for Static Analysis of Clarke Jacobians. Proceedings of the ACM on Programming Languages.
Google Scholar
Digital Library
- Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Appendix to A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://jsl1994.github.io/papers/OOPSLA2022_appendix.pdf
Google Scholar
- Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Artifact for A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://doi.org/10.1145/3554329
Google Scholar
Digital Library
- Yann LeCun. 1998. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
Google Scholar
- Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics (TOG), 37, 4 (2018).
Google Scholar
Digital Library
- Ravi Mangal, Kartik Sarangmath, Aditya V Nori, and Alessandro Orso. 2020. Probabilistic Lipschitz Analysis of Neural Networks. In International Static Analysis Symposium.
Google Scholar
- Antoine Miné. 2006. Symbolic methods to enhance the precision of numerical abstract domains. In International Workshop on Verification, Model Checking, and Abstract Interpretation.
Google Scholar
- Adam Paszke, Daniel D Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew J Johnson, Jonathan Ragan-Kelley, and Dougal Maclaurin. 2021. Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming. Proceedings of the ACM on Programming Languages.
Google Scholar
Digital Library
- Colin Paterson, Haoze Wu, John Grese, Radu Calinescu, Corina S Păsăreanu, and Clark Barrett. 2021. Deepcert: Verification of contextually relevant robustness for neural network image classifiers. In International Conference on Computer Safety, Reliability, and Security.
Google Scholar
Digital Library
- Barak A Pearlmutter and Jeffrey Mark Siskind. 2007. Lazy Multivariate Higher-Order Forward-Mode AD. In Symposium on Principles of Programming Languages.
Google Scholar
- Thomas Reps and Aditya Thakur. 2016. Automating abstract interpretation. In International Conference on Verification, Model Checking, and Abstract Interpretation.
Google Scholar
Digital Library
- Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. 2021. Scalable polyhedral verification of recurrent neural networks. In International Conference on Computer Aided Verification.
Google Scholar
Digital Library
- Benjamin Sherman, Jesse Michel, and Michael Carbin. 2021. λ _S: Computable Semantics for Differentiable Programming with Higher-Order Functions and Datatypes. Proceedings of the ACM on Programming Languages.
Google Scholar
- Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. 2019. Robustness Verification for Transformers. In International Conference on Learning Representations.
Google Scholar
- Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin T Vechev. 2018. Fast and Effective Robustness Certification.. NeurIPS.
Google Scholar
- Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages.
Google Scholar
Digital Library
- Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017. Fast polyhedra abstract domain. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages.
Google Scholar
Digital Library
- Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. A practical construction for decomposing numerical abstract domains. Proceedings of the ACM on Programming Languages.
Google Scholar
Digital Library
- Matthew Sotoudeh and Aditya V Thakur. 2020. Abstract Neural Networks. In International Static Analysis Symposium.
Google Scholar
- Ole Stauning. 1997. Automatic validation of numerical solutions.
Google Scholar
- Jorge Stolfi and Luiz Henrique de Figueiredo. 2003. An introduction to affine arithmetic. Trends in Computational and Applied Mathematics, 4 (2003).
Google Scholar
- Vassilis Vassiliadis, Jan Riehme, Jens Deussen, Konstantinos Parasyris, Christos D Antonopoulos, Nikolaos Bellas, Spyros Lalis, and Uwe Naumann. 2016. Towards automatic significance analysis for approximate computing. In 2016 IEEE/ACM International Symposium on Code Generation and Optimization.
Google Scholar
Digital Library
- Andrea Walther and Andreas Griewank. 2012. Getting Started with ADOL-C. Combinatorial Scientific Computing.
Google Scholar
- Rem Yang, Jacob Laurel, Sasa Misailovic, and Gagandeep Singh. 2022. Provable Defense Against Geometric Transformations. arXiv preprint arXiv:2207.11177.
Google Scholar
Index Terms
A general construction for abstract interpretation of higher-order automatic differentiation
Recommendations
Abstract interpretation of resolution-based semantics
We extend the abstract interpretation point of view on context-free grammars by Cousot and Cousot to resolution-based logic programs and proof systems. Starting from a transition-based small-step operational semantics of Prolog programs (akin to the ...
Stable relations and abstract interpretation of higher-order programs
We present a novel denotational semantics for the untyped call-by-value λ-calculus, where terms are interpreted as stable relations, i.e. as binary relations between substitutions and values, enjoying a monotonicity property. The denotation captures the ...
Abstract Interpretation as Automated Deduction
Automata theory, algorithmic deduction and abstract interpretation provide the foundation behind three approaches to implementing program verifiers. This article is a first step towards a mathematical translation between these approaches. By extending ...






Comments