skip to main content
research-article
Open Access

Interval universal approximation for neural networks

Published:12 January 2022Publication History
Skip Abstract Section

Abstract

To verify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using the interval abstract domain. In this paper, we study the theoretical power and limits of the interval domain for neural-network verification.

First, we introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks not only can approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrarily close approximation of the set semantics of f (the result of applying f to a set of inputs). We call this notion of approximation interval approximation. Our theorem generalizes the recent result of Baader et al. from ReLUs to a rich class of activation functions that we call squashable functions. Additionally, the IUA theorem implies that we can always construct provably robust neural networks under ℓ-norm using almost any practical activation function.

Second, we study the computational complexity of constructing neural networks that are amenable to precise interval analysis. This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain. We boil this question down to the problem of approximating the range of a neural network with squashable activation functions. We show that the range approximation problem (RA) is a Δ2-intermediate problem, which is strictly harder than NP-complete problems, assuming coNPNP. As a result, IUA is an inherently hard problem: No matter what abstract domain or computational tools we consider to achieve interval approximation, there is no efficient construction of such a universal approximator. This implies that it is hard to construct a provably robust network, even if we have a robust network to start with.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

This is the short video presentation of the paper.

References

  1. Aws Albarghouthi. 2021. Introduction to Neural Network Verification. CoRR, abs/2109.10317 (2021), arXiv:2109.10317. https://verifieddeeplearning.comGoogle ScholarGoogle ScholarCross RefCross Ref
  2. Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. 2019. Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). Association for Computing Machinery, New York, NY, USA. 731–744. isbn:9781450367127 https://doi.org/10.1145/3314221.3314614 Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Cem Anil, James Lucas, and Roger Grosse. 2019. Sorting Out Lipschitz Function Approximation. In Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, Long Beach, California, USA. 291–301. http://proceedings.mlr.press/v97/anil19a.htmlGoogle ScholarGoogle Scholar
  4. Maximilian Baader, Matthew Mirman, and Martin Vechev. 2020. Universal Approximation with Certified Networks. In International Conference on Learning Representations. https://openreview.net/forum?id=B1gX8kBtPrGoogle ScholarGoogle Scholar
  5. James Bergstra, Guillaume Desjardins, Pascal Lamblin, and Yoshua Bengio. 2009. Quadratic polynomials learn better image features. Technical report, 1337.Google ScholarGoogle Scholar
  6. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). arxiv:1511.07289Google ScholarGoogle Scholar
  7. Jeremy E. J. Cohen, Todd Huster, and Ra Cohen. 2019. Universal Lipschitz Approximation in Bounded Depth Neural Networks. arxiv:1904.04861.Google ScholarGoogle Scholar
  8. Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL ’77). Association for Computing Machinery, New York, NY, USA. 238–252. isbn:9781450373500 https://doi.org/10.1145/512950.512973 Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Patrick Cousot and Nicolas Halbwachs. 1978. Automatic discovery of linear restraints among variables of a program. In Proceedings of the 5th ACM SIGACT-SIGPLAN symposium on Principles of programming languages. 84–96. https://doi.org/10.1145/512760.512770 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2 (1989), 303–314. https://doi.org/10.1007/BF02551274 Google ScholarGoogle ScholarCross RefCross Ref
  11. Rick Durrett. 2010. Probability: Theory and Examples (4 ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511779398 Google ScholarGoogle ScholarCross RefCross Ref
  12. Ruediger Ehlers. 2017. Formal verification of piece-wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis. 269–286.Google ScholarGoogle ScholarCross RefCross Ref
  13. T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. 2018. AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). 3–18. https://doi.org/10.1109/SP.2018.00058 Google ScholarGoogle ScholarCross RefCross Ref
  14. Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The zonotope abstract domain taylor1+. In International Conference on Computer Aided Verification. 627–633. https://doi.org/10.1007/978-3-642-02658-4_47 Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep Sparse Rectifier Neural Networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.) (Proceedings of Machine Learning Research, Vol. 15). PMLR, Fort Lauderdale, FL, USA. 315–323. https://proceedings.mlr.press/v15/glorot11a.htmlGoogle ScholarGoogle Scholar
  16. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations. arxiv:1412.6572Google ScholarGoogle Scholar
  17. Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Arthur Mann, and Pushmeet Kohli. 2019. Scalable Verified Training for Provably Robust Image Classification. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 4841–4850. https://doi.org/10.1109/ICCV.2019.00494 Google ScholarGoogle ScholarCross RefCross Ref
  18. Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. 2020. ReLU Deep Neural Networks and Linear Finite Elements. Journal of Computational Mathematics, 38, 3 (2020), 502–527. issn:1991-7139 https://doi.org/10.4208/jcm.1901-m2018-0160 Google ScholarGoogle ScholarCross RefCross Ref
  19. Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators.. Neural networks, 2, 5 (1989), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8 Google ScholarGoogle ScholarCross RefCross Ref
  20. Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019. Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. 4081–4091. https://doi.org/10.18653/v1/D19-1419 Google ScholarGoogle ScholarCross RefCross Ref
  21. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification. 97–117. https://doi.org/10.1007/978-3-319-63387-9_5 Google ScholarGoogle Scholar
  22. Patrick Kidger and Terry Lyons. 2019. Universal approximation with deep narrow networks. arXiv preprint arXiv:1905.08539.Google ScholarGoogle Scholar
  23. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.Google ScholarGoogle Scholar
  24. Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. 1993. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6, 6 (1993), 861 – 867. issn:0893-6080 https://doi.org/10.1016/S0893-6080(05)80131-5 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Hongzhou Lin and Stefanie Jegelka. 2018. ResNet with one-neuron hidden layers is a Universal Approximator. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 6169–6178. http://papers.nips.cc/paper/7855-resnet-with-one-neuron-hidden-layers-is-a-universal-approximator.pdfGoogle ScholarGoogle Scholar
  26. Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. 2017. The Expressive Power of Neural Networks: A View from the Width. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, USA. 6232–6240. isbn:9781510860964Google ScholarGoogle ScholarCross RefCross Ref
  27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.Google ScholarGoogle Scholar
  28. Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable Abstract Interpretation for Provably Robust Neural Networks. In Proceedings of the 35th International Conference on Machine Learning, Jennifer Dy and Andreas Krause (Eds.) (Proceedings of Machine Learning Research, Vol. 80). PMLR, 3578–3586. https://proceedings.mlr.press/v80/mirman18b.htmlGoogle ScholarGoogle Scholar
  29. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML.Google ScholarGoogle Scholar
  30. Michael A Nielsen. 2015. Neural networks and deep learning. 2018, Determination press San Francisco, CA.Google ScholarGoogle Scholar
  31. Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". SIGPLAN Not., 50, 1 (2015), jan, 111–124. issn:0362-1340 https://doi.org/10.1145/2775051.2677009 Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. 2018. Fast and effective robustness certification. In Advances in Neural Information Processing Systems. 10802–10813.Google ScholarGoogle Scholar
  33. Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3, POPL (2019), 1–30. https://doi.org/10.1145/3290354 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019. Evaluating Robustness of Neural Networks with Mixed Integer Programming. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=HyGIdiRqtmGoogle ScholarGoogle Scholar
  35. V.V. Vazirani. 2003. Approximation Algorithms. Springer Berlin Heidelberg. isbn:978-3-642-08469-0 https://doi.org/10.1007/978-3-662-04565-7 Google ScholarGoogle ScholarCross RefCross Ref
  36. Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal Security Analysis of Neural Networks Using Symbolic Intervals. In Proceedings of the 27th USENIX Conference on Security Symposium (SEC’18). USENIX Association, USA. 1599–1614. isbn:9781931971461Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit S. Dhillon, and Luca Daniel. 2018. Towards Fast Computation of Certified Robustness for ReLU Networks. In International Conference on Machine Learning (ICML).Google ScholarGoogle Scholar
  38. Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le. 2020. Smooth Adversarial Training. arXiv preprint arXiv:2006.14536.Google ScholarGoogle Scholar
  39. Yuhao Zhang, Aws Albarghouthi, and Loris D’Antoni. 2020. Robustness to Programmable String Transformations via Augmented Abstract Training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 11023–11032. http://proceedings.mlr.press/v119/zhang20b.htmlGoogle ScholarGoogle Scholar
  40. Yuhao Zhang, Aws Albarghouthi, and Loris D’Antoni. 2021. Certified Robustness to Programmable Transformations in LSTMs. CoRR, abs/2102.07818 (2021), arxiv:2102.07818. arxiv:2102.07818Google ScholarGoogle Scholar

Index Terms

  1. Interval universal approximation for neural networks

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Programming Languages
          Proceedings of the ACM on Programming Languages  Volume 6, Issue POPL
          January 2022
          1886 pages
          EISSN:2475-1421
          DOI:10.1145/3511309
          Issue’s Table of Contents

          Copyright © 2022 Owner/Author

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 12 January 2022
          Published in pacmpl Volume 6, Issue POPL

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!