Abstract
To verify safety and robustness of neural networks, researchers have successfully applied abstract interpretation, primarily using the interval abstract domain. In this paper, we study the theoretical power and limits of the interval domain for neural-network verification.
First, we introduce the interval universal approximation (IUA) theorem. IUA shows that neural networks not only can approximate any continuous function f (universal approximation) as we have known for decades, but we can find a neural network, using any well-behaved activation function, whose interval bounds are an arbitrarily close approximation of the set semantics of f (the result of applying f to a set of inputs). We call this notion of approximation interval approximation. Our theorem generalizes the recent result of Baader et al. from ReLUs to a rich class of activation functions that we call squashable functions. Additionally, the IUA theorem implies that we can always construct provably robust neural networks under ℓ∞-norm using almost any practical activation function.
Second, we study the computational complexity of constructing neural networks that are amenable to precise interval analysis. This is a crucial question, as our constructive proof of IUA is exponential in the size of the approximation domain. We boil this question down to the problem of approximating the range of a neural network with squashable activation functions. We show that the range approximation problem (RA) is a Δ2-intermediate problem, which is strictly harder than NP-complete problems, assuming coNP⊄NP. As a result, IUA is an inherently hard problem: No matter what abstract domain or computational tools we consider to achieve interval approximation, there is no efficient construction of such a universal approximator. This implies that it is hard to construct a provably robust network, even if we have a robust network to start with.
Supplemental Material
Available for Download
This is the full paper with the appendix section. It contains a generalization from the two dimensional theorem to a high-dimensional one, elided proofs and specifications in the main text.
- Aws Albarghouthi. 2021. Introduction to Neural Network Verification. CoRR, abs/2109.10317 (2021), arXiv:2109.10317. https://verifieddeeplearning.comGoogle Scholar
Cross Ref
- Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. 2019. Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). Association for Computing Machinery, New York, NY, USA. 731–744. isbn:9781450367127 https://doi.org/10.1145/3314221.3314614 Google Scholar
Digital Library
- Cem Anil, James Lucas, and Roger Grosse. 2019. Sorting Out Lipschitz Function Approximation. In Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.) (Proceedings of Machine Learning Research, Vol. 97). PMLR, Long Beach, California, USA. 291–301. http://proceedings.mlr.press/v97/anil19a.htmlGoogle Scholar
- Maximilian Baader, Matthew Mirman, and Martin Vechev. 2020. Universal Approximation with Certified Networks. In International Conference on Learning Representations. https://openreview.net/forum?id=B1gX8kBtPrGoogle Scholar
- James Bergstra, Guillaume Desjardins, Pascal Lamblin, and Yoshua Bengio. 2009. Quadratic polynomials learn better image features. Technical report, 1337.Google Scholar
- Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). arxiv:1511.07289Google Scholar
- Jeremy E. J. Cohen, Todd Huster, and Ra Cohen. 2019. Universal Lipschitz Approximation in Bounded Depth Neural Networks. arxiv:1904.04861.Google Scholar
- Patrick Cousot and Radhia Cousot. 1977. Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL ’77). Association for Computing Machinery, New York, NY, USA. 238–252. isbn:9781450373500 https://doi.org/10.1145/512950.512973 Google Scholar
Digital Library
- Patrick Cousot and Nicolas Halbwachs. 1978. Automatic discovery of linear restraints among variables of a program. In Proceedings of the 5th ACM SIGACT-SIGPLAN symposium on Principles of programming languages. 84–96. https://doi.org/10.1145/512760.512770 Google Scholar
Digital Library
- George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2 (1989), 303–314. https://doi.org/10.1007/BF02551274 Google Scholar
Cross Ref
- Rick Durrett. 2010. Probability: Theory and Examples (4 ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511779398 Google Scholar
Cross Ref
- Ruediger Ehlers. 2017. Formal verification of piece-wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis. 269–286.Google Scholar
Cross Ref
- T. Gehr, M. Mirman, D. Drachsler-Cohen, P. Tsankov, S. Chaudhuri, and M. Vechev. 2018. AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). 3–18. https://doi.org/10.1109/SP.2018.00058 Google Scholar
Cross Ref
- Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The zonotope abstract domain taylor1+. In International Conference on Computer Aided Verification. 627–633. https://doi.org/10.1007/978-3-642-02658-4_47 Google Scholar
Digital Library
- Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep Sparse Rectifier Neural Networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.) (Proceedings of Machine Learning Research, Vol. 15). PMLR, Fort Lauderdale, FL, USA. 315–323. https://proceedings.mlr.press/v15/glorot11a.htmlGoogle Scholar
- Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations. arxiv:1412.6572Google Scholar
- Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Arthur Mann, and Pushmeet Kohli. 2019. Scalable Verified Training for Provably Robust Image Classification. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 4841–4850. https://doi.org/10.1109/ICCV.2019.00494 Google Scholar
Cross Ref
- Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. 2020. ReLU Deep Neural Networks and Linear Finite Elements. Journal of Computational Mathematics, 38, 3 (2020), 502–527. issn:1991-7139 https://doi.org/10.4208/jcm.1901-m2018-0160 Google Scholar
Cross Ref
- Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators.. Neural networks, 2, 5 (1989), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8 Google Scholar
Cross Ref
- Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019. Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. 4081–4091. https://doi.org/10.18653/v1/D19-1419 Google Scholar
Cross Ref
- Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification. 97–117. https://doi.org/10.1007/978-3-319-63387-9_5 Google Scholar
- Patrick Kidger and Terry Lyons. 2019. Universal approximation with deep narrow networks. arXiv preprint arXiv:1905.08539.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.Google Scholar
- Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. 1993. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6, 6 (1993), 861 – 867. issn:0893-6080 https://doi.org/10.1016/S0893-6080(05)80131-5 Google Scholar
Digital Library
- Hongzhou Lin and Stefanie Jegelka. 2018. ResNet with one-neuron hidden layers is a Universal Approximator. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 6169–6178. http://papers.nips.cc/paper/7855-resnet-with-one-neuron-hidden-layers-is-a-universal-approximator.pdfGoogle Scholar
- Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. 2017. The Expressive Power of Neural Networks: A View from the Width. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, USA. 6232–6240. isbn:9781510860964Google Scholar
Cross Ref
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.Google Scholar
- Matthew Mirman, Timon Gehr, and Martin Vechev. 2018. Differentiable Abstract Interpretation for Provably Robust Neural Networks. In Proceedings of the 35th International Conference on Machine Learning, Jennifer Dy and Andreas Krause (Eds.) (Proceedings of Machine Learning Research, Vol. 80). PMLR, 3578–3586. https://proceedings.mlr.press/v80/mirman18b.htmlGoogle Scholar
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML.Google Scholar
- Michael A Nielsen. 2015. Neural networks and deep learning. 2018, Determination press San Francisco, CA.Google Scholar
- Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". SIGPLAN Not., 50, 1 (2015), jan, 111–124. issn:0362-1340 https://doi.org/10.1145/2775051.2677009 Google Scholar
Digital Library
- Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. 2018. Fast and effective robustness certification. In Advances in Neural Information Processing Systems. 10802–10813.Google Scholar
- Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3, POPL (2019), 1–30. https://doi.org/10.1145/3290354 Google Scholar
Digital Library
- Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019. Evaluating Robustness of Neural Networks with Mixed Integer Programming. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=HyGIdiRqtmGoogle Scholar
- V.V. Vazirani. 2003. Approximation Algorithms. Springer Berlin Heidelberg. isbn:978-3-642-08469-0 https://doi.org/10.1007/978-3-662-04565-7 Google Scholar
Cross Ref
- Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal Security Analysis of Neural Networks Using Symbolic Intervals. In Proceedings of the 27th USENIX Conference on Security Symposium (SEC’18). USENIX Association, USA. 1599–1614. isbn:9781931971461Google Scholar
Digital Library
- Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning, Inderjit S. Dhillon, and Luca Daniel. 2018. Towards Fast Computation of Certified Robustness for ReLU Networks. In International Conference on Machine Learning (ICML).Google Scholar
- Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le. 2020. Smooth Adversarial Training. arXiv preprint arXiv:2006.14536.Google Scholar
- Yuhao Zhang, Aws Albarghouthi, and Loris D’Antoni. 2020. Robustness to Programmable String Transformations via Augmented Abstract Training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 11023–11032. http://proceedings.mlr.press/v119/zhang20b.htmlGoogle Scholar
- Yuhao Zhang, Aws Albarghouthi, and Loris D’Antoni. 2021. Certified Robustness to Programmable Transformations in LSTMs. CoRR, abs/2102.07818 (2021), arxiv:2102.07818. arxiv:2102.07818Google Scholar
Index Terms
Interval universal approximation for neural networks
Recommendations
Universal approximation using incremental constructive feedforward networks with random hidden nodes
According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. ...
The universal approximation capabilities of double 2 ∏-periodic approximate identity neural networks
The purpose of this study is to investigate the universal approximation capabilities of a certain class of single-hidden-layer feedforward neural networks, which is called double 2$$\pi $$ -periodic approximate identity neural networks. Using double 2$$\...
The universal approximation capabilities of mellin approximate identity neural networks
ISNN'13: Proceedings of the 10th international conference on Advances in Neural Networks - Volume Part IUniversal approximation capability of feedforward neural networks with one hidden layer states that these networks are dense in the space of functions. In this paper, the concept of the Mellin approximate identity functions is proposed. By using this ...






Comments