Self-Adaptive Perturbation Radii for Adversarial Training

ABSTRACT

Adversarial training has been shown to be the most popular and effective technique to protect models from imperceptible adversarial samples. Despite its success, it also accompanies the significant performance degeneration to clean data. To achieve a good performance on both clean and adversarial samples, the main effort is searching for an adaptive perturbation radius for each training sample. However, this method suffers from a conflict between exact searching and computational overhead. To address this conflict, in this paper, firstly we show the superiority of adaptive perturbation radii on the accuracy and robustness respectively. Then we propose our novel self-adaptive adjustment framework for perturbation radii without tedious searching. We also discuss this framework on both deep neural networks (DNNs) and kernel support vector machines (SVMs). Finally, extensive experimental results show that our framework can improve adversarial robustness without compromising the natural generalization. It is also competitive with existing searching strategies in terms of running time.

References

1. Yogesh Balaji, Tom Goldstein, Judy Hoffman, and Ruitong Huang. 2019. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv:1910.08051 (2019).
2. Runxue Bao, Bin Gu, and Heng Huang. 2022. An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 57--66.
3. Peter L Bartlett and Shahar Mendelson. 2002. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, Vol. 3, Nov (2002), 463--482.
4. Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal. 2019. A kernel perspective for regularizing deep neural networks. In International Conference on Machine Learning. PMLR, 664--674.
5. Battista Biggio, Igino Corona, Blaine Nelson, Benjamin IP Rubinstein, Davide Maiorca, Giorgio Fumera, Giorgio Giacinto, and Fabio Roli. 2014. Security evaluation of support vector machines in adversarial environments. In Support Vector Machines Applications. Springer, 105--153.
6. Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines. International Conference on Machine Learning (2012), 1467--1474.
7. Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010. Springer, 177--186.
8. Sébastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. 2019. Adversarial examples from computational constraints. In International Conference on Machine Learning. PMLR, 831--840.
9. Nicholas Carlini and David Wagner. 2017. Towards Evaluating the Robustness of Neural Networks. IEEE Symposium on Security and Privacy (2017), 39--57.
10. Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 2017. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security. 15--26.
11. Minhao Cheng, Qi Lei, Pin-Yu Chen, Inderjit Dhillon, and Cho-Jui Hsieh. 2020. Cat: Customized adversarial training for improved robustness. arXiv preprint arXiv:2002.06789 (2020).
12. Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning. PMLR, 2206--2216.
13. Bo Dai, Bo Xie, Niao He, Yingyu Liang, Anant Raj, Maria-Florina F Balcan, and Le Song. 2014. Scalable kernel methods via doubly stochastic gradients. In Advances in Neural Information Processing Systems. 3041--3049.
14. Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. 2018. Mma training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637 (2018).
15. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations (2014).
16. Bin Gu, Chenkang Zhang, Huan Xiong, and Heng Huang. 2022. Balanced Self-Paced Learning for AUC Maximization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 6765--6773.
17. Hal Daumé Iii. 2004. From Zero to Reproducing Kernel Hilbert Spaces in Twelve Pages or Less. http://legacydirs.umiacs.umd.edu/ hal/docs/daume04rkhs.pdf.
18. Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. 2015. Self-paced Curriculum Learning. In Twenty-Ninth AAAI Conference on Artificial Intelligence. 2694--2700.
19. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
20. A. Krizhevsky and G. Hinton. 2009. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto (2009).
21. Alexey Kurakin, Ian Goodfellow, Samy Bengio, and David Wagner. 2017. Adversarial Machine Learning at Scale. International Conference on Learning Representations (2017).
22. Y Lecun and L Bottou. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, Vol. 86, 11 (1998), 2278--2324.
23. Michel Ledoux and Michel Talagrand. 1991. Probability in Banach Spaces: isoperimetry and processes. Vol. 23. Springer Science & Business Media.
24. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Attacks. International Conference on Learning Representations (2017).
25. Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, 8 (2018), 1979--1993.
26. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of machine learning. MIT press.
27. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. 2016. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 (2016).
28. Nicolas Papernot, Patrick Mcdaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Learning. Computer and Communications Security (2017), 506--519.
29. L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and Aleksander Mdry. 2018. Adversarially Robust Generalization Requires More Data. Neural Information Processing Systems (2018).
30. Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. 2019. Adversarial training for free! arXiv preprint arXiv:1904.12843 (2019).
31. Wanli Shi and Bin Gu. 2021. Improved Penalty Method via Doubly Stochastic Gradients for Bilevel Hyperparameter Optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 9621--9629.
32. Wanli Shi, Bin Gu, Xiang Li, Xiang Geng, and Heng Huang. 2019. Quadruply stochastic gradients for large scale nonlinear semi-supervised AUC optimization. arXiv preprint arXiv:1907.12416 (2019).
33. Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2018. Robustness May Be at Odds with Accuracy. In International Conference on Learning Representations.
34. Martin J Wainwright. 2019. High-dimensional statistics: A non-asymptotic viewpoint. Vol. 48. Cambridge University Press.
35. Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. 2019. On the Convergence and Robustness of Adversarial Training. In International Conference on Machine Learning.
36. Eric Wong, Leslie Rice, and J Zico Kolter. 2020. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 (2020).
37. Huimin Wu, Zhengmian Hu, and Bin Gu. 2021. Fast and Scalable Adversarial Training of Kernel SVM via Doubly Stochastic Gradients. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 12 (May 2021), 10329--10337.
38. Abraham J Wyner. 2003. On boosting and the exponential loss. In International Workshop on Artificial Intelligence and Statistics. PMLR, 323--329.
39. Dao-Hong Xiang. 2011. Logistic classification with varying Gaussians. Computers & Mathematics with Applications, Vol. 61, 2 (2011), 397--407.
40. Han Xiao, Huang Xiao, and Claudia Eckert. 2012. Adversarial label flips attack on support vector machines. European Conference on Artificial Intelligence (2012), 870--875.
41. Jiancong Xiao, Yanbo Fan, Ruoyu Sun, and Zhi-Quan Luo. 2021. Adversarial Rademacher Complexity of Deep Neural Networks. (2021).
42. Huan Xu, Constantine Caramanis, and Shie Mannor. 2009. Robustness and Regularization of Support Vector Machines. Journal of Machine Learning Research, Vol. 10 (2009), 1485--1510.
43. Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning. PMLR, 7472--7482.
44. Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. 2020. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning. PMLR, 11278--11287.
45. Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Bowei Xi. 2012. Adversarial Support Vector Machine Learning. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Beijing, China) (KDD '12). 1059--1067.

Index Terms

• Computing methodologies

  • Machine learning

    • Learning paradigms

      • Reinforcement learning

        • Adversarial learning

    • Machine learning approaches

      • Kernel methods

        • Support vector machines

      • Neural networks