Abstract
Recently, Graph Neural Networks (GNNs) have been applied for scheduling jobs over clusters, achieving better performance than hand-crafted heuristics. Despite their impressive performance, concerns remain over whether these GNN-based job schedulers meet users’ expectations about other important properties, such as strategy-proofness, sharing incentive, and stability. In this work, we consider formal verification of GNN-based job schedulers. We address several domain-specific challenges such as networks that are deeper and specifications that are richer than those encountered when verifying image and NLP classifiers. We develop vegas, the first general framework for verifying both single-step and multi-step properties of these schedulers based on carefully designed algorithms that combine abstractions, refinements, solvers, and proof transfer. Our experimental results show that vegas achieves significant speed-up when verifying important properties of a state-of-the-art GNN-based scheduler compared to previous methods.
- Alfred V Aho, Monica S Lam, Ravi Sethi, and Jeffrey D Ullman. 2007. Compilers: principles, techniques, & tools. Pearson Education India.
Google Scholar
- Guy Amir, Michael Schapira, and Guy Katz. 2021. Towards Scalable Verification of Deep Reinforcement Learning. In 2021 Formal Methods in Computer-Aided Design (FMCAD). 193–203. https://doi.org/10.34727/2021/isbn.978-3-85448-046-4_28
Google Scholar
Cross Ref
- Greg Anderson, Shankara Pailoor, Isil Dillig, and Swarat Chaudhuri. 2019. Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness. In Proc. Programming Language Design and Implementation (PLDI). 731–744. https://doi.org/10.1145/3314221.3314614
Google Scholar
Digital Library
- Stanley Bak, Hoang-Dung Tran, Kerianne Hobbs, and Taylor T Johnson. 2020. Improved geometric path enumeration for verifying ReLU neural networks. In International Conference on Computer Aided Verification. 66–96. https://doi.org/10.1007/978-3-030-53288-8_4
Google Scholar
Digital Library
- Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle. 2013. The datacenter as a computer: An introduction to the design of warehouse-scale machines. Synthesis lectures on computer architecture, 8, 3 (2013), 1–154.
Google Scholar
- Aleksandar Bojchevski and Stephan Günnemann. 2019. Certifiable robustness to graph perturbations. arXiv preprint arXiv:1910.14356.
Google Scholar
- Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel. 2019. Cnn-cert: An efficient framework for certifying robustness of convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence. 33, 3240–3247. https://doi.org/10.1609/aaai.v33i01.33013240
Google Scholar
Digital Library
- Elena Botoeva, Panagiotis Kouvaros, Jan Kronqvist, Alessio Lomuscio, and Ruth Misener. 2020. Efficient verification of relu-based neural networks via dependency analysis. In Proceedings of the AAAI Conference on Artificial Intelligence. 34, 3291–3299. https://doi.org/10.1609/aaai.v34i04.5729
Google Scholar
Cross Ref
- Rudy Bunel, Jingyue Lu, Ilker Turkaslan, Pushmeet Kohli, P Torr, and P Mudigonda. 2020. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research, 21, 2020 (2020).
Google Scholar
- Rudy R Bunel, Ilker Turkaslan, Philip Torr, Pushmeet Kohli, and Pawan K Mudigonda. 2018. A Unified View of Piecewise Linear Neural Network Verification. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). 31, Curran Associates, Inc.. https://proceedings.neurips.cc/paper/2018/file/be53d253d6bc3258a8160556dda3e9b2-Paper.pdf
Google Scholar
- Hongkai Dai, Benoit Landry, Lujie Yang, Marco Pavone, and Russ Tedrake. 2021. Lyapunov-stable neural-network control. arXiv preprint arXiv:2109.14152.
Google Scholar
- Alessandro De Palma, Harkirat Singh Behl, Rudy Bunel, Philip HS Torr, and M Pawan Kumar. 2021. Scaling the convex barrier with active sets. arXiv preprint arXiv:2101.05844.
Google Scholar
- Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29 (2016), https://doi.org/10.5555/3157382.3157527
Google Scholar
Digital Library
- Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. 2018. Output Range Analysis for Deep Feedforward Neural Networks. In NASA Formal Methods - 10th International Symposium, NFM 2018, Newport News, VA, USA, April 17-19, 2018, Proceedings.
Google Scholar
- David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 2224–2232.
Google Scholar
- Ruediger Ehlers. 2017. Formal verification of piece-wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis. 269–286.
Google Scholar
Cross Ref
- Matteo Fischetti and Jason Jo. 2017. Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility Study. CoRR, abs/1712.06174 (2017).
Google Scholar
- Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. 2017. Protein Interface Prediction using Graph Convolutional Networks. In Proc. Advances in Neural Information Processing Systems (NeurIPS). 6530–6539.
Google Scholar
- Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, and Corina Păsăreanu. 2020. Fast geometric projections for local robustness certification. arXiv preprint arXiv:2002.04742.
Google Scholar
- Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin T. Vechev. 2018. AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In 2018 IEEE Symposium on Security and Privacy, SP 2018, Proceedings, 21-23 May 2018, San Francisco, California, USA. 3–18. https://doi.org/10.1109/SP.2018.00058
Google Scholar
Cross Ref
- Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion Stoica. 2011. Dominant resource fairness: Fair allocation of multiple resource types. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11). https://doi.org/10.5555/1972457.1972490
Google Scholar
Digital Library
- Patrick Henriksen and Alessio Lomuscio. 2021. DEEPSPLIT: An Efficient Splitting Method for Neural Network Verification via Indirect Effect Analysis. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 2549–2555. https://doi.org/10.24963/ijcai.2021/351 Main Track
Google Scholar
Cross Ref
- Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. 2017. Safety Verification of Deep Neural Networks. In CAV.
Google Scholar
- Kirthevasan Kandasamy, Gur-Eyal Sela, Joseph E Gonzalez, Michael I Jordan, and Ion Stoica. 2020. Online learning demands in max-min fairness. arXiv preprint arXiv:2012.08648.
Google Scholar
- G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer. 2017. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In Proc. 29th Int. Conf. on Computer Aided Verification (CAV). 97–117.
Google Scholar
- Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, and Aleksandar Zeljić. 2019. The marabou framework for verification and analysis of deep neural networks. In International Conference on Computer Aided Verification. 443–452. https://doi.org/10.1007/978-3-030-25540-4_26
Google Scholar
Cross Ref
- Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. 2017. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30 (2017), https://doi.org/10.5555/3295222.3295382
Google Scholar
Digital Library
- Haitham Khedr, James Ferlez, and Yasser Shoukry. 2020. PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier. arXiv preprint arXiv:2006.10864.
Google Scholar
- Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proc. International Conference on Learning Representations, (ICLR). OpenReview.net.
Google Scholar
- Yann LeCun and Corinna Cortes. 2010. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/. http://yann.lecun.com/exdb/mnist/
Google Scholar
- Jingyue Lu and M Pawan Kumar. 2019. Neural network branching for neural network verification. arXiv preprint arXiv:1912.01329.
Google Scholar
- Zhaoyang Lyu, Ching-Yun Ko, Zhifeng Kong, Ngai Wong, Dahua Lin, and Luca Daniel. 2020. Fastened crown: Tightened neural network robustness certificates. In Proceedings of the AAAI Conference on Artificial Intelligence. 34, 5037–5044.
Google Scholar
Cross Ref
- Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. 2019. Learning scheduling algorithms for data processing clusters. In Proceedings of the ACM Special Interest Group on Data Communication. 270–288. https://doi.org/10.1145/3341302.3342080
Google Scholar
Digital Library
- Christoph Müller, François Serre, Gagandeep Singh, Markus Püschel, and Martin Vechev. 2021. Scaling Polyhedral Neural Network Verification on GPUs. Proceedings of Machine Learning and Systems, 3 (2021).
Google Scholar
- Mark Niklas Müller, Gleb Makarchuk, Gagandeep Singh, Markus Püschel, and Martin Vechev. 2021. Precise Multi-Neuron Abstractions for Neural Network Certification. arXiv preprint arXiv:2103.03638.
Google Scholar
- Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning Convolutional Neural Networks for Graphs. In Proc. International Conference on Machine Learning, ICML. 48, 2014–2023.
Google Scholar
- Junyoung Park, Jaehyeong Chun, Sang Hun Kim, Youngkook Kim, and Jinkyoo Park. 2021. Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning. International Journal of Production Research, 59, 11 (2021), 3360–3377.
Google Scholar
Cross Ref
- Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Semidefinite relaxations for certifying robustness to adversarial examples. arXiv preprint arXiv:1811.01057.
Google Scholar
- Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. 2021. Scalable Polyhedral Verification of Recurrent Neural Networks. In International Conference on Computer Aided Verification. 225–248.
Google Scholar
- Hadi Salman, Greg Yang, Huan Zhang, Cho-Jui Hsieh, and Pengchuan Zhang. 2019. A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (Eds.). 32, Curran Associates, Inc.. https://proceedings.neurips.cc/paper/2019/file/246a3c5544feb054f3ea718f61adfa16-Paper.pdf
Google Scholar
- Ali Ghodsi Matei Zaharia Scott Shenker and Ion Stoica. 2013. Choosy: Max-Min Fair Sharing for Datacenter Jobs with Constraints. https://doi.org/10.1145/2465351.2465387
Google Scholar
Digital Library
- Gagandeep Singh, Rupanshu Ganvir, Markus Püschel, and Martin Vechev. 2019. Beyond the single neuron convex barrier for neural network certification. Advances in Neural Information Processing Systems, 32 (2019), 15098–15109.
Google Scholar
- Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. 2018. Fast and effective robustness certification. Advances in Neural Information Processing Systems, 31 (2018), 10802–10813.
Google Scholar
- Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3, POPL (2019), 1–30. https://doi.org/10.1145/3290354
Google Scholar
Digital Library
- Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. Boosting Robustness Certification of Neural Networks. In International Conference on Learning Representations.
Google Scholar
- Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017. Fast polyhedra abstract domain. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages. ACM New York, NY, USA, 46–59.
Google Scholar
Digital Library
- Gagandeep Singh, Markus Püschel, and Martin T. Vechev. 2018. A practical construction for decomposing numerical abstract domains. Proc. ACM Program. Lang., 2, POPL (2018), 55:1–55:28.
Google Scholar
- Penghao Sun, Zehua Guo, Junchao Wang, Junfei Li, Julong Lan, and Yuxiang Hu. 2021. Deepweave: Accelerating job completion time with deep reinforcement learning-based coflow scheduling. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 3314–3320. https://doi.org/10.5555/3491440.3491898
Google Scholar
Digital Library
- Xiaowu Sun, Haitham Khedr, and Yasser Shoukry. 2019. Formal verification of neural network controlled autonomous systems. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control. 147–156. https://doi.org/10.1145/3302504.3311802
Google Scholar
Digital Library
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
Google Scholar
- Christian Tjandraatmadja, Ross Anderson, Joey Huchette, Will Ma, KRUNAL KISHOR PATEL, and Juan Pablo Vielma. 2020. The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). 33, Curran Associates, Inc., 21675–21686. https://proceedings.neurips.cc/paper/2020/file/f6c2a0c4b566bc99d596e58638e342b0-Paper.pdf
Google Scholar
- Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019. Evaluating Robustness of Neural Networks with Mixed Integer Programming. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. https://openreview.net/forum?id=HyGIdiRqtm
Google Scholar
- Hoang-Dung Tran, Stanley Bak, Weiming Xiang, and Taylor T Johnson. 2020. Verification of deep convolutional neural networks using imagestars. In International Conference on Computer Aided Verification. 18–42.
Google Scholar
Digital Library
- Caterina Urban, Maria Christakis, Valentin Wüstholz, and Fuyuan Zhang. 2020. Perfectly parallel fairness certification of neural networks. Proceedings of the ACM on Programming Languages, 4, OOPSLA (2020), 1–30. https://doi.org/10.1145/3428253
Google Scholar
Digital Library
- Joseph A Vincent and Mac Schwager. 2020. Reachable Polyhedral Marching (RPM): A Safety Verification Algorithm for Robotic Systems with Deep Neural Network Components. arXiv preprint arXiv:2011.11609.
Google Scholar
- Binghui Wang, Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. 2021. Certified robustness of graph neural networks against adversarial structural perturbation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1645–1653.
Google Scholar
Digital Library
- Shen Wang, Zhengzhang Chen, Xiao Yu, Ding Li, Jingchao Ni, Lu-An Tang, Jiaping Gui, Zhichun Li, Haifeng Chen, and Philip S. Yu. 2019. Heterogeneous Graph Matching Networks for Unknown Malware Detection. In Proc. International Joint Conference on Artificial Intelligence, (IJCAI). 3762–3770.
Google Scholar
- Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Efficient Formal Safety Analysis of Neural Networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada. 6369–6379. http://papers.nips.cc/paper/7873-efficient-formal-safety-analysis-of-neural-networks
Google Scholar
- Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal Security Analysis of Neural Networks using Symbolic Intervals. In 27th USENIX Security Symposium, USENIX Security 2018, Baltimore, MD, USA, August 15-17, 2018. 1599–1614. https://www.usenix.org/conference/usenixsecurity18/presentation/wang-shiqi
Google Scholar
- Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. 2021. Beta-crown: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network verification. arXiv preprint arXiv:2103.06624.
Google Scholar
- Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. 2018. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning. 5276–5285.
Google Scholar
- Eric Wong and Zico Kolter. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning. 5286–5295.
Google Scholar
- Haoze Wu, Clark Barrett, Mahmood Sharif, Nina Narodytska, and Gagandeep Singh. 2022. Artifact for Paper Scalable Verification of GNN- Based Job Schedulers. https://doi.org/10.5281/zenodo.7080246
Google Scholar
Digital Library
- Haoze Wu, Alex Ozdemir, Aleksandar Zeljić, Kyle Julian, Ahmed Irfan, Divya Gopinath, Sadjad Fouladi, Guy Katz, Corina Pasareanu, and Clark Barrett. 2020. Parallelization techniques for verifying neural networks. In 2020 Formal Methods in Computer Aided Design (FMCAD). 128–137. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_20
Google Scholar
Cross Ref
- Haoze Wu, Aleksandar Zeljić, Guy Katz, and Clark Barrett. 2022. Efficient Neural Network Analysis with Sum-of-Infeasibilities. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 143–163.
Google Scholar
- Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32, 1 (2020), 4–24.
Google Scholar
Cross Ref
- Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. 2018. Output reachable set estimation and verification for multilayer neural networks. IEEE transactions on neural networks and learning systems, 29, 11 (2018), 5777–5783.
Google Scholar
- Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. 2020. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. arXiv preprint arXiv:2011.13824.
Google Scholar
- Pengfei Yang, Renjue Li, Jianlin Li, Cheng-Chao Huang, Jingyi Wang, Jun Sun, Bai Xue, and Lijun Zhang. 2021. Improving neural network verification through spurious region guided refinement. Tools and Algorithms for the Construction and Analysis of Systems, 12651 (2021), 389.
Google Scholar
- Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In Proc. ACM SIGKDD Knowledge Discovery & Data Mining, KDD. ACM, 974–983.
Google Scholar
Digital Library
- Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica. 2010. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems. 265–278. https://doi.org/10.1145/1755913.1755940
Google Scholar
Digital Library
- Tom Zelazny, Haoze Wu, Clark Barrett, and Guy Katz. 2022. On Optimizing Back-Substitution Methods for Neural Network Verification. arXiv preprint arXiv:2208.07669.
Google Scholar
- Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. 2018. Efficient Neural Network Robustness Certification with General Activation Functions. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). 31, Curran Associates, Inc.. https://proceedings.neurips.cc/paper/2018/file/d04863f100d59b3eb688a11f95b0ae60-Paper.pdf
Google Scholar
Index Terms
Scalable verification of GNN-based job schedulers
Recommendations
Formal verification of ASMs using MDGs
We present a framework for the formal verification of abstract state machine (ASM) designs using the multiway decision graphs (MDG) tool. ASM is a state based language for describing transition systems. MDG provides symbolic representation of transition ...
Effective Liveness Verification Using a Transformation-Based Framework
VLSID '14: Proceedings of the 2014 27th International Conference on VLSI Design and 2014 13th International Conference on Embedded SystemsLiveness properties such as "will every request eventually get a grant?" are crucial to the verification of a variety of design types. Liveness properties may only be falsified by infinite-length counterexamples, represented using lasso-shaped traces ...
Assertion Based Verification using Yosys: A Case Study from Nuclear Domain
ISEC '23: Proceedings of the 16th Innovations in Software Engineering ConferenceAssertion Based Verification is a design methodology that integrates Formal Methods as part of the design process. As each module is designed, the designer expresses the functional, structural and interface requirements of the module as logical formulas ...






Comments