research-article

Open access

Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning

Authors:

Vasilios Mavroudis,

Tim WatsonAuthors Info & Claims

AISec '23: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security

Pages 91 - 101

https://doi.org/10.1145/3605764.3623986

Published: 26 November 2023 Publication History

Abstract

Communication networks able to withstand hostile environments are critically important for disaster relief operations. In this paper, we consider a challenging scenario where drones have been compromised in the supply chain, during their manufacture, and harbour malicious software capable of wide-ranging and infectious disruption. We investigate multi-agent deep reinforcement learning as a tool for learning defensive strategies that maximise communications bandwidth despite continual adversarial interference. Using a public challenge for learning network resilience strategies, we propose a state-of-the-art symbolic technique and study its superiority over deep reinforcement learning agents. Correspondingly, we identify three specific methods for improving the performance of our neural agents: (1) ensuring each observation contains the necessary information, (2) using symbolic agents to provide a curriculum for learning, and (3) paying close attention to reward. We apply our methods and present a new mixed strategy enabling symbolic and neural agents to work together and improve on all prior results.

References

[1]

2022. Cyber Operations Research Gym. https://github.com/cage-challenge/ CybORG. Created by Maxwell Standen, David Bowman, Son Hoang, Toby Richer, Martin Lucas, Richard Van Tassel, Phillip Vu, Mitchell Kiely, KC C., Natalie Konschnik, Joshua Collyer.

[2]

Amrin Maria Khan Adawadkar and Nilima Kulkarni. 2022. Cyber-Security and Reinforcement Learning -- A Brief Survey. Engineering Applications of Artificial Intelligence (2022).

[3]

Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir. 2022. Developing Optimal Causal Cyber-Defence Agents via Cyber Security Simulation. In Workshop on Machine Learning for Cybersecurity as part of the Proceedings of the 39th International Conference on Machine Learning (ML4Cyber '22).

[4]

Marcin Andrychowicz, Anton Raichuk, and Others. 2021. What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study. In International Conference on Learning Representations (ICLR '21).

[5]

Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, AdrianWebster, and Melody Wolk. 2022. Bridging Automated to Autonomous Cyber Defense: Foundational Analysis of Tabular Q-Learning. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security (AISec'22).

Digital Library

[6]

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2020. Emergent Tool Use From Multi-Agent Autocurricula. In International Conference on Learning Representations (ICLR '20).

[7]

Liz Bates, Chris Hicks, and Vasilios Mavroudis. 2023. Reward Shaping for Happier Autonomous Cyber Security Agents. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec '23), Copenhagen, Denmark (2023).

Digital Library

[8]

Sofia Belikovetsky, Mark Yampolskiy, Jinghui Toh, Jacob Gatlin, and Yuval Elovici. 2017. dr0wned -- Cyber-Physical Attack with Additive Manufacturing. In 11th USENIX Workshop on Offensive Technologies (WOOT '17).

[9]

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (ICML '09).

Digital Library

[10]

Katharina L. Best, Jon Schmid, Shane Tierney, Jalal Awan, Nahom M. Beyene, Maynard A. Holliday, Raza Khan, and Karen Lee. 2020. How to Analyze the Cyber Threat from Drones: Background, Analysis Frameworks, and Analysis Tools. Online. RAND Corporation.

[11]

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym.

[12]

Anthony Burke. 2020. Robust artificial intelligence for active cyber defence. Online. The Alan Turing Institute.

[13]

Robert G. Campbell, Magdalini Eirinaki, and Younghee Park. 2023. A Curriculum Framework for Autonomous Network Defense using Multi-agent Reinforcement Learning. In 2023 Silicon Valley Cybersecurity Conference (SVCC).

[14]

Andrei Costin, Jonas Zaddach, Aurélien Francillon, and Davide Balzarotti. 2014. A Large-Scale Analysis of the Security of Embedded Firmwares. In 23rd USENIX Security Symposium (USENIX Security '14).

[15]

Vishal Dey, Vikramkumar Pudi, Anupam Chattopadhyay, and Yuval Elovici. 2018. Security Vulnerabilities of Unmanned Aerial Vehicles and Countermeasures: An Experimental Study. In 17th International Conference on Embedded Systems (VLSID '18).

[16]

Edsger W Dijkstra. 1959. A note on two problems in connexion with graphs. Numerische mathematik (1959).

Digital Library

[17]

Benjamin C. Fernando, Greg Beams, and Reece Rivera. 2019. Security Analysis of DJI Phantom 3 Standard. Online. Massachusetts Institute of Technology. https: //courses.csail.mit.edu/6.857/2016/files/9.pdf

[18]

Myles Foley, Chris Hicks, Kate Highnam, and Vasilios Mavroudis. 2022. Autonomous Network Defence Using Reinforcement Learning. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security (ASIA CCS '22).

Digital Library

[19]

Myles Foley and Sergio Maffeis. 2022. Haxss: Hierarchical Reinforcement Learning for XSS Payload Generation. In 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). 147--158. https://doi.org/10.1109/TrustCom56396.2022.00031 ISSN: 2324--9013.

[20]

Myles Foley, MiaWang, Zoe M, Chris Hicks, and Vasilios Mavroudis. 2022. Inroads into Autonomous Network Defence using Explained Reinforcement Learning. In Proceedings of the Conference on Applied Machine Learning in Information Security (CAMLIS '22).

[21]

Kunihiko Fukushima. 1975. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics (1975).

[22]

Edward J Glantz, Frank E Ritter, Don Gilbreath, Sarah J Stager, Alexandra Anton, and Rahul Emani. 2020. UAV use in disaster management. In Proceedings of the 17th ISCRAM Conference.

[23]

Vasudev Gohil, Hao Guo, Satwik Patnaik, and Jeyavijayan Rajendran. 2022. ATTRITION: Attacking Static Hardware Trojan Detection Techniques Using Reinforcement Learning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS '22).

Digital Library

[24]

TTCP CAGE Working Group. 2022. TTCP CAGE Challenge 3. https://github. com/cage-challenge/cage-challenge-3.

[25]

Kim Hammar and Rolf Stadler. 2020. Finding Effective Security Strategies through Reinforcement Learning and Self-Play. In 2020 16th International Conference on Network and Service Management (CNSM '20).

[26]

Yi Han, Benjamin I. P. Rubinstein, Tamas Abraham, et al. 2018. Reinforcement Learning for Autonomous Defence in Software-Defined Networking. In Decision and Game Theory for Security.

[27]

John Hannay. 2022. CAGE Challenge 2 Winning Submission. PPO Greedy Decoys. https://github.com/john-cardiff/-cyborg-cage-2.

[28]

Yunhan Huang, Linan Huang, and Quanyan Zhu. 2022. Reinforcement Learning for feedback-enabled cyber resilience. Annual Reviews in Control (2022).

[29]

Andrew J. Kerns, Daniel P. Shepard, Jahshan A. Bhatti, and Todd E. Humphreys. 2014. Unmanned Aircraft Capture and Control Via GPS Spoofing. Journal of Field Robotics (2014).

[30]

Thomas Kunz, Christian Fisher, James La Novara-Gsell, Christopher Nguyen, and Li Li. 2023. A Multiagent CyberBattleSim for RL Cyber Operation Agents. In 9th International Conference on Computational Science and Computational Intelligence (CSCI '22).

[31]

Andrew Lohn, Anna Knack, Ant Burke, and Krystal Jackson. 2023. Autonomous Cyber Defence. A roadmap from lab to ops. Online. Centre for Emerging Technology and Security (CETaS) at The Alan Turing Institute.

[32]

Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS'17).

[33]

Mulong Luo, Wenjie Xiong, et al. 2023. AutoCAT: Reinforcement Learning for Automated Exploration of Cache-Timing Attacks. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA).

[34]

Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, et al. 2023. Faster sorting algorithms discovered using deep reinforcement learning. Nature (2023).

[35]

David Bowman Toby JR? icher Junae Kim Maxwell Standen, Martin Lucas and Damian Marriott. 2021. CybORG: A Gym for the Development of Autonomous Cyber Agents. In 1st International Workshop on Adaptive Cyber Defense (IJCAI '21).

[36]

Yassine Mekdad, Ahmet Aris, Leonardo Babun, Abdeslam El Fergougui, Mauro Conti, Riccardo Lazzeretti, and A. Selcuk Uluagac. 2023. A survey on security and privacy issues of UAVs. Computer Networks (2023).

[37]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. 2015. Human-level control through deep reinforcement learning. Nature (2015).

[38]

Thanh Thi Nguyen and Vijay Janapa Reddi. 2021. Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems (2021).

[39]

OpenAI, :, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, and Others. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680

[40]

Julien Perolat, Bart De Vylder, et al. 2022. Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning. Science (2022).

[41]

Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus. 2023. NetHack is Hard to Hack. arXiv:2305.19240

[42]

Theodore Reed, Joseph Geis, and Sven Dietrich. 2011. SkyNET: A 3G-Enabled Mobile Attack Drone and Stealth Botmaster. In 5th USENIXWorkshop on Offensive Technologies ((WOOT '11)).

[43]

Eyal Ronen, Adi Shamir, Achi-OrWeingarten, and Colin O'Flynn. 2017. IoT Goes Nuclear: Creating a ZigBee Chain Reaction. In 2017 IEEE Symposium on Security and Privacy (SP).

[44]

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347

[45]

Mohit Sewak, Sanjay K. Sahay, and Hemant Rathore. 2022. Deep Reinforcement Learning in the Advanced Cybersecurity Threat Detection and Protection. Information Systems Frontiers (2022).

[46]

David Silver, Aja Huang, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature (2016).

[47]

David Silver, Thomas Hubert, Julian Schrittwieser, et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science (2018).

[48]

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. 2019. Individualized Controlled Continuous Communication Model for Multiagent Cooperative and Competitive Tasks. In International Conference on Learning Representations (ICLR '19).

[49]

Wei Song, Xuezixiang Li, Sadia Afroz, Deepali Garg, Dmitry Kuznetsov, and Heng Yin. 2022. MAB-Malware: A Reinforcement Learning Framework for Blackbox Generation of Adversarial Malware. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security (ASIA CCS '22).

Digital Library

[50]

Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction.

Digital Library

[51]

Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems (NeurIPS '99).

Digital Library

[52]

Microsoft Defender Research Team. [n. d.].

[53]

J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, Niall Williams, Yashas Lokesh, and Praveen Ravi. 2021. PettingZoo: Gym for Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS '21).

[54]

U.S. DoD 2022. Securing Defense-Critical Supply Chains. An action plan developed in response to President Biden's Executive Order 14017. Online. U.S. DoD.

[55]

Junia Valente and Alvaro A. Cardenas. 2017. Understanding Security Threats in Consumer Drones Through the Lens of the Discovery Quadcopter Family. In Proceedings of the 2017 Workshop on Internet of Things Security and Privacy (IoTS&P '17).

[56]

Sanyam Vyas, John Hannay, Andrew Bolton, and Pete Burnap. 2023. Automated Cyber Defence: A Review. Proceedings of the ACM on Measurement and Analysis of Computing Systems (2023).

[57]

Salim Al Wahaibi, Myles Foley, and Sergio Maffeis. 2023. SQIRL: Grey-Box Detection of SQL Injection Vulnerabilities Using Reinforcement Learning. In 32nd USENIX Security Symposium (USENIX Security '23).

[58]

Guihong Wang and Jinglun Shi. 2019. Actor-Critic for Multi-agent System with Variable Quantity of Agents. In IoT as a Service.

[59]

Weixun Wang, Tianpei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. 2020. From Fewto More: Large-Scale Dynamic Multiagent Curriculum Learning. Proceedings of the AAAI Conference on Artificial Intelligence (2020).

[60]

Christopher J.C.H. Watkins and Peter Dayan. 1992. Technical Note: Q-Learning. Machine Learning (1992).

[61]

Steven D. Whitehead and Long-Ji Lin. 1995. Reinforcement learning of non- Markov decision processes. Artificial Intelligence (1995).

[62]

Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, and Adrian Webster. 2022. Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies. In Reinforcement Learning for Real Life Workshop in the 36th Conference on Neural Information Processing Systems (RL4RealLife '22).

[63]

Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. 2018. Mean Field Multi-Agent Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning.

[64]

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The Surprising Effectiveness of PPO in Cooperative Multi- Agent Games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS '22).

[65]

Ruotong Yu, Francesca Del Nin, Yuchen Zhang, Shan Huang, Pallavi Kaliyar, Sarah Zakto, Mauro Conti, Georgios Portokalidis, and Jun Xu. 2022. Building Embedded Systems Like It's 1996. In 29th Annual Network and Distributed System Security Symposium (NDSS '22).

[66]

Ce Zhou, Qiben Yan, Yan Shi, and Lichao Sun. 2022. DoubleStar: Long-Range Attack Towards Depth Estimation based Obstacle Avoidance in Autonomous Systems. In 31st USENIX Security Symposium (USENIX Security '22).

Cited By

McFadden SMaugeri MHicks CMavroudis VPierazzi F(2024)WENDIGO: Deep Reinforcement Learning for Denial-of-Service Query Discovery in GraphQL2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00012(68-75)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00012
Goel DMoore KGuo MWang DKim MCamtepe S(2024)Optimizing Cyber Defense in Dynamic Active Directories Through Reinforcement LearningComputer Security – ESORICS 202410.1007/978-3-031-70879-4_17(332-352)Online publication date: 5-Sep-2024
https://doi.org/10.1007/978-3-031-70879-4_17

Index Terms

Canaries and Whistles: Resilient Drone Communication Networks with (or without) Deep Reinforcement Learning
1. Security and privacy

Recommendations

Autonomous Network Defence using Reinforcement Learning
ASIA CCS '22: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security

In the network security arms race, the defender is significantly disadvantaged as they need to successfully detect and counter every malicious attack. In contrast, the attacker needs to succeed only once. To level the playing field, we investigate the ...
Reward Shaping in Episodic Reinforcement Learning
AAMAS '17: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems

Recent advancements in reinforcement learning confirm that reinforcement learning techniques can solve large scale problems leading to high quality autonomous decision making. It is a matter of time until we will see large scale applications of ...
Conversational Recommender System Using Deep Reinforcement Learning
RecSys '22: Proceedings of the 16th ACM Conference on Recommender Systems

Deep Reinforcement Learning (DRL) uses the best of both Reinforcement Learning and Deep Learning for solving problems which cannot be addressed by them individually. Deep Reinforcement Learning has been used widely for games, robotics etc. Limited work ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AISec '23: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security

November 2023

252 pages

ISBN:9798400702600

DOI:10.1145/3605764

Program Chairs:
Maura Pintor
University of Cagliari, Italy
,
Xinyun Chen
Google Brain, USA
,
Florian Tramèr
ETH Zürich, Switzerland

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 November 2023

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research funded by the Defence Science and Technology Laboratory (Dstl).

Conference

CCS '23

Sponsor:

SIGSAC

CCS '23: ACM SIGSAC Conference on Computer and Communications Security

November 30, 2023

Copenhagen, Denmark

Acceptance Rates

Overall Acceptance Rate 94 of 231 submissions, 41%

Upcoming Conference

CCS '24

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 14 - 18, 2024

Salt Lake City , UT , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
451
Total Downloads

Downloads (Last 12 months)451
Downloads (Last 6 weeks)39

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

McFadden SMaugeri MHicks CMavroudis VPierazzi F(2024)WENDIGO: Deep Reinforcement Learning for Denial-of-Service Query Discovery in GraphQL2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00012(68-75)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00012
Goel DMoore KGuo MWang DKim MCamtepe S(2024)Optimizing Cyber Defense in Dynamic Active Directories Through Reinforcement LearningComputer Security – ESORICS 202410.1007/978-3-031-70879-4_17(332-352)Online publication date: 5-Sep-2024
https://doi.org/10.1007/978-3-031-70879-4_17

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents