Neuroevolution for Autonomous Cyber Defense
Author: 
Kade HeckelAuthors Info & Claims
Kade HeckelAuthors Info & ClaimsAbstract
This work presents the preliminary results of and discusses current challenges in ongoing research of neuroevolution for the task of evolving agents for autonomous cyber operations (ACO). The application of reinforcement learning to the cyber domain is especially challenging due to the extremely limited observability of the environment over extended time frames where an adversary can potentially take many actions without being detected. To promote research within this space The Technical Cooperation Program (TTCP), which is an international collaboration organization between the US, UK, Canada, Australia, and New Zealand, released the Cyber Operations Research Gym (CybORG) to enable experimentation with RL algorithms in both simulated and emulated environments. Using competition to spur investigation and innovation, TTCP has released the CAGE Challenges which for evaluating RL in network defense.[1] This work evolves agents for ACO using the python-based neuroevolution library Evosax[2] which supports high performance, GPU accelerated evolutionary algorithms for the purpose of optimizing artificial neural network parameters. The use of neuroevolution in this paper is a first for the ACO task and benchmarks two popular algorithms to identify factors which impact their effectiveness.
References
[1]
CybORG: A Gym for the Development of Autonomous Cyber Agents. arXiv, 2021.
[2]
Robert Tjarko Lange. evosax: Jax-based evolution strategies. arXiv preprint arXiv:2212.04180, 2022.
[3]
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/1712.06567, 2017.
[4]
George Rush, Daniel R. Tauritz, and Alexander D. Kent. Coevolutionary agent-based network defense lightweight event system (candles). In Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO Companion '15, page 859--866, New York, NY, USA, 2015. Association for Computing Machinery.
[5]
Erb Lugo and Anthony. Coevolutionary genetic algorithms for proactive computer network defenses. 2017.
[6]
Erik Hemberg, Joseph R. Zipkin, Richard W. Skowyra, Neal Wagner, and Una-May O'Reilly. Adversarial co-evolution of attack and defense in a segmented computer network environment. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO '18, page 1648--1655, New York, NY, USA, 2018. Association for Computing Machinery.
[7]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.
[8]
John Hannay. John-cardiff/-cyborg-cage-2, Jul 2022.
[9]
Hans-Georg Beyer and Bernhard Sendhoff. Simplify your covariance matrix adaptation evolution strategy. IEEE Transactions on Evolutionary Computation, 21(5):746--759, 2017.
[10]
Ilya Loshchilov, Tobias Glasmachers, and Hans-Georg Beyer. Limited-memory matrix adaptation for large scale black-box optimization. CoRR, abs/1705.06693, 2017.
[11]
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551--559, 2010. The 18th International Conference on Artificial Neural Networks, ICANN 2008.
[12]
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Goldberg, and Ion Stoica. Ray rllib: A composable and scalable reinforcement learning library. CoRR, abs/1712.09381, 2017.
[13]
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[14]
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. CoRR, abs/1712.05889, 2017.
[15]
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1--8, 2021.
[16]
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067--1074, 2022.
[17]
Evan Caville, Wai Weng Lo, Siamak Layeghy, and Marius Portmann. Anomal-e: A self-supervised network intrusion detection system based on graph neural networks. Knowledge-Based Systems, 258:110030, Dec 2022.
[18]
TTCP CAGE Working Group. Ttcp cage challenge 3. https://github.com/cage-challenge/cage-challenge-3, 2022.
[19]
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. CoRR, abs/1504.04909, 2015.
[20]
Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 81--89. ACM, 2019.
Index Terms
- Neuroevolution for Autonomous Cyber Defense
Recommendations
Multiagent learning through neuroevolution
WCCI'12: Proceedings of the 2012 World Congress conference on Advances in Computational IntelligenceNeuroevolution is a promising approach for constructing intelligent agents in many complex tasks such as games, robotics, and decision making. It is also well suited for evolving team behavior for many multiagent tasks. However, new challenges and ...
A Tale of Three Cyber-Defense Workshops
The National Cyber Defense Initiative (NCDI) has been working behind the scenes to help inform the US research agenda for strategic cyber defense. An important part of the NDCI's activities has been sponsorship of three workshops: the 2006 Safe-...
Optimizing Active Cyber Defense
GameSec 2013: 4th International Conference on Decision and Game Theory for Security - Volume 8252Active cyber defense is one important defensive method for combating cyber attacks. Unlike traditional defensive methods such as firewall-based filtering and anti-malware tools, active cyber defense is based on spreading "white" or "benign" worms to ...
Comments
Information & Contributors
Information
Published In
July 2023
2519 pages
Copyright © 2023 Owner/Author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 July 2023
Check for updates
Author Tags
Qualifiers
- Poster
Conference
GECCO '23 Companion
Sponsor:
GECCO '23 Companion: Companion Conference on Genetic and Evolutionary Computation
July 15 - 19, 2023
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 112Total Downloads
- Downloads (Last 12 months)86
- Downloads (Last 6 weeks)1
Reflects downloads up to 24 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in