Towards a High Fidelity Training Environment for Autonomous Cyber Defense Agents
Authors: 
Sean Oesch, Amul Chaulagain, Brian Weber, Matthew Dixson, Amir Sadovnik, Benjamin Roberson, Cory Watson, 
Phillipe AustriaAuthors Info & Claims
Sean Oesch, Amul Chaulagain, Brian Weber, Matthew Dixson, Amir Sadovnik, Benjamin Roberson, Cory Watson, Phillipe AustriaAuthors Info & ClaimsPages 91 - 99
Abstract
Cyber defenders are overwhelmed by the frequency and scale of attacks against their networks. This problem will only be exacerbated as attackers leverage AI to automate their workflows. Autonomous cyber defense capabilities could aid defenders by automating operations and adapting dynamically to novel threats. However, existing training environments fall short in areas such as generalization, explainability, scalability, and transferability, making it intractable to train agents that will be effective in real networks. In this paper we take an important step towards creating autonomous cyber defense agents — we present a high fidelity training environment called Cyberwheel that includes both simulation and emulation capabilities. Cyberwheel simplifies customization of the training network and easily allows redefining the agent’s reward function, observation space, and action space to support rapid experimentation of novel approaches to agent design. It also provides visibility into agent behaviors necessary for agent evaluation and sufficient documentation / examples to lower the barrier to entry. As an example use case of Cyberwheel, we present initial results training an autonomous agent to deploy cyber deception strategies in simulation.
References
[1]
Elizabeth Bates, Vasilios Mavroudis, and Chris Hicks. 2023. Reward Shaping for Happier Autonomous Cyber Security Agents. arXiv preprint arXiv:2310.13565 (2023).
[2]
CAGE (Ed.). 2022. TTCP CAGE Challenge 2.
[3]
Ashutosh Dutta, Samrat Chatterjee, Arnab Bhattacharya, and Mahantesh Halappanavar. 2023. Deep Reinforcement Learning for Cyber System Defense under Dynamic Adversarial Uncertainties. arXiv preprint arXiv:2302.01595 (2023).
[4]
Kasimir Georg Gabert, Adam Vail, Tan Q. Thai, Ian Burns, Michael J. McDonald, Steven Elliott, John Vivian Montoya, Jenna Marie Kallaher, and Stephen T. Jones. 2015. Firewheel - A Platform for Cyber Analysis. (11 2015). https://www.osti.gov/biblio/1333803
[5]
Kim Hammar and Rolf Stadler. 2020. Finding effective security strategies through reinforcement learning and self-play. In 2020 16th International Conference on Network and Service Management (CNSM). IEEE, 1–9.
[6]
Kim Hammar and Rolf Stadler. 2022. Learning Security Strategies through Game Play and Optimal Stopping. arXiv preprint arXiv:2205.14694 (2022).
[7]
Zhenguo Hu, Razvan Beuran, and Yasuo Tan. 2020. Automated penetration testing using deep reinforcement learning. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2–10.
[8]
Jaromír Janisch, Tomáš Pevnỳ, and Viliam Lisỳ. 2023. NASimEmu: Network Attack Simulator & Emulator for Training Agents Generalizing to Novel Scenarios. arXiv preprint arXiv:2305.17246 (2023).
[9]
Michael Kouremetis, Dean Lawrence, Ron Alford, Zoe Cheuvront, David Davila, Benjamin Geyer, Trevor Haigh, Ethan Michalak, Rachel Murphy, and Gianpaolo Russo. 2024. Mirage: cyber deception against autonomous cyber attacks in emulation and simulation. Annals of Telecommunications (2024), 1–15.
[10]
Microsoft Defender Research Team. 2021. CyberBattleSim. Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei (2021).
[11]
Andres Molina-Markham, Cory Miniter, Becky Powell, and Ahmad Ridley. 2021. Network environment design for autonomous cyberdefense. arXiv preprint arXiv:2103.07583 (2021).
[12]
Jakob Nyberg and Pontus Johnson. 2023. Training Automated Defense Strategies Using Graph-based Cyber Attack Simulations. arXiv preprint arXiv:2304.11084 (2023).
[13]
Sean Oesch, Phillipe Austria, Amul Chaulagain, Brian Weber, Cory Watson, Matthew Dixson, and Amir Sadovnik. 2024. The Path To Autonomous Cyber Defense. arXiv preprint arXiv:2404.10788 (2024).
[14]
Ahmad Ridley. 2018. Machine learning for autonomous cyber defense. The Next Wave 22, 1 (2018), 7–14.
[15]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
[16]
Jonathon Schwartz and Hanna Kurniawati. 2019. Autonomous penetration testing using reinforcement learning. arXiv preprint arXiv:1905.05965 (2019).
[17]
Maxwell Standen. 2022. Cyber Autonomy Gym for Experimentation Challenge 2. https://github.com/cage-challenge/cage-challenge-2. Created by Maxwell Standen, David Bowman, Son Hoang, Toby Richer, Martin Lucas, Richard Van Tassel, Phillip Vu, Mitchell Kiely.
[18]
Maxwell Standen, Martin Lucas, David Bowman, Toby J Richer, Junae Kim, and Damian Marriott. 2021. Cyborg: A gym for the development of autonomous cyber agents. arXiv preprint arXiv:2108.09118 (2021).
[19]
Sanyam Vyas, John Hannay, Andrew Bolton, and Professor Pete Burnap. 2023. Automated Cyber Defence: A Review. arXiv preprint arXiv:2303.04926 (2023).
[20]
Erich Walter, Kimberly Ferguson-Walter, and Ahmad Ridley. 2021. Incorporating Deception into CyberBattleSim for Autonomous Defense. arXiv preprint arXiv:2108.13980 (2021).
[21]
Melody Wolk, Andy Applebaum, Camron Denver, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, 2022. Beyond cage: Investigating generalization of learned autonomous network defense policies. arXiv preprint arXiv:2211.15557 (2022).
Recommendations
Use of cyber attack and defense agents in cyber ranges: A case study
AbstractWith the ever-changing cybersecurity landscape, the need for a continuous training for new cybersecurity skill sets is a requirement. Such continuous training programs can be delivered on platforms like cyber ranges. Cyber ranges ...
Unified Emulation-Simulation Training Environment for Autonomous Cyber Agents
Machine Learning for NetworkingAbstractAutonomous cyber agents may be developed by applying reinforcement and deep reinforcement learning (RL/DRL), where agents are trained in a representative environment. The training environment must simulate with high-fidelity the network Cyber ...
Optimizing Active Cyber Defense
GameSec 2013: 4th International Conference on Decision and Game Theory for Security - Volume 8252Active cyber defense is one important defensive method for combating cyber attacks. Unlike traditional defensive methods such as firewall-based filtering and anti-malware tools, active cyber defense is based on spreading "white" or "benign" worms to ...
Comments
Information & Contributors
Information
Published In
August 2024
115 pages
ISBN:9798400709579
DOI:10.1145/3675741
Copyright © 2024 Public Domain.
This paper is authored by an employee(s) of the United States Government and is in the public domain. Non-exclusive copying or redistribution is allowed, provided that the article citation is given and the authors and agency are clearly identified as its source.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 August 2024
Check for updates
Author Tag
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
CSET 2024
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 180Total Downloads
- Downloads (Last 12 months)180
- Downloads (Last 6 weeks)153
Reflects downloads up to 24 Sep 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in