poster

Autonomous Network Defence using Reinforcement Learning

Authors:

Vasilios MavroudisAuthors Info & Claims

ASIA CCS '22: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security

Pages 1252 - 1254

https://doi.org/10.1145/3488932.3527286

Published: 30 May 2022 Publication History

Abstract

In the network security arms race, the defender is significantly disadvantaged as they need to successfully detect and counter every malicious attack. In contrast, the attacker needs to succeed only once. To level the playing field, we investigate the effectiveness of autonomous agents in a realistic network defence scenario. We first outline the problem, provide the background on reinforcement learning and detail our proposed agent design. Using a network environment simulation, with 13 hosts spanning 3 subnets, we train a novel reinforcement learning agent and show that it can reliably defend continual attacks by two advanced persistent threat (APT) red agents: one with complete knowledge of the network layout and another which must discover resources through exploration but is more general.

Supplementary Material

MP4 File (CCS_CAGE_POSTER.mp4)

In this video we present our hierarchical reinforcement learning architecture for protecting computer networks from at least two unique classes of advanced persistent threat adversary. Our defensive capability is trained within the newly launched Cyber Autonomy Gym for Experimentation (CAGE) environment which provides realistic network defence scenarios, including autonomous adversaries, based on the MITRE ATT&CK and OASIS OpenC2 frameworks. Our solution allows specialised autonomous agents, which we train using Proximal Policy Optimisation (PPO) enhanced with curiosity, to be deployed at the discretion of a controller agent who is also trained in the CAGE environment. Our results support that our hierarchical approach substantially outperforms either of the specialists and may provide a more generalised and extensible defensive capability for securing computer networks autonomously.

Download
18.17 MB

References

[1]

CAGE. 2021. CAGE Challenge 1. In IJCAI-21 1st International Workshop on Adaptive Cyber Defense.

[2]

L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. 2018. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. arXiv:1802.01561 [cs].

[3]

OpenAI et al. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680 [cs, stat].

[4]

M. Feng and H. Xu. 2017. Deep reinforecement learning based optimal defense for cyber-physical system in presence of unknown cyber-attack. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE.

[5]

D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. 2018. Distributed Prioritized Experience Replay. In arXiv:1803.00933 [cs].

[6]

Z. Hu, R. Beuran, and Y. Tan. 2020. Automated Penetration Testing Using Deep Reinforcement Learning. In 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&P W).

[7]

FireEye Inc. 2021. M-Trends 2021: Cyber Security Insights. Technical Report. https://vision.fireeye.com/content/fireeye-vision/en_US/editions/11/11-m-trends.html

[8]

E. Liang, R. Liaw, P. Moritz, R. Nishihara, R. Fox, K. Goldberg, J E. Gonzalez, M I. Jordan, and I. Stoica. 2018. RLlib: Abstractions for Distributed Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning (ICML'18).

[9]

T T. Nguyen and V J. Reddi. 2021. Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems.

[10]

D. Pathak, P. Agrawal, Alexei A. Efros, and T. Darrell. 2017. Curiosity-Driven Exploration by Self-Supervised Prediction. In Proceedings of the 34th International Conference on Machine Learning (ICML'17).

[11]

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. Proximal Policy Optimization Algorithms. In arXiv:1707.06347 [cs].

[12]

P. Speicher, M. Steinmetz, J. Hoffmann, M. Backes, and R. Kunnemann. 2019. Towards automated network mitigation analysis. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (SAC '19).

[13]

M. Standen, D. Bowman, S. Hoang, T. Richer, M. Lucas, and R. Van Tassel. 2021 a. Cyber Autonomy Gym for Experimentation Challenge 1. https://github.com/cage-challenge/cage-challenge-1.

[14]

M. Standen, M. Lucas, David B., T J. Richer, J. Kim, and D. Marriott. 2021 b. CybORG: A Gym for the Development of Autonomous Cyber Agents. In IJCAI-21 1st International Workshop on Adaptive Cyber Defense.

[15]

R S. Sutton and A G. Barto. 2018. Reinforcement Learning: An Introduction 2nd ed.).

Cited By

Farooq MKunz T(2024)A Generic Blue Agent Training Framework for Autonomous Cyber Operations2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619771(515-521)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619771
Vyas SHicks CMavroudis V(2024)Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00013(76-86)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00013
McFadden SMaugeri MHicks CMavroudis VPierazzi F(2024)WENDIGO: Deep Reinforcement Learning for Denial-of-Service Query Discovery in GraphQL2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00012(68-75)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00012
Show More Cited By

Index Terms

Autonomous Network Defence using Reinforcement Learning
1. Security and privacy
  1. Network security

Recommendations

Bridging Automated to Autonomous Cyber Defense: Foundational Analysis of Tabular Q-Learning
AISec'22: Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security

Leveraging security automation and orchestration technologies enables security analysts to respond more quickly and accurately to threats. However, current tooling is limited to automating very finely scoped and hand-coded situations, such as ...
Reinforcement Learning for Autonomous Defence in Software-Defined Networking
Decision and Game Theory for Security
Abstract
Despite the successful application of machine learning (ML) in a wide range of domains, adaptability—the very property that makes machine learning desirable—can be exploited by adversaries to contaminate training and evade classification. In this ...
Reinforcement learning-based autonomous attacker to uncover computer network vulnerabilities
Abstract
In today’s intricate information technology landscape, the escalating complexity of computer networks is accompanied by a myriad of malicious threats seeking to compromise network components. To address these security challenges, we propose an ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASIA CCS '22: Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security

May 2022

1291 pages

ISBN:9781450391405

DOI:10.1145/3488932

General Chairs:
Yuji Suga
Internet Initiative Japan Inc., Japan
,
Kouichi Sakurai
Kyushu University, Japan
,
Program Chairs:
Xuhua Ding
Singapore Management University, Singapore
,
Kazue Sako
Waseda University, Japan

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 May 2022

Check for updates

Author Tags

Qualifiers

Poster

Funding Sources

The Defence and Security Programme at The Alan Turing Institute funded by the Government Communications Headquarters (GCHQ).

Conference

ASIA CCS '22

Sponsor:

SIGSAC

ASIA CCS '22: ACM Asia Conference on Computer and Communications Security

May 30 - June 3, 2022

Nagasaki, Japan

Acceptance Rates

Overall Acceptance Rate 418 of 2,322 submissions, 18%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
820
Total Downloads

Downloads (Last 12 months)330
Downloads (Last 6 weeks)12

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Farooq MKunz T(2024)A Generic Blue Agent Training Framework for Autonomous Cyber Operations2024 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking62109.2024.10619771(515-521)Online publication date: 3-Jun-2024
https://doi.org/10.23919/IFIPNetworking62109.2024.10619771
Vyas SHicks CMavroudis V(2024)Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00013(76-86)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00013
McFadden SMaugeri MHicks CMavroudis VPierazzi F(2024)WENDIGO: Deep Reinforcement Learning for Denial-of-Service Query Discovery in GraphQL2024 IEEE Security and Privacy Workshops (SPW)10.1109/SPW63631.2024.00012(68-75)Online publication date: 23-May-2024
https://doi.org/10.1109/SPW63631.2024.00012
Zhang LBurbano LChen XCardenas ADrager SAnderson MKong F(2024)Fast Attack Recovery for Stochastic Cyber-Physical Systems2024 IEEE 30th Real-Time and Embedded Technology and Applications Symposium (RTAS)10.1109/RTAS61025.2024.00030(280-293)Online publication date: 13-May-2024
https://doi.org/10.1109/RTAS61025.2024.00030
Loevenich JAdler EMercier RVelazquez ALopes R(2024)Design of an Autonomous Cyber Defence Agent using Hybrid AI models2024 International Conference on Military Communication and Information Systems (ICMCIS)10.1109/ICMCIS61231.2024.10540988(1-10)Online publication date: 23-Apr-2024
https://doi.org/10.1109/ICMCIS61231.2024.10540988
Mcdonald GLi LMallah R(2024)Finding the Optimal Security Policies for Autonomous Cyber Operations With Competitive Reinforcement LearningIEEE Access10.1109/ACCESS.2024.344631012(120292-120305)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3446310
Mohamed Ahmed ANguyen TAbdelrazek MAryal S(2024)Reinforcement learning-based autonomous attacker to uncover computer network vulnerabilitiesNeural Computing and Applications10.1007/s00521-024-09668-036:23(14341-14360)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00521-024-09668-0
Faillon MBout BFrancq JNeal CBoulahia-Cuppens NCuppens FYaich R(2024)How to Better Fit Reinforcement Learning for Pentesting: A New Hierarchical ApproachComputer Security – ESORICS 202410.1007/978-3-031-70903-6_16(313-332)Online publication date: 5-Sep-2024
https://doi.org/10.1007/978-3-031-70903-6_16
Goel DMoore KGuo MWang DKim MCamtepe S(2024)Optimizing Cyber Defense in Dynamic Active Directories Through Reinforcement LearningComputer Security – ESORICS 202410.1007/978-3-031-70879-4_17(332-352)Online publication date: 5-Sep-2024
https://doi.org/10.1007/978-3-031-70879-4_17
Nankya MChataut RAkl R(2023)Securing Industrial Control Systems: Components, Cyber Threats, and Machine Learning-Driven Defense StrategiesSensors10.3390/s2321884023:21(8840)Online publication date: 30-Oct-2023
https://doi.org/10.3390/s23218840
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents