research-article

Reinforcement Learning Algorithms for Adaptive Cyber Defense against Heartbleed

Authors:

Peng LiuAuthors Info & Claims

MTD '14: Proceedings of the First ACM Workshop on Moving Target Defense

Pages 51 - 58

https://doi.org/10.1145/2663474.2663481

Published: 03 November 2014 Publication History

Abstract

In this paper, we investigate a model where a defender and an attacker simultaneously and repeatedly adjust the defenses and attacks. Under this model, we propose two iterative reinforcement learning algorithms which allow the defender to identify optimal defenses when the information about the attacker is limited. With probability one, the adaptive reinforcement learning algorithm converges to the best response with respect to the attacks when the attacker diminishingly explores the system. With a probability arbitrarily close to one, the robust reinforcement learning algorithm converges to the min-max strategy despite that the attacker persistently explores the system. The algorithm convergence is formally proven and the algorithm performance is verified via numerical simulations.

References

[1]

http://heartbleed.com/.

[2]

T. Alpcan and T. Basar. Network Security: A Decision and Game Theoretic Approach. Cambridge University Press, 2011.

Digital Library

[3]

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235--256, 2002.

Digital Library

[4]

D.P. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.

Digital Library

[5]

R. Boome. Security metrics and security investment models. In Proceedings of the 5th International Conference on Advances in Information and Computer Security, pages 10--24, 2010.

Digital Library

[6]

R. Boome and T. Moore. The iterated weakest link - A model of adaptive security investment. In Workshop on Economics of Information Security, pages 2406--2411, 2009.

[7]

S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1--122, 2012.

[8]

L. Demetz and D. Bachlechner. To invest or not to invest? Assessing the economic viability of a policy and security configuration management tool. The Economics of Information Security and Privacy, pages 25--47, 2013.

[9]

M. Freidlin and A. Wentzell. Random perturbations of dynamical systems. New York: Springer Verlag, 1984.

[10]

S. Jajodia, A. Ghosh, V. Swarup, C. Wang, and X. Wang. Moving Target Defense: Creating Asymmetric Uncertainty for Cyber Threats. Springer, 2011.

Digital Library

[11]

S. Jajodia, A. Ghosh, V. Swarup, C. Wang, and X. Wang. Moving Target Defense II: Application of Game Theory and Adversarial Modeling. Springer, 2013.

Digital Library

[12]

J. Lin, P. Liu, and J. Jing. Using signaling games to model the multi-step attack-defense scenarios on confidentiality. In GameSec, pages 118--137, 2012.

[13]

P. Liu, W. Zang, and M. Yu. Incentive-based modeling and inference of attacker intent, objectives, and strategies. ACM Transactions on Information and System Security, 8(1):1094--9224, 2005.

Digital Library

[14]

Y. Luo, F. Szidarovszky, Y. Al-Nashif, and S. Hariri. Game theory based network security. Journal of Information Security, 1(1):41--44, 2010.

[15]

K. Lye and J. Wing. Game strategies in network security. International Journal of Information Security, 4(1):71--86, 2005.

Digital Library

[16]

M.H. Manshaei, Q. Zhu, T. Alpcan, T. Basar, and J.P. Hubaux. Game theory meets network security and privacy. ACM Computing Surveys, 45(3):25--39, 2013.

Digital Library

[17]

J.R. Marden, H.P. Young, G. Arslan, and J.S. Shamma. Payo based dynamics for multi-player weakly acyclic games. 48(1):373--396, February 2009.

Digital Library

[18]

H. Okhravi, M. Rabe, T. Mayberry, W. Leonard, T. Hobson, D. Bigelow, and W. Streilein. Survey of cyber moving target techniques. Technical report, Lincoln Lab, Massachusetts Institute of Technology, 2013.

[19]

S. Roy, C. Ellis, S. Shiva, D. Dasgupta, V. Shandilya, and Q. Wu. A survey of game theory as applied to network security. pages 1{10, Hawaii, USA, 2010.

[20]

R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.

Digital Library

[21]

G. Theodorakopoulos and J. Baras. Game theoretic modeling of malicious users in collaborative networks. IEEE Journal on Selected Areas in Communications, 26(7):1317--1327, 2008.

Digital Library

[22]

H.P. Young. The evolution of conventions. Econometrica, 61:57--84, January 1993.

[23]

M. Zhu and S. Martínez. Distributed coverage games for mobile visual sensors (I): Reaching the set of Nash equilibria. In IEEE International Conference on Decision and Control, pages 169--174, Shanghai, China, Dec 2009.

[24]

M. Zhu and S. Martínez. Distributed coverage games for mobile visual sensors (II): Reaching the set of global optima. In IEEE International Conference on Decision and Control, pages 175--180, Shanghai, China, Dec 2009.

[25]

M. Zhu and S. Martínez. Distributed coverage games for energy-aware mobile sensor networks. SIAM Journal on Control and Optimization, 51(1):1--27, 2013.

[26]

Q. Zhu, H. Tembine, and T. Basar. Hybrid learning in stochastic games and its applications in network security. Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, pages 305--329, 2013.

Cited By

Hammar KStadler R(2024)Learning Near-Optimal Intrusion Responses Against Dynamic AttackersIEEE Transactions on Network and Service Management10.1109/TNSM.2023.329341321:1(1158-1177)Online publication date: Mar-2024
https://doi.org/10.1109/TNSM.2023.3293413
Mohamed Ahmed ANguyen TAbdelrazek MAryal S(2024)Reinforcement learning-based autonomous attacker to uncover computer network vulnerabilitiesNeural Computing and Applications10.1007/s00521-024-09668-036:23(14341-14360)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00521-024-09668-0
Sun RZhu YFei JChen X(2023)A Survey on Moving Target Defense: Intelligently Affordable, Optimized and Self-AdaptiveApplied Sciences10.3390/app1309536713:9(5367)Online publication date: 25-Apr-2023
https://doi.org/10.3390/app13095367
Show More Cited By

Index Terms

Reinforcement Learning Algorithms for Adaptive Cyber Defense against Heartbleed
1. Security and privacy
2. Social and professional topics
  1. Computing / technology policy
    1. Computer crime
  2. Professional topics
    1. Management of computing and information systems
      1. Information system economics

Recommendations

Reinforcement Learning for Adaptive Cyber Defense Against Zero-Day Attacks
Adversarial and Uncertain Reasoning for Adaptive Cyber Defense
Abstract
In this chapter, we leverage reinforcement learning as a unified framework to design effective adaptive cyber defenses against zero-day attacks. Reinforcement learning is an integration of control theory and machine learning. A salient feature of ...
Reinforcement Learning for Adaptive Cyber Defense: Algorithms, Analysis and Experiments
Deep Reinforcement Learning for Adaptive Cyber Defense in Network Security
AICCONF '24: Proceedings of the Cognitive Models and Artificial Intelligence Conference

In the labyrinthine world of cybersecurity, the ever-evolving specter of cyber-attacks offers an inevitable challenge to the fortifications of protection measures. Past investigations have underlined the exigency for adaptive and aggressive strategies in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MTD '14: Proceedings of the First ACM Workshop on Moving Target Defense

November 2014

116 pages

ISBN:9781450331500

DOI:10.1145/2663474

Program Chairs:
Sushil Jajodia
George Mason University
,
Kun Sun
College of William and Mary

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CCS'14

Sponsor:

SIGSAC

CCS'14: 2014 ACM SIGSAC Conference on Computer and Communications Security

November 7, 2014

Arizona, Scottsdale, USA

Acceptance Rates

MTD '14 Paper Acceptance Rate 9 of 16 submissions, 56%;

Overall Acceptance Rate 40 of 92 submissions, 43%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
997
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)9

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hammar KStadler R(2024)Learning Near-Optimal Intrusion Responses Against Dynamic AttackersIEEE Transactions on Network and Service Management10.1109/TNSM.2023.329341321:1(1158-1177)Online publication date: Mar-2024
https://doi.org/10.1109/TNSM.2023.3293413
Mohamed Ahmed ANguyen TAbdelrazek MAryal S(2024)Reinforcement learning-based autonomous attacker to uncover computer network vulnerabilitiesNeural Computing and Applications10.1007/s00521-024-09668-036:23(14341-14360)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s00521-024-09668-0
Sun RZhu YFei JChen X(2023)A Survey on Moving Target Defense: Intelligently Affordable, Optimized and Self-AdaptiveApplied Sciences10.3390/app1309536713:9(5367)Online publication date: 25-Apr-2023
https://doi.org/10.3390/app13095367
Nguyen TReddi V(2023)Deep Reinforcement Learning for Cyber SecurityIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.312187034:8(3779-3795)Online publication date: Aug-2023
https://doi.org/10.1109/TNNLS.2021.3121870
Dekel LLeybovich IZilberman PPuzis R(2023)MABAT: A Multi-Armed Bandit Approach for Threat-HuntingIEEE Transactions on Information Forensics and Security10.1109/TIFS.2022.321501018(477-490)Online publication date: 2023
https://doi.org/10.1109/TIFS.2022.3215010
Xu SXie ZZhu CWang XShi L(2023)Enhancing Cybersecurity in Industrial Control System with Autonomous Defense Using Normalized Proximal Policy Optimization Model2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00138(928-935)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00138
Malik MSaini K(2023)Network Intrusion Detection System Using Reinforcement Learning Techniques2023 International Conference on Circuit Power and Computing Technologies (ICCPCT)10.1109/ICCPCT58313.2023.10245608(1642-1649)Online publication date: 10-Aug-2023
https://doi.org/10.1109/ICCPCT58313.2023.10245608
Liu R(2023)Towards an Uncertainty-aware Decision Engine for Proactive Self-Protecting Software2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C)10.1109/ACSOS-C58168.2023.00027(21-23)Online publication date: 25-Sep-2023
https://doi.org/10.1109/ACSOS-C58168.2023.00027
Zhang QCho JMoore TKim DLim HNelson F(2023)EVADE: Efficient Moving Target Defense for Autonomous Network Topology Shuffling Using Deep Reinforcement LearningApplied Cryptography and Network Security10.1007/978-3-031-33488-7_21(555-582)Online publication date: 29-May-2023
https://doi.org/10.1007/978-3-031-33488-7_21
Hammar KStadler R(2022)An Online Framework for Adapting Security Policies in Dynamic IT EnvironmentsProceedings of the 18th International Conference on Network and Service Management10.5555/3581644.3581696(1-5)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.5555/3581644.3581696
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents