research-article

Public Access

Detecting Stealthy Botnets in a Resource-Constrained Environment using Reinforcement Learning

Authors:

Sridhar Venkatesan,

Massimiliano Albanese,

Rajesh Ganesan,

Sushil JajodiaAuthors Info & Claims

MTD '17: Proceedings of the 2017 Workshop on Moving Target Defense

Pages 75 - 85

https://doi.org/10.1145/3140549.3140552

Published: 30 October 2017 Publication History

Abstract

Modern botnets can persist in networked systems for extended periods of time by operating in a stealthy manner. Despite the progress made in the area of botnet prevention, detection, and mitigation, stealthy botnets continue to pose a significant risk to enterprises. Furthermore, existing enterprise-scale solutions require significant resources to operate effectively, thus they are not practical. In order to address this important problem in a resource-constrained environment, we propose a reinforcement learning based approach to optimally and dynamically deploy a limited number of defensive mechanisms, namely honeypots and network-based detectors, within the target network. The ultimate goal of the proposed approach is to reduce the lifetime of stealthy botnets by maximizing the number of bots identified and taken down through a sequential decision-making process. We provide a proof-of-concept of the proposed approach, and study its performance in a simulated environment. The results show that the proposed approach is promising in protecting against stealthy botnets.

References

[1]

Tansu Alpcan and Tamer Başar. 2006. An Intrusion Detection Game with Limited Observations. In Proceedings of the 12th International Symposium on Dynamic Games and Applications. Sophia-Antipolis, France.

[2]

Michael Bailey, Evan Cooke, Farnam Jahanian, Yunjing Xu, and Manish Karir. 2009. A Survey of Botnet Technology and Defenses. In Proceedings of the Cybersecurity Applications & Technology Conference for Homeland Security (CATCH 2009). 299--304.

Digital Library

[3]

Elaheh Biglar Beigi, Hossein Hadian Jazi, Natalia Stakhanova, and Ali A. Ghorbani. 2014. Towards Effective Feature Selection in Machine Learning-Based Botnet Detection Approaches. In Proceedings of the IEEE Conference on Communications and Network Security (CNS 2014). IEEE, San Francisco, CA, USA, 247--255.

[4]

Richard Bellman. 1957. Dynamic Programming. Princeton University Press, Princeton, NJ, USA.

[5]

Rajesh Ganesan, Sushil Jajodia, Ankit Shah, and Hasan Cam. 2016. Dynamic Scheduling of Cybersecurity Analysts for Minimizing Risk Using Reinforcement Learning. ACM Transactions on Intelligent Systems and Technology, Vol. 8, 1 (2016).

Digital Library

[6]

Abhijit Gosavi. 2003. Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning. Springer.

[7]

Guofei Gu, Roberto Perdisci, Junjie Zhang, and Wenke Lee. 2008. BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-Independent Botnet Detection. In Proceedings of the 17th USENIX Security Symposium. San Jose, CA, USA, 139--154.

Digital Library

[8]

Guofei Gu, Phillip Porras, Vinod Yegneswaran, Martin Fong, and Wenke Lee. 2007. BotHunter: Detecting Malware Infection Through IDS-Driven Dialog Correlation. In Proceedings of the 16th USENIX Security Symposium. Boston, MA, USA, 167--182.

Digital Library

[9]

Guofei Gu, Junjie Zhang, and Wenke Lee. 2008. BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic. In Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS 2008). San Diego, CA, USA.

[10]

Kaspersky Lab. 2014. The Regin Platform Nation-State Ownage of GSM Networks. Technical Report. Kaspersky Lab.

[11]

Karim Khalil, Zhiyun Qian, Paul Yu, Srikanth Krishnamurthy, and Ananthram Swam. 2016. Optimal Monitor Placement for Detection of Persistent Threats. In Proceedings of the IEEE Global Communications Conference (IEEE GLOBECOM 2016). IEEE, Washington, DC USA.

[12]

Marion Marschalek, Paul Kimayong, and Fengmin Gong. 2014. POS Malware Revisited: Look What We Found Inside Your Cashdesk. Technical Report. Cyphort Labs.

[13]

Alberto Montresor and Márk Jelasity. 2009. PeerSim: A Scalable P2P Simulator. In Proceedings of the 9th IEEE International Conference on Peer-to-Peer Computing (P2P 2009). Seattle, WA, USA, 99--100.

[14]

Juliana M. Nascimento and Warren B. Powell. 2009. An Optimal Approximate Dynamic Programming Algorithm for the Lagged Asset Acquisition Problem. Mathematics of Operations Research Vol. 34, 1 (February 2009), 210--237.

Digital Library

[15]

Warren B. Powell. 2011. Approximate Dynamic Programming: Solving the Curses of Dimensionality (2nd ed.). John Wiley & Sons.

[16]

Christian Rossow, Dennis Andriesse, Tillmann Werner, Brett Stone-Gross, Daniel Plohmann, Christian J. Dietrich, and Herbert Bos. 2013. SoK: P2PWNED - Modeling and Evaluating the Resilience of Peer-to-Peer Botnets. In Proceedings of the 2013 IEEE Symposium on Security and Privacy (S&P 2013). IEEE, San Francisco, CA, USA, 97--111.

Digital Library

[17]

Stephan Schmidt, Tansu Alpcan, Şahin Albayrak, Tamer Başar, and Achim Mueller. 2007. A Malware Detector Placement Game for Intrusion Detection. In Proceedings of the 2nd International Workshop on Critical Information Infrastructures Security (CRITIS 2007). Springer, Benalmádena, Málaga, Spain, 311--326.

[18]

Seungwon Shin, Lei Xu, Sungmin Hong, and Guofei Gu. 2016. Enhancing Network Security through Software Defined Networking (SDN). In Proceedings of the 25th International Conference on Computer Communication and Networks (ICCCN 2016). IEEE, Waikoloa, HI, USA.

[19]

Ali Shiravi, Hadi Shiravi, Mahbod Tavallaee, and Ali A. Ghorbani. 2012. Toward developing a systematic approach to generate benchmark datasets for intrusion detection. Computers & Security Vol. 31, 3 (May 2012), 357--37.

Digital Library

[20]

Lance Spitzner. 2002. Honeypots: Tracking Hackers. Addison Wesley, Boston, MA, USA.

Digital Library

[21]

Patrick John Sweeney. 2014. Designing Effective And Stealthy Botnets for Cybet Espionage And Interdiction - Finding the Cyber High Ground. Ph.D. Dissertation. Thayer School of Engineering, Darthmouth College.

[22]

Symantec Security Response. 2011. W32.Duqu: The Precursor to the Next Stuxnet. https://www.symantec.com/connect/w32_duqu_precursor_next_stuxnet. (October 2011).

[23]

Trend Micro. 2013. Lateral Movement: How Do Threat Actors Move Deeper Into Your Network? (2013).

[24]

Sridhar Venkatesan, Massimiliano Albanese, George Cybenko, and Sushil Jajodia. 2016. A Moving Target Defense Approach to Disrupting Stealthy Botnets. In Proceedings of the 3rd ACM Workshop on Moving Target Defense (MTD 2016). ACM, Vienna, Austria, 37--46.

Digital Library

[25]

Yini Wang, Sheng Wen, Yang Xiang, and Wanlei Zhou. 2014. Modeling the Propagation of Worms in Networks: A Survey. IEEE Communications Surveys & Tutorials Vol. 16, 2 (2014), 942--960.

[26]

Michael P. Wellman and Achintya Prakash. 2014. Empirical Game-Theoretic Analysis of an Adaptive Cyber-Defense Scenario (Preliminary Report). In Proceedings of the International Conference on Decision and Game Theory for Security (GameSec 2014) (Lecture Notes in Computer Science), Vol. 8840. Springer, Los Angeles, CA, USA, 43--58.

[27]

Michael West. 2009. Computer and Information Security Handbook. Morgan Kaufmann, Chapter Preventing System Intrusions, 39--51.

[28]

Junjie Zhang, Roberto Perdisci, Wenke Lee, Xiapu Luo, and Unum Sarfraz. 2014. Building a Scalable System for Stealthy P2P-Botnet Detection. IEEE Transactions on Information Forensics and Security, Vol. 9, 1 (January 2014), 27--38.

Digital Library

[29]

Junjie Zhang, Roberto Perdisci, Wenke Lee, Unum Sarfraz, and Xiapu Luo. 2011. Detecting stealthy P2P botnets using statistical traffic fingerprints. In Proceedings of the 41st IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2011). IEEE, Hong Kong, China, 121--132.

Digital Library

Cited By

Feltus C(2022)Reinforcement Learning's Contribution to the Cyber Security of Distributed SystemsInternational Journal of Distributed Artificial Intelligence10.4018/IJDAI.202007010312:2(35-55)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.4018/IJDAI.2020070103
Feltus C(2022)Reinforcement Learning's Contribution to the Cyber Security of Distributed SystemsResearch Anthology on Convergence of Blockchain, Internet of Things, and Security10.4018/978-1-6684-7132-6.ch025(421-444)Online publication date: 8-Jul-2022
https://doi.org/10.4018/978-1-6684-7132-6.ch025
Wang WSun DJiang FChen XZhu C(2022)Research and Challenges of Reinforcement Learning in Cyber Defense Decision-Making for Intranet SecurityAlgorithms10.3390/a1504013415:4(134)Online publication date: 18-Apr-2022
https://doi.org/10.3390/a15040134
Show More Cited By

Index Terms

Detecting Stealthy Botnets in a Resource-Constrained Environment using Reinforcement Learning
1. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
    1. Intrusion detection systems
  2. Network security

Recommendations

A Moving Target Defense Approach to Disrupting Stealthy Botnets
MTD '16: Proceedings of the 2016 ACM Workshop on Moving Target Defense

Botnets are increasingly being used for exfiltrating sensitive data from mission-critical systems. Research has shown that botnets have become extremely sophisticated and can operate in stealth mode by minimizing their host and network footprint. In ...
Detecting stealthy P2P botnets using statistical traffic fingerprints
DSN '11: Proceedings of the 2011 IEEE/IFIP 41st International Conference on Dependable Systems&Networks

Peer-to-peer (P2P) botnets have recently been adopted by botmasters for their resiliency to take-down efforts. Besides being harder to take down, modern botnets tend to be stealthier in the way they perform malicious activities, making current detection ...
Detecting, validating and characterizing computer infections in the wild
IMC '11: Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference

Although network intrusion detection systems (IDSs) have been studied for several years, their operators are still overwhelmed by a large number of false-positive alerts. In this work we study the following problem: from a large archive of intrusion ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MTD '17: Proceedings of the 2017 Workshop on Moving Target Defense

October 2017

126 pages

ISBN:9781450351768

DOI:10.1145/3140549

Program Chairs:
Hamed Okhravi
MIT Lincoln Laboratory, USA
,
Xinming Ou
University of South Florida, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

CCS '17

Sponsor:

SIGSAC

CCS '17: 2017 ACM SIGSAC Conference on Computer and Communications Security

October 30, 2017

Texas, Dallas, USA

Acceptance Rates

MTD '17 Paper Acceptance Rate 9 of 26 submissions, 35%;

Overall Acceptance Rate 40 of 92 submissions, 43%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
701
Total Downloads

Downloads (Last 12 months)110
Downloads (Last 6 weeks)17

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Feltus C(2022)Reinforcement Learning's Contribution to the Cyber Security of Distributed SystemsInternational Journal of Distributed Artificial Intelligence10.4018/IJDAI.202007010312:2(35-55)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.4018/IJDAI.2020070103
Feltus C(2022)Reinforcement Learning's Contribution to the Cyber Security of Distributed SystemsResearch Anthology on Convergence of Blockchain, Internet of Things, and Security10.4018/978-1-6684-7132-6.ch025(421-444)Online publication date: 8-Jul-2022
https://doi.org/10.4018/978-1-6684-7132-6.ch025
Wang WSun DJiang FChen XZhu C(2022)Research and Challenges of Reinforcement Learning in Cyber Defense Decision-Making for Intranet SecurityAlgorithms10.3390/a1504013415:4(134)Online publication date: 18-Apr-2022
https://doi.org/10.3390/a15040134
Minjiao ZYufeng MBo WZhang Q(2022)A Dynamic Deceptive Honeynet System with A Hybrid of Virtual and Real Devices2022 5th International Conference on Computing and Big Data (ICCBD)10.1109/ICCBD56965.2022.10080304(113-117)Online publication date: 16-Dec-2022
https://doi.org/10.1109/ICCBD56965.2022.10080304
Cody TRahman ARedino CHuang LClark RKakkar AKushwaha DPark PBeling PBowen E(2022)Discovering Exfiltration Paths Using Reinforcement Learning with Attack Graphs2022 IEEE Conference on Dependable and Secure Computing (DSC)10.1109/DSC54232.2022.9888919(1-8)Online publication date: 22-Jun-2022
https://doi.org/10.1109/DSC54232.2022.9888919
Huang YHuang LZhu Q(2022)Reinforcement Learning for feedback-enabled cyber resilienceAnnual Reviews in Control10.1016/j.arcontrol.2022.01.00153(273-295)Online publication date: 2022
https://doi.org/10.1016/j.arcontrol.2022.01.001
Venkatesan S(2022)Detector Placement StrategiesEncyclopedia of Cryptography, Security and Privacy10.1007/978-3-642-27739-9_1769-1(1-3)Online publication date: 14-Dec-2022
https://doi.org/10.1007/978-3-642-27739-9_1769-1
Shinde SMane S(2021)Malicious Profile Detection on Social Media: A Survey Paper2021 9th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)10.1109/ICRITO51393.2021.9596322(1-5)Online publication date: 3-Sep-2021
https://doi.org/10.1109/ICRITO51393.2021.9596322
de Neira AAraujo ANogueira M(2020)Early Botnet Detection for the Internet and the Internet of Things by Autonomous Machine Learning2020 16th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN50589.2020.00087(516-523)Online publication date: Dec-2020
https://doi.org/10.1109/MSN50589.2020.00087
Patel KAgrahari SSrivastava S(2020)Survey on Fake Profile Detection on Social Sites by Using Machine Learning Algorithm2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)10.1109/ICRITO48877.2020.9197935(1236-1240)Online publication date: Jun-2020
https://doi.org/10.1109/ICRITO48877.2020.9197935
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents