Abstract
Malware and code-reuse attacks are the most significant threats to current systems operation. Solutions developed to countermeasure them have their weaknesses exploited by attackers through sandbox evasion and antidebug crafting. To address such weaknesses, we propose a framework that relies on the modern processors’ branch monitor feature to allow us to analyze malware while reducing evasion effects. The use of hardware assistance aids in increasing stealthiness, a key feature for debuggers, as modern software (malicious or benign) may be antianalysis armored. We achieve stealthier code execution control by using the branch monitor hardware’s inherent interrupt capabilities, keeping the code under execution intact. Previous works on branch monitoring have already addressed the ROP attack problem but require code injection and/or are limited in their capture window size. Therefore, we also propose a ROP detector without these limitations.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Enhancing Branch Monitoring for Security Purposes: From Control Flow Integrity to Malware Analysis and Debugging
- Julien Ahrens. 2014. Easy File Management Web Server 5.3 - ‘UserID’ Remote Buffer Overflow (ROP). Retrieved November 6, 2017, from https://www.exploit-db.com/exploits/33610/.Google Scholar
- Y. Akao and T. Yamauchi. 2015. Proposal of kernel rootkits detection method by monitoring branches using hardware features. In Proceedings of the 2015 IIAI 4th International Congress on Advanced Applied Informatics. IEEE, Los Alamitos, CA, 721--722. Google Scholar
Digital Library
- Erdem Aktas and Kanad Ghose. 2013. Run-time control flow authentication: An assessment on contemporary x86 platforms. In Proceedings of the 28th Annual ACM Symposium on Applied Computing (SAC’13). ACM, New York, NY, 1859--1866.Google Scholar
Digital Library
- AMD. 2012. AMD64 Architecture Programmer’s Manual Volume 2. AMD.Google Scholar
- ARM. 2011. Cortex-A Series Programmer’s Guide. ARM.Google Scholar
- Davide Balzarotti, Marco Cova, Christoph Karlberger, Christopher Kruegel, Engin Kirda, and Giovanni Vigna. 2010. Efficient detection of split personalities in malware. In Proceedings of the17th Annual Network and Distributed System Security Symposium. 1--16.Google Scholar
- Julian Bangert, Sergey Bratus, Rebecca Shapiro, and Sean W. Smith. 2013. The page-fault weird machine: Lessons in instruction-less computation. In Proceedings of the 7th USENIX Conference on Offensive Technologies (WOOT’13). 1--13.Google Scholar
- Gabriel Negreira Barbosa and Rodrigo Rubira Branco. 2014. Prevalent Characteristics in Modern Malware. Retrieved November 6, 2017, from https://www.blackhat.com/docs/us-14/materials/us-14-Branco-Prevalent-Characteristics-In-Modern-Malware.pdf.Google Scholar
- Georgios Bitzes and Andrzej Nowak. 2014. The Overhead of Profiling Using PMU Hardware Counters. Retrieved November 6, 2017, from https://zenodo.org/record/10800/files/TheOverheadOfProfilingUsingPMUhardwareCounters.pdf.Google Scholar
- Tyler Bletsch, Xuxian Jiang, and Vince Freeh. 2011a. Mitigating code-reuse attacks with control-flow locking. In Proceedings of the 27th Annual Computer Security Applications Conference (ACSAC’11). ACM, New York, NY, 353--362. Google Scholar
Digital Library
- Tyler Bletsch, Xuxian Jiang, Vince W. Freeh, and Zhenkai Liang. 2011b. Jump-oriented programming: A new class of code-reuse attack. In Proceedings of the 6th ACM Symposium on Information, Computer, and Communication Security (ASIACCS’11). 30--40. Google Scholar
Digital Library
- Marcus Felipe Botacin, Paulo Lício de Geus, and André Ricardo Abed Grégio. 2017. The other guys: Automated analysis of marginalized malware. Journal of Computer Virology and Hacking Techniques 2017, 1--12. DOI:http://dx.doi.org/10.1007/s11416-017-0292-8 Google Scholar
Cross Ref
- Rodrigo Rubira Branco, Gabriel Negreira Barbosa, and Pedro Drimel Neto. 2012. Scientific but Not Academical Overview of Malware Anti-Debugging, Anti-Disassembly and Anti-VM Technologies. Retrieved November 6, 2017, from https://media.blackhat.com/bh-us-12/Briefings/Branco/BH_US_12_Branco_Scientific_Academic_Slides.pdf.Google Scholar
- Capstone. 2016. Home Page. Retrieved November 6, 2017, from http://www.capstone-engine.org/.Google Scholar
- Alexander Chailytko and Stanislav Skuratovich. 2016. Defeating Sandbox Evasion: How to Increase the Successful Emulation Rate in Your Virtual Environment. Retrieved November 6, 2017, from https://www.virusbulletin.com/uploads/pdf/magazine/2016/VB2016-Chailytko-Skuratovich.pdf.Google Scholar
- Ping Chen, Hai Xiao, Xiaobin Shen, Xinchun Yin, Bing Mao, and Li Xie. 2009. DROP: Detecting return-oriented programming malicious code. In Proceedings of the 5th International Conference on Information Systems Security (ICISS’09). 163--177.Google Scholar
Digital Library
- Yueqiang Cheng, Zongwei Zhou, Yu Miao, Xuhua Ding, Huijie Deng, and Robert Deng. 2014. ROPecker: A generic and practical approach for defending against ROP attack. In Proceedings of the 2014 NDSS Symposium. Google Scholar
Cross Ref
- Andrei Chiş, Marcus Denker, Tudor Gîrba, and Oscar Nierstrasz. 2015. Practical domain-specific debuggers using the Moldable Debugger framework. Computer Languages, Systems and Structures 44, PA, 89--113.Google Scholar
- CloudBurst. 2016. Reverse Engineering for Malware: Shellcodes and AV/API Hook Evasion. Retrieved November 6, 2017, from https://www.cloudburstsecurity.com/2016/06/10/reverse-engineering-for-malware-shellcodes-and-avapi-hook-evasion.Google Scholar
- L. Davi, M. Hanreich, D. Paul, A. R. Sadeghi, P. Koeberl, D. Sullivan, O. Arias, and Y. Jin. 2015. HAFIX: Hardware-assisted flow integrity extension. In Proceedings of the 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC’15). ACM, Los Alamitos, CA, 1--6. 0738-100XGoogle Scholar
- Lucas Davi, Ahmad-Reza Sadeghi, and Marcel Winandy. 2009. Dynamic integrity measurement and attestation: Towards defense against return-oriented programming attacks. In Proceedings of the 2009 ACM Workshop on Scalable Trusted Comp. (STC’09). ACM, New York, NY, 1--6.Google Scholar
Digital Library
- S. Debray and J. Patel. 2010. Reverse engineering self-modifying code: Unpacker extraction. In Proceedings of the 2010 17th Working Conference on Reverse Engineering. IEEE, Los Alamitos, CA, 131--140. Google Scholar
Digital Library
- Artem Dinaburg, Paul Royal, Monirul Sharif, and Wenke Lee. 2008. Ether: Malware analysis via hardware virtualization extensions. In Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS’08). 51--62.Google Scholar
Digital Library
- Manuel Egele, Theodoor Scholte, Engin Kirda, and Christopher Kruegel. 2008. A survey on automated dynamic malware-analysis techniques and tools. ACM Computing Surveys 44, 2, 6:1--6:42.Google Scholar
- Aristide Fattori, Roberto Paleari, Lorenzo Martignoni, and Mattia Monga. 2010. Dynamic and transparent analysis of commodity production systems. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering (ASE’10). 417--426. Google Scholar
Digital Library
- Yuxin Gao, Zexin Lu, and Yuqing Luo. 2014. Survey on malware anti-analysis. In Proceedings of the 2014 5th International Conference on Intelligent Control and Information Processing (ICICIP’14). IEEE, Los Alamitos, CA, 270--275. Google Scholar
Cross Ref
- Enes Göktaş, Elias Athanasopoulos, Michalis Polychronakis, Herbert Bos, and Georgios Portokalidis. 2014. Size does matter: Why using gadget-chain length to prevent code-reuse attacks is hard. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security’14). 417--432.Google Scholar
- Mariano Graziano, Davide Balzarotti, and Alain Zidouemba. 2016. ROPMEMU: A framework for the analysis of complex code-reuse attacks. In Proceedings of the 11th ACM Asia Conference on Computer and Commications Security (ASIACCS’16). Google Scholar
Digital Library
- Groundworkstech. 2016. A Python Interface to the GNU Binary File Descriptor (BFD) Library. Retrieved November 6, 2017, from https://github.com/Groundworkstech/pybfd.Google Scholar
- Claudio Guarnieri. 2013. Cuckoo Sandbox. Retrieved November 6, 2017, from http://www.cuckoosandbox.org/.Google Scholar
- Jason Hiser, Anh Nguyen-Tuong, Michele Co, Matthew Hall, and Jack W. Davidson. 2012. ILR: Where’d my gadgets go? In Proceedings of the 2012 IEEE Symposium on Security and Privacy (SP’12). 571--585.Google Scholar
- A. Ho, S. Hand, and T. Harris. 2004. PDB: Pervasive debugging with Xen. In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. IEEE, Los Alamitos, CA, 260--265. Google Scholar
Digital Library
- Intel. 2015. Intel® 64 and IA-32 Architectures Software Developer’s Manual. Intel.Google Scholar
- Intel. 2016. Control-Flow Enforcement Technology Preview. Retrieved November 6, 2017, from https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf.Google Scholar
- Jun Jiang, Xiaoqi Jia, Dengguo Feng, Shengzhi Zhang, and Peng Liu. 2011. HyperCrop: A hypervisor-based countermeasure for return oriented programming. In Proceedings of the 13th International Conference on Information and Communications Security (ICICS’11). 360--373. Google Scholar
Cross Ref
- Noah M. Johnson, Juan Caballero, Kevin Zhijie Chen, Stephen McCamant, Pongsin Poosankam, Daniel Reynaud, and Dawn Song. 2011. Differential slicing: Identifying causal execution differences for security applications. In Proceedings of the 2011 IEEE Symposium on Security and Privacy (SP’11). IEEE, Los Alamitos, CA, 347--362. DOI:http://dx.doi.org/10.1109/SP.2011.41 Google Scholar
Digital Library
- Dhilung Kirat and Giovanni Vigna. 2015. MalGene: Automatic extraction of malware analysis evasion signature. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS’15). ACM, New York, NY, 769--780. DOI:http://dx.doi.org/10.1145/2810103.2813642 Google Scholar
Digital Library
- Dhilung Kirat, Giovanni Vigna, and Christopher Kruegel. 2014. Barecloud: Bare-metal analysis-based evasive malware detection. In Proceedings of the 23rd USENIX Security Symposium (SEC’14). 287--301.Google Scholar
- Knaps. 2015. Easy File Sharing Web Server 7.2 - Remote Buffer Overflow (SEH) (DEP Bypass + ROP). Retrieved November 6, 2017, from https://www.exploit-db.com/exploits/38829/.Google Scholar
- S. Kompalli. 2014. Using existing hardware services for malware detection. In Proceedings of the 2014 IEEE Security and Privacy Workshops (SPW’14). IEEE, Los Alamitos, CA, 204--208. Google Scholar
Digital Library
- Bingchen Lan, Yan Li, Hao Sun, Chao Su, Yao Liu, and Qingkai Zeng. 2015. Loop-oriented programming: A new code reuse attack to bypass modern defenses. In Proceedings of the IEEE Trustcom/BigDataSE/ISPA Conference. 190--197. Google Scholar
Digital Library
- Jari-Matti Mäkelä, Ville Leppänen, and Martti Forsell. 2013. Towards a parallel debugging framework for the massively multi-threaded, step-synchronous replica architecture. In Proceedings of the 14th International Conference on Computer Systems and Technologies (CompSysTech’13). ACM, New York, NY, 1--8.Google Scholar
Digital Library
- J. A. P. Marpaung, M. Sain, and H.-J. Lee. 2012. Survey on malware evasion techniques: State of the art and challenges. In Proceedings of the 14th International Conference on Advanced Communication Technology (ICACT’12). 744--749.Google Scholar
- A. Moser, C. Kruegel, and E. Kirda. 2007. Limits of static analysis for malware detection. In Proceedings of the 23rd Annual Computer Security Applications Conference (ACSAC’07). ACM, New York, NY, 421--430. Google Scholar
Cross Ref
- James Newsome and Dawn Song. 2005. Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS’05). 1--17.Google Scholar
- Anh M. Nguyen, Nabil Schear, HeeDong Jung, Apeksha Godiyal, Samuel T. King, and Hai D. Nguyen. 2009. MAVMM: Lightweight and purpose built VMM for malware analysis. In Proceedings of the IEEE Annual Computer Security Applications Conference (ACSAC’09). 441--450.Google Scholar
- Kaan Onarlioglu, Leyla Bilge, Andrea Lanzi, Davide Balzarotti, and Engin Kirda. 2010. G-Free: Defeating return-oriented programming through gadget-less binaries. In Proceedings of the 26th Annual Computer Security Applications Conference (ACSAC’10). ACM, New York, NY, 1--10.Google Scholar
Digital Library
- Vasilis Pappas, Michalis Polychronakis, and Angelos D. Keromytis. 2012. Smashing the gadgets: Hindering return-oriented programming using in-place code randomization. In Proceedings of the 2012 IEEE Symposium on Security and Privacy (SP’12). IEEE, Los Alamitos, CA, 1--15.Google Scholar
- Vasilis Pappas, Michalis Polychronakis, and Angelos D. Keromytis. 2013. Transparent ROP exploit mitigation using indirect branch tracing. In Proceedings of the 22nd USENIX Security Symposium (SEC’13). 447--462.Google Scholar
- Rian Quinn. 2012. Detection of Malware via Side Channel Information. Ph.D. Dissertation. Binghamton University.Google Scholar
- James Reinders.2013. Processor Tracing. Retrieved November 6, 2017, from https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing.Google Scholar
- Thomas Roccia. 2016. An Overview of Malware Self-Defense and Protection. Retrieved November 6, 2017, from https://securingtomorrow.mcafee.com/mcafee-labs/overview-malware-self-defense-protection/.Google Scholar
- Jonathan B. Rosenberg. 1996. How Debuggers Work: Algorithms, Data Structures, and Architecture. John Wiley 8 Sons, New York, NY.Google Scholar
- Christian Rossow, Christian J. Dietrich, Christian Kreibich, Chris Grier, Vern Paxson, Norbert Pohlmann, Herbert Bos, and Maarten van Steen. 2012. Prudent practices for designing malware experiments: Status quo and outlook. In Proceedings of the 33rd IEEE Symposium on Security and Privacy (S8P’12). 65--79.Google Scholar
Digital Library
- Daniel Schulz and Frank Mueller. 2000. A thread-aware debugger with an open interface. In Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’00). ACM, New York, NY, 1--11.Google Scholar
Digital Library
- Felix Schuster, Thomas Tendyck, Jannik Pewny, Andreas Maaß, Martin Steegmanns, Moritz Contag, and Thorsten Holz. 2014. Evaluating the effectiveness of current anti-ROP defenses. In Research in Attacks, Intrusions and Defenses. Lecture Notes in Computer Science, Vol. 8688. Springer, 88--108.Google Scholar
- Edward J. Schwartz, Thanassis Avgerinos, and David Brumley. 2010. All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In Proceedings of the IEEE Symposium on Security and Privacy (SP’10). IEEE, Los Alamitos, CA, 317--331.Google Scholar
Digital Library
- Rebecca Shapiro, Sergey Bratus, and Sean W. Smith. 2013. “W eird machines” in ELF: A spotlight on the underappreciated metadata. In Proceedings of the 7th USENIX Conference on Offensive Technologies (WOOT’13). 11.Google Scholar
- Ahmad Sharif and Hsien-Hsin S. Lee. 2008. Total recall: A debugging framework for GPUs. In Proceedings of the 23rd ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware (GH’08). 13--20.Google Scholar
- Hao Shi, Abdulla Alwabel, and Jelena Mirkovic. 2014. Cardinal pill testing of system virtual machines. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security’14). 271--285.Google Scholar
- Michael Sikorski and Andrew Honig. 2012. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software.No Starch Press, San Francisco, CA.Google Scholar
- M. L. Soffa, K. R. Walcott, and J. Mars. 2011. Exploiting hardware advances for software testing and debugging: NIER track. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). 888--891.Google Scholar
- Nguyen Hong Son. 2011. ROP Chain for Windows 8. Retrieved November 6, 2017, from http://security.bkav.com/home/-/blogs/rop-chain-for-windows-8/normal.Google Scholar
- J. Vanegue. 2014. The weird machines in proof-carrying code. In Proceedings of the 2014 IEEE Security and Privacy Workshops (SPW’14). IEEE, Los Alamitos, CA, 209--213. Google Scholar
Digital Library
- Amit Vasudevan and Ramesh Yerraballi. 2005. Stealth breakpoints. In Proceedings of the 21st IEEE Annual Computer Security Applications Conference (ACSAC’05). 381--392.Google Scholar
Digital Library
- Amit Vasudevan and Ramesh Yerraballi. 2006a. Cobra: Fine-grained malware analysis using stealth localized-executions. In Proceedings of the IEEE Symposium on Security and Privacy (SP’06). 264--279.Google Scholar
Digital Library
- Amit Vasudevan and Ramesh Yerraballi. 2006b. SPiKE: Engineering malware analysis tools using unobtrusive binary-instrumentation. In Proceedings of the 29th Australasian Computer Science Conference (ACSC’06). 311--320.Google Scholar
- Richard Wartell, Vishwath Mohan, Kevin W. Hamlen, and Zhiqiang Lin. 2012. Binary stirring: Self-randomizing instruction addresses of legacy x86 binary code. In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS’12). ACM, New York, NY, 1--12.Google Scholar
Digital Library
- C. Willems, T. Holz, and F. Freiling. 2007. Toward automated dynamic malware analysis using CWSandbox. IEEE Security and Privacy 5, 2, 32--39. Google Scholar
Digital Library
- Carsten Willems, Ralf Hund, Andreas Fobian, Dennis Felsch, Thorsten Holz, and Amit Vasudevan. 2012. Down to the bare metal: Using processor features for binary analysis. In Proceedings of the 28th Annual Computer Security Applications Conference (ACSAC’12). ACM, New York, NY, 1--10.Google Scholar
Digital Library
- Jiyoung Woo and Huy Kang Kim. 2012. Survey and research direction on online game security. In Proceedings of the Workshop at SIGGRAPH Asia (WASA’12). ACM, New York, NY, 1--7.Google Scholar
Digital Library
- Y. Xia, Y. Liu, H. Chen, and B. Zang. 2012. CFIMon: Detecting violation of control flow integrity using performance counters. In Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’12). 1--12.Google Scholar
- M. Xianya, Z. Yi, W. Baosheng, and T. Yong. 2015. A survey of software protection methods based on self-modifying code. In Proceedings of the IEEE International Conference on Computational Intelligence and Communication Networks (CICN’15). 589--593. Google Scholar
Cross Ref
- Heng Yin, Dawn Song, Manuel Egele, Christopher Kruegel, and Engin Kirda. 2007. Panorama: Capturing system-wide information flow for malware detection and analysis. In Proceedings of the 14th ACM Conference on Computer and Communications Security. ACM, New York, NY, 116--127. Google Scholar
Digital Library
- Liwei Yuan, Weichao Xing, Haibo Chen, and Binyu Zang. 2011. Security breaches as PMU deviation: Detecting and identifying security attacks using performance counters. In Proceedings of the 2nd Asia-Pacific Workshop on Systems (APSys’11). ACM, New York, NY, Article 6, 5 pages.Google Scholar
Digital Library
- F. Zhang, K. Leach, A. Stavrou, H. Wang, and K. Sun. 2015. Using hardware features for increased debugging transparency. In Proceedings of the IEEE Symposium on Security and Privacy (SP’15). IEEE, Los Alamitos, CA, 55--69. Google Scholar
Digital Library
- Fengwei Zhang, Kevin Leach, Kun Sun, and Angelos Stavrou. 2013. SPECTRE: A dependable introspection framework via system management mode. In Proceedings of the 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’13). 1--12.Google Scholar
Digital Library
- Y. Zhong, H. Yamaki, and H. Takakura. 2012. A malware classification method based on similarity of function structure. In Proceedings of the IEEE/IPSJ 12th International Symposium on Applications and the Internet. 256--261. Google Scholar
Digital Library
Index Terms
Enhancing Branch Monitoring for Security Purposes: From Control Flow Integrity to Malware Analysis and Debugging
Recommendations
Antivirus security: naked during updates
The security of modern computer systems heavily depends on security tools, especially on antivirus software solutions. In the anti-malware research community, development of techniques for evading detection by antivirus software is an active research ...
Rope: Covert Multi-process Malware Execution with Return-Oriented Programming
Computer Security – ESORICS 2021AbstractDistributed execution designs challenge behavioral analyses of anti-malware solutions by spreading seemingly benign chunks of a malicious payload to multiple processes. Researchers have explored methods to chop payloads, spread chunks to victim ...
SeBROP: blind ROP attacks without returns
AbstractCurrently, security-critical server programs are well protected by various defense techniques, such as Address Space Layout Randomization(ASLR), eXecute Only Memory(XOM), and Data Execution Prevention(DEP), against modern code-reuse attacks like ...






Comments