research-article

VMAttack: Deobfuscating Virtualization-Based Packed Binaries

Authors:

Anatoli Kalysch,

Johannes Götzfried,

Tilo MüllerAuthors Info & Claims

ARES '17: Proceedings of the 12th International Conference on Availability, Reliability and Security

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/3098954.3098995

Published: 29 August 2017 Publication History

Abstract

We present VMAttack, a deobfuscation tool for virtualization-packed binaries based on automated static and dynamic analysis, which offers a simplified view of the disassembly. VMAttack is implemented as a plug-in for IDA Pro and as such, integrates seamlessly with manual reverse engineering. The complexity of the disassembly view is notably reduced by analyzing the inner working principles of the VM layer of protected binaries. Using static analysis, complex bytecode sequences of the VM are mapped to easy-to-read pseudo-code instructions, based on an intermediate representation specifically designed for stack-based virtual machines. Using dynamic analysis, we identify structural components like the interpreter loop and compress instruction sequences by filtering out semantically redundant instructions of the execution trace. The integrated result, which rates both static and dynamic analysis's results, provides the reverse engineer with a deobfuscated disassembly that tolerates weaknesses of a single analysis technique. VMAttack is currently limited to stack-based virtual machines like VMProtect. We evaluated VMAttack using binaries obfuscated with VMProtect and achieved an average execution trace reduction of 89.86% for the dynamic and 96.67% for the combined static and dynamic analysis.

References

[1]

Rambus Inc. 2009. About Self-Protecting Digital Content. (2009). https://www.rambus.com/about-spdc/, accessed on 06. March 2017.

[2]

Hiralal Agrawal and Joseph R Horgan. 1990. Dynamic program slicing. In ACM SIG Plan Notices, Vol. 25. ACM, 246--256.

Digital Library

[3]

Bertrand Anckaert, Mariusz Jakubowski, and Ramarathnam Venkatesan. 2006. Proteus: virtualization for diversified tamper-resistance. In Proceedings of the ACM workshop on Digital rights management. ACM, 47--58.

Digital Library

[4]

Samuel Chevet. 2015. Inside VMProtect. (2015). http://lille1tv.univ-lille1.fr/telecharge.aspx?id=d5b2487e-cacc-4596-ab37-dab2b362cb9e, accessed on 10. March 2017.

[5]

Kevin Coogan, Gen Lu, and Saumya Debray. 2011. Deobfuscation of Virtualization-obfuscated Software: A Semantics-based Approach. In Proceedings of the 18th ACM Conference on Computer and Communications Security (CCS '11). ACM, New York, NY, USA, 275--284.

Digital Library

[6]

Kevin Patrick Coogan. 2011. Deobfuscation of Packed and Virtualization-obfuscation Protected Binaries. Ph.D. Dissertation. Tucson, AZ, USA. Advisor(s) Debray, Saumya. AAI3468656.

[7]

Iain D. Craig. 2006. Virtual Machines. Springer-Verlag.

[8]

Yoni De Mulder, Brecht Wyseur, and Bart Preneel. 2010. Cryptanalysis of a perturbated white-box AES implementation. In International Conference on Cryptology in India. Springer.

[9]

E. Eilam and E. J. Chikofsky. 2005. Reversing: secrets of reverse engineering. Wiley.

Digital Library

[10]

Hui Fang, Yongdong Wu, Shuhong Wang, and Yin Huang. 2011. Multi-stage Binary Code Obfuscation Using Improved Virtual Machine. In Information Security, Xuejia Lai, Jianying Zhou, and Hui Li (Eds.). Lecture Notes in Computer Science, Vol. 7001. Springer Berlin Heidelberg, 168--181.

Digital Library

[11]

Ilfak Guilfanov. 2016. IDA Pro Plug-in Contest 2016. (2016). https://www.hex-rays.com/contests/2016/index.shtml, accessed on 23. March 2017.

[12]

Yoann Guillot and Alexandre Gazet. 2010. Automatic binary deobfuscation. Journal in Computer Virology (2010).

[13]

Min Gyung Kang, Pongsin Poosankam, and Heng Yin. 2007. Renovo: A hidden code extractor for packed executables. In Proceedings of the 2007 ACM workshop on Recurring malcode. ACM, 46--53.

Digital Library

[14]

Johannes Kinder. 2012. Towards static analysis of virtualization-obfuscated binaries. In 2012 19th Working Conference on Reverse Engineering. IEEE, 61--70.

Digital Library

[15]

Jasvir Nagra and Christian Collberg. 2009. Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing for Software Protection. Pearson Education.

Digital Library

[16]

Eric Chien Nicolas Falliere, Patrick Fitzgerald. 2009. Inside the Jaws of Trojan.Clampi. (2009). https://www.symantec.com/content/en/us/enterprise/media/security_response/whitepapers/inside_trojan_clampi.pdf, accessed on 17. October 2016.

[17]

Oreans Technologies. 2016. Themida. (2016). http://www.oreans.com, accessed on 10. March 2017.

[18]

Frederic Perriot. 2009. Countering polymorphic malicious computer code through code optimization. (Nov. 24 2009). US Patent 7,624,449.

[19]

Jason Raber. 2013. Virtual Deobfuscator -- a DARPA Cyber Fast Track funded effort. (2013). http://www.cerosecurity.com/blackhat-usa-2013-presentaciones-y-diapositivas/, accessed on 10. March 2017.

[20]

Babak Bashari Rad, Maslin Masrom, and Suhaimi Ibrahim. 2012. Camouflage in malware: from encryption to metamorphism. In International Journal of Computer Science and Network Security. 74--83.

[21]

Rolf Rolles. 2007. Defeating HyperUnpackMe2. (2007). http://www.openrce.org/articles/full_view/28, accessed on 10. March 2017.

[22]

Rolf Rolles. 2009. Unpacking Virtualization Obfuscators. In Proceedings of the 3rd USENIX Conference on Offensive Technologies (WOOT'09). USENIX Association, Berkeley, CA, USA.

Digital Library

[23]

Shared Encyclopedia. 2016. VMProtect Logical instruction. (2016). http://et97.com/view/1281031.htm, accessed on 10. March 2017.

[24]

Monirul I. Sharif, Andrea Lanzi, Jonathon T. Giffin, and Wenke Lee. 2009. Automatic Reverse Engineering of Malware Emulators. In 30th IEEE Symposium on Security and Privacy (S&P 2009), 17-20 May 2009, Oakland, California, USA. 94--109.

Digital Library

[25]

Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Andrew Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, and Giovanni Vigna. 2016. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. (2016).

[26]

Craig Smith. 2008. Creating Code Obfuscation Virtual Machines. In Proceedings of the RECON 2008, Reverse Engineering Conference. Neohapsis, Inc.

[27]

Jim Smith and Ravi Nair. 2005. Virtual machines: versatile platforms for systems and processes. Elsevier.

Digital Library

[28]

Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. (2016).

[29]

VMPSoft. 2016. VMProtect. (2016). http://www.vmpsoft.com, accessed on 10. March 2017.

[30]

Babak Yadegari, Brian Johannesmeyer, Ben Whitely, and Saumya Debray. 2015. A Generic Approach to Automatic Deobfuscation of Executable Code. In 2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015. 674--691.

Digital Library

Cited By

Zhang ZXue JBaker TChen TTan YLi Y(2024)COVER: Enhancing virtualization obfuscation through dynamic scheduling using flash controller-based secure moduleComputers & Security10.1016/j.cose.2024.104038146(104038)Online publication date: Nov-2024
https://doi.org/10.1016/j.cose.2024.104038
Anwar MZuo CYagemann CLin Z(2023)Extracting Threat Intelligence From Cheat Binaries For Anti-CheatingProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607211(17-31)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3607199.3607211
Luo CMing JFu JPeng GLi Z(2023)Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics TestingIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328925418(3891-3905)Online publication date: 2023
https://doi.org/10.1109/TIFS.2023.3289254
Show More Cited By

Recommendations

Hybrid Analysis Technique to detect Advanced Persistent Threats

Advanced persistent threats APT are major threats in the field of system and network security. They are extremely stealthy and use advanced evasion techniques like packing and behaviour obfuscation to hide their malicious behaviour and evade the ...
Integrated Static and Dynamic Analysis for Malware Detection
Abstract
The number of malware is increasing rapidly regardless of the common use of anti-malware software. Detection of malware continues to be a challenge as attackers device new techniques to evade from the detection methods. Most of the anti-virus ...
Building Deobfuscated Applications from Polymorphic Binaries
Innovative Security Solutions for Information Technology and Communications
Abstract
Along with the rise of the cyber threats industry, attackers have become more fluent in developing and integrating various obfuscation layers. This is mainly focused on impeding or at least slowing the analysis and the reverse engineering process, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ARES '17: Proceedings of the 12th International Conference on Availability, Reliability and Security

August 2017

853 pages

ISBN:9781450352574

DOI:10.1145/3098954

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

German Research Foundation (DFG)

Conference

ARES '17

ARES '17: International Conference on Availability, Reliability and Security

August 29 - September 1, 2017

Reggio Calabria, Italy

Acceptance Rates

ARES '17 Paper Acceptance Rate 100 of 191 submissions, 52%;

Overall Acceptance Rate 228 of 451 submissions, 51%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
341
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)6

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZXue JBaker TChen TTan YLi Y(2024)COVER: Enhancing virtualization obfuscation through dynamic scheduling using flash controller-based secure moduleComputers & Security10.1016/j.cose.2024.104038146(104038)Online publication date: Nov-2024
https://doi.org/10.1016/j.cose.2024.104038
Anwar MZuo CYagemann CLin Z(2023)Extracting Threat Intelligence From Cheat Binaries For Anti-CheatingProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3607199.3607211(17-31)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3607199.3607211
Luo CMing JFu JPeng GLi Z(2023)Reverse Engineering of Obfuscated Lua Bytecode via Interpreter Semantics TestingIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328925418(3891-3905)Online publication date: 2023
https://doi.org/10.1109/TIFS.2023.3289254
Chen YLin SHuang SLei CHuang C(2023)Guided Malware Sample Analysis Based on Graph Neural NetworksIEEE Transactions on Information Forensics and Security10.1109/TIFS.2023.328391318(4128-4143)Online publication date: 2023
https://doi.org/10.1109/TIFS.2023.3283913
Bonfante GIthayakumar A(2023)Where is the Virtual Machine Within Cpython?Foundations and Practice of Security10.1007/978-3-031-30122-3_11(175-191)Online publication date: 1-Apr-2023
https://doi.org/10.1007/978-3-031-30122-3_11
Usui TOtsuki YKawakoya YIwamura MMatsuura K(2022)Script Tainting Was Doomed From The Start (By Type Conversion): Converting Script Engines into Dynamic Taint Analysis FrameworksProceedings of the 25th International Symposium on Research in Attacks, Intrusions and Defenses10.1145/3545948.3545969(380-394)Online publication date: 26-Oct-2022
https://dl.acm.org/doi/10.1145/3545948.3545969
Kochberger PSchrittwieser SSchweighofer SKieseberg PWeippl E(2021)SoK: Automatic Deobfuscation of Virtualization-protected ApplicationsProceedings of the 16th International Conference on Availability, Reliability and Security10.1145/3465481.3465772(1-15)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1145/3465481.3465772
Xue LYan YYan LJiang MLuo XWu DZhou YCadar CZhang X(2021)Parema: an unpacking framework for demystifying VM-based Android packersProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464839(152-164)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464839
Usui TOtsuki YIkuse TKawakoya YIwamura MMiyoshi JMatsuura K(2021)Automatic Reverse Engineering of Script Engine Binaries for Building Script API TracersDigital Threats: Research and Practice10.1145/34161262:1(1-31)Online publication date: Mar-2021
https://doi.org/10.1145/3416126
Suk JLee D(2020)VCF: Virtual Code Folding to Enhance Virtualization ObfuscationIEEE Access10.1109/ACCESS.2020.30126848(139161-139175)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3012684
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents