research-article

Binary-code obfuscations in prevalent packer tools

Authors:

Kevin A. Roundy,

Barton P. MillerAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 46, Issue 1

Article No.: 4, Pages 1 - 32

https://doi.org/10.1145/2522968.2522972

Published: 11 July 2013 Publication History

Abstract

The first steps in analyzing defensive malware are understanding what obfuscations are present in real-world malware binaries, how these obfuscations hinder analysis, and how they can be overcome. While some obfuscations have been reported independently, this survey consolidates the discussion while adding substantial depth and breadth to it. This survey also quantifies the relative prevalence of these obfuscations by using the Dyninst binary analysis and instrumentation tool that was recently extended for defensive malware analysis. The goal of this survey is to encourage analysts to focus on resolving the obfuscations that are most prevalent in real-world malware.

References

[1]

Anckaert, B., Madou, M., and De Bosschere, K. 2007. A model for self-modifying code. In Proceedings of the Workshop on Information Hiding. 232--248.

Digital Library

[2]

Aucsmith, D. 1996. Tamper resistant software: An implementation. In Proceedings of the Workshop on Information Hiding. 317--333.

Digital Library

[3]

Babic, D., Martignoni, L., McCamant, S., and Song, D. 2011. Statically-directed dynamic automated test generation. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'11).

Digital Library

[4]

Balakrishnan, G. and Reps, T. 2004. Analyzing memory accesses in ×86 executables. In Proceedings of the Conference on Compiler Construction (CC'04). 5--23.

[5]

Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. 2003. Xen and the art of virtualization. In Proceedings of the Symposium on Operating Systems Principles.

Digital Library

[6]

Bayer, U., Moser, A., Kruegel, C., and Kirda, E. 2006. Dynamic analysis of malicious code. J. Comput. Virology 2, 1, 66--77.

[7]

Bellard, F. 2005. QEMU, a fast and portable dynamic translator. In Proceedings of the USENIX Annual Technical Conference.

Digital Library

[8]

Bernat, A. R. and Miller, B. P. 2011. Anywhere, any time binary instrumentation. In Proceedings of the Workshop on Program Analysis for Software Tools and Engineering (PASTE'11).

Digital Library

[9]

Bernat, A. R., Roundy, K. A., and Miller, B. P. 2011. Efficient, sensitivity resistant binary instrumentation. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'11).

Digital Library

[10]

BitDefender. 2007. BitDefender anti-virus technology. White Paper. http://www.bitdefender.com/news/&quest;f_year=2007.

[11]

Bohne, L. and Holz, T. 2008. Pandora's bochs: Automated malware unpacking. M.S. thesis, University of Mannheim. http://archive.hack.lu/2009/PandorasBochs.pdf.

[12]

Bruening, D. 2004. Efficient, transparent, and comprehensive runtime code manipulation. Ph.D. thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology.

Digital Library

[13]

Bruschi, D., Martignoni, L., and Monga, M. 2007. Code normalization for self-mutating malware. IEEE Secur. Privacy 5, 2.

Digital Library

[14]

Bustamante, P. 2008a. Malware prevalence. Panda Research web article. http://research.pandasecurity.com/malware-prevalence-august-2008/.

[15]

Bustamante, P. 2008b. Packer (r)evolution. Panda Research web article. http://research.pandasecurity.com/packer-revolution/.

[16]

Charney, M. 2010. Xed2 user guide. http://www.cs.virginia.edu/kim/publicity/pin/docs/36111/Xed/html/main.html.

[17]

Chow, J., Garfinkel, T., and Chen, P. M. 2008. Decoupling dynamic program analysis from execution in virtual environments. In Proceedings of the USENIX Annual Technical Conference. 24.

Digital Library

[18]

Christodorescu, M., Kinder, J., Jha, S., Katzenbeisser, S., and Veith, H. 2005. Malware normalization. Tech. rep. 1539, Computer Sciences Department, University of Wisconsin.

[19]

Cifuentes, C. and Fraboulet, A. 1997. Intraprocedural static slicing of binary executables. In Proceedings of the International Conference on Software Maintenance (ICSM'97).

Digital Library

[20]

Cifuentes, C. and Van Emmerik, M. 1999. Recovery of jump table case statements from binary code. In Proceedings of the International Workshop on Program Comprehension (ICPC'99).

Digital Library

[21]

Collberg, C., Thomborson, C., and Low, D. 1998. Manufacturing cheap, resilient, and stealthy opaque constructs. In Proceedings of the Symposium on Principles of Programming Languages (POPL'98).

Digital Library

[22]

Coogan, K., Debray, S., Kaochar, T., and Townsend, G. 2009. Automatic static unpacking of malware binaries. In Proceedings of the Working Conference on Reverse Engineering.

Digital Library

[23]

Danehkar, A. 2005. Inject your code into a portable executable file. http://www.codeproject.com/KB/system/inject2exe.aspx.

[24]

Dark Paranoid. 1998. Engine of eternal encryption. Moon bug 7. http://www.eset.com/us/threat-center/encyclopedia/threats/darkparanoid/.

[25]

Debray, S. and Evans, W. 2002. Profile-guided code compression. In Proceedings Conference on Programming Language Design and Implementation (PLDI'02).

Digital Library

[26]

Debray, S. and Patel, J. 2010. Reverse engineering self-modifying code: Unpacker extraction. In Proceedings of the Working Conference on Reverse Engineering.

Digital Library

[27]

Dinaburg, A., Royal, P., Sharif, M., and Lee, W. 2008. Ether: Malware analysis via hardware virtualization extensions. In Proceedings of the Conference on Computer and Communications Security.

Digital Library

[28]

Falliere, N. 2007. Windows anti-debug reference. Infocus web article. http://www.securityfocus.com/infocus/1893.

[29]

Ferrie, P. 2008a. Anti-unpacker tricks. In Proceedings of the International CARO Workshop.

[30]

Ferrie, P. 2008b. Anti-unpacker tricks - part one. Virus Bulletin. www.virusbtn.com/pdf/magazine/2008/200812.pdf.

[31]

Fog, A. 2011. Calling conventions for different c++ compilers and operating systems. http://www.agner.org/optimize/.

[32]

Giffin, J. T., Christodorescu, M., and Kruger, L. 2005. Strengthening software self-checksumming via self-modifying code. In Proceedings of the Annual Computer Security Applications Conference (ACSAC'05).

Digital Library

[33]

Gnu Project - Free Software Foundation. 2011. Objdump, gnu manuals online. Version 2.22 http://sourceware.org/binutils/docs/binutils/.

[34]

Graf, T. 2005. Generic unpacking - How to handle modified or unknown pe compression engines&quest; In Proceedings of the Virus Bulletin Conference.

[35]

Guilfanov, I. 2005. Using the universal pe unpacker plug-in included in ida pro 4.9 to unpack compressed executables. http://www.hex-rays.com/idapro/unpack_pe/unpacking.pdf.

[36]

Guilfanov, I. 2011. The ida pro disassembler and debugger. DataRescue, version 6.2. http://www.hex-rays.com/idapro/.

[37]

Guo, F., Ferrie, P., and Chiueh, T. 2008. A study of the packer problem and its solutions. In Proceedings of the Symposium on Recent Advances in Intrusion Detection (RAID'08).

Digital Library

[38]

Harris, L. C. and Miller, B. P. 2005. Practical analysis of stripped binary code. SIGARCH Comput. Archit. News 33, 5.

Digital Library

[39]

Hex-Rays Decompiler. 2011. Version 1.6. http://hex-rays.com.

[40]

Hollingsworth, J. K., Miller, B. P., and Cargille, J. 1994. Dynamic program instrumentation for scalable performance tools. In Proceedings of the Scalable High Performance Computing Conference.

[41]

Hunt, G. and Brubacher, D. 1999. Detours: Binary interception of win32 functions. In Proceedings of the USENIX Windows NT Symposium.

Digital Library

[42]

Jacobson, E. R., Rosenblum, N. E., and Miller, B. P. 2011. Labeling library functions in stripped binaries. In Proceedings of the Workshop on Program Analysis for Software Tools and Engineering (PASTE'11).

Digital Library

[43]

Kang, M. G., Poosankam, P., and Yin, H. 2007. Renovo: A hidden code extractor for packed executables. In Proceedings of the Workshop on Recurring Malcode.

Digital Library

[44]

Kruegel, C., Robertson, W., Valeur, F., and Vigna, G. 2004. Static disassembly of obfuscated binaries. In Proceedings of the USENIX Security Symposium.

Digital Library

[45]

Lakhotia, A., Kumar, E. U., and Venable, M. 2005. A method for detecting obfuscated calls in malicious binaries. Trans. Softw. Engin. 31, 11.

Digital Library

[46]

Lim, J. and Reps, T. 2008. A system for generating static analyzers for machine instructions. In Proceedings of the International Conference on Compiler Construction (CC'08).

Digital Library

[47]

Lindholm, T. and Yellin, F. 1999. Java Virtual Machine Specification 2^nd Ed. Addison-Wesley Longman Publishing Co., Boston, MA.

Digital Library

[48]

Linn, C. and Debray, S. 2003. Obfuscation of executable code to improve resistance to static disassembly. In Proceedings of the Conference on Computer and Communications Security.

Digital Library

[49]

Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V. J., and Hazelwood, K. 2005. PIN: Building customized program analysis tools with dynamic instrumentation. In Proceedings of the Conference on Programming Language Design and Implementation (PLDI'05).

Digital Library

[50]

Madou, M., Anckaert, B., De Sutter, B., and De Bosschere, K. 2005. Hybrid static-dynamic attacks against software protection mechanisms. In Proceedings of the ACM Workshop on Digital Rights Management. 25

Digital Library

[51]

Maebe, J. and De Bosschere, K. 2003. Instrumenting self-modifying code. In Proceedings of the Workshop on Automated and Algorithmic Debugging.

[52]

Maebe, J., Ronsse, M., and De Bosschere, K. 2002. DIOTA: Dynamic instrumentation, optimization and transformation of applications. In Proceedings of the Workshop on Binary Translation held in conjunction with the Conference on Parallel Architectures and Compilation Techniques (PACT'02).

[53]

Martignoni, L., Christodorescu, M., and Jha, S. 2007. Omniunpack: Fast, generic, and safe unpacking of malware. In Proceedings of the Annual Computer Security Applications Conference (ACSAC'07).

[54]

Miller, B. P., Fredriksen, L., and So, B. 1990. An empirical study of the reliability of unix utilities. Comm. ACM 33, 12.

Digital Library

[55]

Moser, A., Kruegel, C., and Kirda, E. 2007. Exploring multiple execution paths for malware analysis. In Proceedings of the Symposium on Security and Privacy.

Digital Library

[56]

Muth, R. and Debray, S. 2000. On the complexity of ow-sensitive dataow analyses. In Proceedings of the Symposium on Principles of Programming Languages (POPL'00).

Digital Library

[57]

Nanda, S., Li, W., Lam, L.-C., and Chiueh, T.-C. 2006. Bird: Binary interpretation using runtime disassembly. In Proceedings of the Symposium on Code Generation and Optimization (CGO'06).

Digital Library

[58]

Paradyn Tools Project. 2011. ParseAPI programmer's guide. Version 7.0.1. http://www.paradyn.org/html/manuals.html.

[59]

Payne, B. D. 2011. LibVMI, version 0.6. http://vmitools.sandia.gov/.

[60]

Payne, B. D., Carbone, M., and Lee, W. 2007. Secure and flexible monitoring of virtual machines. In Proceedings of the Annual Computer Security Applications Conference (ACSAC'07).

[61]

Perriot, F. and Ferrie, P. 2004. Principles and practise of x-raying. In Proceedings of the Virus Bulletin Conference.

[62]

Popov, I., Debray, S., and andrews, G. 2007. Binary obfuscation using signals. In Proceedings of the USENIX Security Symposium.

Digital Library

[63]

Porras, P., Saidi, H., and Yegneswaran, V. 2009. A foray into conficker's logic and rendezvous points. Tech. rep., SRI International. https://www.usenix.org/legacy/event/leet09/tech/full_papers/porras/porras.pdf.

[64]

Prakash, C. 2007. Design of ×86 emulator for generic unpacking. In Proceedings of the Association of Anti-Virus Asia Researchers International Conference.

[65]

Quinlan, D. 2000. Rose: Compiler support for object-oriented frameworks. In Proceedings of the Conference on Parallel Compilers (CPC'00).

[66]

Quist, D. and Smith, V. 2007. Covert debugging: Circumventing software armoring techniques. Blackhat USA. http://www.blackhat.com/presentations/bh-usa-07/Quist_and_Valsmith/Whitepaper/bh-usa-07-quist_and_valsmith-WP.pdf.

[67]

Rosenblum, N. E., Cooksey, G., and Miller, B. P. 2008a. Virtual machine-provided context sensitive page mappings. In Proceedings of the Conference on Virtual Execution Environments (VEE'08).

Digital Library

[68]

Rosenblum, N. E., Miller, B. P., and Zhu, X. 2010. Extracting compiler provenance from program binaries. In Proceedings of the Workshop on Program Analysis for Software Tools and Engineering (PASTE'10).

Digital Library

[69]

Rosenblum, N. E., Zhu, X., Miller, B. P., and Hunt, K. 2008b. Learning to analyze binary computer code. In Proceedings of the Conference on Artificial Intelligence (AAAI'08).

Digital Library

[70]

Roundy, K. A. 2012. Hybrid analysis and control of malicious code. Ph.D. thesis, Department of Computer Science, University of Wisconsin. http://pages.cs.wisc.edu/&sim;roundy/dissertation.pdf.

Digital Library

[71]

Roundy, K. A. and Miller, B. P. 2010. Hybrid analysis and control of malware. In Proceedings of the Symposium on Recent Advances in Intrusion Detection (RAID'10).

Digital Library

[72]

Royal, P., Halpin, M., Dagon, D., Edmonds, R., and Lee, W. 2006. PolyUnpack: Automating the hidden-code extraction of unpack-executing malware. In Proceedings of the Annual Computer Security Applications Conference (ACSAC'06).

Digital Library

[73]

Russinovich, M. and Cogswell, B. 1997. Windows NT system call hooking. Dr. Dobb's J. 22, 1.

[74]

Schwarz, B., Debray, S., and andrews, G. 2002. Disassembly of executable code revisited. In Proceedings of the 9^th Working Conference on Reverse Engineering. 45.

Digital Library

[75]

Sharif, M., Lanzi, A., Giffin, J., and Lee, W. 2008. Impeding malware analysis using conditional code obfuscation. In Proceedings of the Network and Distributed System Security Symposium (NDSS'08).

[76]

Sharif, M., Lanzi, A., Giffin, J., and Lee, W. 2009. Automatic reverse engineering of malware emulators. In Proceedings of the Symposium on Security and Privacy.

Digital Library

[77]

Sites, R. L., Chernoff, A., Kirk, M. B., Marks, M. P., and Robinson, S. G. 1993. Binary translation. Comm. ACM 36, 2.

Digital Library

[78]

Stepan, A. E. 2005. Defeating polymorphism: Beyond emulation. In Proceedings of the Virus Bulletin Conference.

[79]

Stewart, J. 2007. Unpacking with ollybone. Online tutorial. http://www.joestewart.org/ollybone/tutorial.html.

[80]

Szappanos, G. 2007. Exepacker blacklisting. Virus Bulletin.

[81]

Theiling, H. 2000. Extracting safe and precise control flow from binaries. In Proceedings of the Conference on Real-Time Computing Systems and Applications. 23.

Digital Library

[82]

Trilling, S. 2008. Project green bay-calling a blitz on packers. CIO digest: Strategies and analysis from symantec. http://eval.symantec.com/mktginfo/enterprise/articles/biodigest_october08_magazine.pdf.

[83]

Van Emmerik, M. and Waddington, T. 2004. Using a decompiler for real-world source recovery. In Proceedings of the Working Conference on Reverse Engineering (WCRE'04). 26.

Digital Library

[84]

Vigna, G. 2007. Static disassembly and code analysis. In Malware Detection. Advances in Information Security. Vol. 27, Springer, 19--41.

[85]

Willems, C., Holz, T., and Freiling, F. 2007. Toward automated dynamic malware analysis using cwsandbox. In Proceedings of the Symposium on Security and Privacy.

Digital Library

[86]

Wurster, G., Van Oorschot, P. C., and Somayaji, A. 2005. A generic attack on checksumming-based software tamper resistance. In Proceedings of the Symposium on Security and Privacy.

Digital Library

[87]

Yason, M. V. 2007. The art of unpacking. Blackhat USA. http://www.blackhat.com/presentations/bh-usa-07/Yason/Presentation/bh-usa-07-yason.pdf.

[88]

Yegneswaran, V., Saidi, H., and Porras, P. 2008. Eureka: A framework for enabling static analysis on malware. Tech. rep. SRI-CSL-08-01, SRI International.

[89]

Yuschuk, O. 2000. OllyDbg. Version 1.10. http://www.ollydbg.de.

Cited By

Lee GKim MYi JCho H(2024)Pinicorn: Towards Automated Dynamic Analysis for Unpacking 32-Bit PE MalwareElectronics10.3390/electronics1311208113:11(2081)Online publication date: 27-May-2024
https://doi.org/10.3390/electronics13112081
Gori GRinieri LMelis AAl Sadi ACallegati FPrandini M(2024)A Systematic Analysis of Security Metrics for Industrial Cyber–Physical SystemsElectronics10.3390/electronics1307120813:7(1208)Online publication date: 25-Mar-2024
https://doi.org/10.3390/electronics13071208
Zhang PWu CWang Z(2024)BinCodex: A comprehensive and multi-level dataset for evaluating binary code similarity detection techniquesBenchCouncil Transactions on Benchmarks, Standards and Evaluations10.1016/j.tbench.2024.1001634:2(100163)Online publication date: Jun-2024
https://doi.org/10.1016/j.tbench.2024.100163
Show More Cited By

Index Terms

Binary-code obfuscations in prevalent packer tools
1. Security and privacy
  1. Intrusion/anomaly detection and malware mitigation
2. Social and professional topics
  1. Computing / technology policy
    1. Computer crime

Recommendations

Obfuscation: The Hidden Malware

A cyberwar exists between malware writers and antimalware researchers. At this war's heart rages a weapons race that originated in the 80s with the first computer virus. Obfuscation is one of the latest strategies to camouflage the telltale signs of ...
Testing malware detectors

In today's interconnected world, malware, such as worms and viruses, can cause havoc. A malware detector (commonly known as virus scanner) attempts to identify malware. In spite of the importance of malware detectors, there is a dearth of testing ...
Testing malware detectors
ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis

In today's interconnected world, malware, such as worms and viruses, can cause havoc. A malware detector (commonly known as virus scanner) attempts to identify malware. In spite of the importance of malware detectors, there is a dearth of testing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 46, Issue 1

October 2013

551 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/2522968

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2013

Accepted: 01 October 2012

Revised: 01 August 2012

Received: 01 March 2012

Published in CSUR Volume 46, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

94
Total Citations
View Citations
1,540
Total Downloads

Downloads (Last 12 months)122
Downloads (Last 6 weeks)26

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lee GKim MYi JCho H(2024)Pinicorn: Towards Automated Dynamic Analysis for Unpacking 32-Bit PE MalwareElectronics10.3390/electronics1311208113:11(2081)Online publication date: 27-May-2024
https://doi.org/10.3390/electronics13112081
Gori GRinieri LMelis AAl Sadi ACallegati FPrandini M(2024)A Systematic Analysis of Security Metrics for Industrial Cyber–Physical SystemsElectronics10.3390/electronics1307120813:7(1208)Online publication date: 25-Mar-2024
https://doi.org/10.3390/electronics13071208
Zhang PWu CWang Z(2024)BinCodex: A comprehensive and multi-level dataset for evaluating binary code similarity detection techniquesBenchCouncil Transactions on Benchmarks, Standards and Evaluations10.1016/j.tbench.2024.1001634:2(100163)Online publication date: Jun-2024
https://doi.org/10.1016/j.tbench.2024.100163
Patsakis CCasino FLykousas N(2024)Assessing LLMs in malicious code deobfuscation of real-world malware campaignsExpert Systems with Applications10.1016/j.eswa.2024.124912(124912)Online publication date: Jul-2024
https://doi.org/10.1016/j.eswa.2024.124912
Assaiante CNicchi SD’Elia DQuerzoni L(2024)Evading Userland API Hooking, Again: Novel Attacks and a Principled Defense MethodDetection of Intrusions and Malware, and Vulnerability Assessment10.1007/978-3-031-64171-8_8(150-173)Online publication date: 9-Jul-2024
https://doi.org/10.1007/978-3-031-64171-8_8
Pham TOgawa M(2024)Original Entry Point Detection Based on Graph SimilarityFoundations and Practice of Security10.1007/978-3-031-57537-2_22(355-371)Online publication date: 25-Apr-2024
https://doi.org/10.1007/978-3-031-57537-2_22
Kim HKim SLee JJee KCha SCalandrino JTroncoso C(2023)Reassembly is hardProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620320(1469-1486)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620320
Gori GRinieri LAl Sadi AMelis ACallegati FPrandini M(2023)GRAPH4: A Security Monitoring Architecture Based on Data Plane Anomaly Detection Metrics Calculated over Attack GraphsFuture Internet10.3390/fi1511036815:11(368)Online publication date: 15-Nov-2023
https://doi.org/10.3390/fi15110368
Li YKang FShu HXiong XZhao YSun R(2023)APIASO: A Novel API Call Obfuscation Technique Based on Address Space ObscurityApplied Sciences10.3390/app1316905613:16(9056)Online publication date: 8-Aug-2023
https://doi.org/10.3390/app13169056
Zhang PWu CPeng MZeng KYu DLai YKang YWang WWang ZDubach CBruening DHardekopf B(2023)Khaos: The Impact of Inter-procedural Code Obfuscation on Binary Diffing TechniquesProceedings of the 21st ACM/IEEE International Symposium on Code Generation and Optimization10.1145/3579990.3580007(55-67)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3579990.3580007
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents