Abstract
An alarming trend in recent malware incidents is that they are armed with stealthy techniques to detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based antimalware systems is that they run inside the very hosts they are protecting (“in-the-box”), making them vulnerable to counter detection and subversion by malware. To address this limitation, recent solutions based on virtual machine (VM) technologies advocate placing the malware detection facilities outside of the protected VM (“out-of-the-box”). However, they gain tamper resistance at the cost of losing the internal semantic view of the host, which is enjoyed by “in-the-box” approaches. This poses a technical challenge known as the semantic gap.
In this article, we present the design, implementation, and evaluation of VMwatcher—an “out-of-the-box” approach that overcomes the semantic gap challenge. A new technique called guest view casting is developed to reconstruct internal semantic views (e.g., files, processes, and kernel modules) of a VM nonintrusively from the outside. More specifically, the new technique casts semantic definitions of guest OS data structures and functions on virtual machine monitor (VMM)-level VM states, so that the semantic view can be reconstructed. Furthermore, we extend guest view casting to reconstruct details of system call events (e.g., the process that makes the system call as well as the system call number, parameters, and return value) in the VM, enriching the semantic view. With the semantic gap effectively narrowed, we identify three unique malware detection and monitoring capabilities: (i) view comparison-based malware detection and its demonstration in rootkit detection; (ii) “out-of-the-box” deployment of off-the-shelf anti malware software with improved detection accuracy and tamper-resistance; and (iii) nonintrusive system call monitoring for malware and intrusion behavior observation. We have implemented a proof-of-concept VMwatcher prototype on a number of VMM platforms. Our evaluation experiments with real-world malware, including elusive kernel-level rootkits, demonstrate VMwatcher's practicality and effectiveness.
- Adore-ng Rootkit. 2004. Homepage. http://stealth.openwall.net/rootkits/.Google Scholar
- Agobot. 2004. Description. http://www.f-secure.com/v-descs/agobot.shtml.Google Scholar
- Anagnostakis, K. G., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., and Keromytis, A. D. 2005. Detecting targeted attacks using shadow honey-pots. In Proceedings of the 14th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Apache. 2007. The Apache HTTP Server Project. http://httpd.apache.org.Google Scholar
- Arbaugh, W. A., Farbert, D. J., and Smith, J. M. 1997. A secure and reliable bootstrap architecture. In Proceedings of the 1997 IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA. Google Scholar
Digital Library
- Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., A. Ho, R. N., Pratt, I., and Warfield, A. 2003. Xen and the art of virtualization. In Proceedings of the 19th ACM Symposium on Operating Systems Principles. ACM, New York. Google Scholar
Digital Library
- Bellard, F. 2005. QEMU, a fast and portable dynamic translator. In Proceedings of USENIX Annual Technical Conference 2005 (FREENIX Track). USENIX, Berkeley, CA. Google Scholar
Digital Library
- Bellard, F. 2006. QEMU accelerator user documentation. http://fabrice.bellard.free.fr/qemu/kqemudoc.html.Google Scholar
- Blacklight. 2007. Homepage. http://www.f-secure.com/blacklight/.Google Scholar
- Bryant, E., Early, J., Gopalakrishna, R., Roth, G., Spafford, E. H., Watson, K., Williams, P., and Yost, S. 2003. Poly2 Paradigm: A secure network service architecture. In Proceedings of the 19th Annual Computer Security Applications Conference. IEEE, Los Alamitos, CA. Google Scholar
Digital Library
- Bugcheck. 2006. GREPEXEC: Grepping executive objects from pool memory. http://www. uninformed.org/?v=4&a=2&t=sumry.Google Scholar
- Chen, P. M. and Noble, B. D. 2001. When virtual is better than real. HotOS VIII, Schoss Elmau, Germany. Google Scholar
Digital Library
- Corey, J. 2004. Local honey-pot identification. Phrack 62, 7.Google Scholar
- Dike, J. 2002. User mode Linux. http://user-mode-linux.sourceforge.net.Google Scholar
- Dornseif, M., Holz, T., and Klein, C. 2004. NoSEBrEaK - Attacking honey-nets. In Proceedings of the 5th Annual IEEE Information Assurance Workshop. IEEE, Los Alamitos, CA.Google Scholar
- Dunlap, G. W., King, S. T., Cinar, S., Basrai, M. A., and Chen, P. M. 2002. ReVirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI). USENIX, Berkeley, CA. Google Scholar
Digital Library
- Fu. 2005. Rootkit. http://www.rootkit.com/board_project_fused.php?did=proj12.Google Scholar
- Garfinkel, T., Pfaff, B., Chow, J., Rosenblum, M., and Boneh, D. 2003. Terra: A virtual machine-based platform for trusted computing. In Proceedings of the 2003 Symposium on Operating Systems Principles (SOSP'03). ACM, New York. Google Scholar
Digital Library
- Garfinkel, T. and Rosenblum, M. 2003. A virtual machine introspection-based architecture for intrusion detection. In Proceedings of the 2003 Network and Distributed System Security Symposium. IEEE, Los Alamitos, CA.Google Scholar
- Honeynet. 2008. Homepage. http://www.honeynet.org.Google Scholar
- hxdef. http://hxdef.czweb.org.Google Scholar
- Jiang, X., Wang, X., and Xu, D. 2007. Stealthy malware detection through VMM-based “out-of-the-box” semantic view reconstruction. In Proceedings of the 14th ACM Conference on Computer and Communications Security (CCS'07). ACM, New York. Google Scholar
Digital Library
- Jiang, X. and Xu, D. 2004. Collapsar: A VM-based architecture for network attack detention center. In Proceedings of the 13th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Jiang, X., Xu, D., Wang, H. J., and Spafford, E. H. 2005. Virtual playgrounds for worm behavior investigation. In Proceedings of 8th International Symposium on Recent Advances in Intrusion Detection (RAID'05). Springer, Berlin. Google Scholar
Digital Library
- Joshi, A., King, S. T., Dunlap, G. W., and Chen, P. M. 2005. Detecting past and present intrusions through vulnerability-specific predicates. In Proceedings of the 2005 Symposium on Operating Systems Principles (SOSP'05). ACM, New York. Google Scholar
Digital Library
- Kim, G. H. and Spafford, E. H. 1994. Experiences with tripwire: Using integrity checkers for intrusion detection. In Proceedings of the Systems Administration, Networking and Security Conference III. USENIX, Berkeley, CA.Google Scholar
- King, S. T. and Chen, P. M. 2003. Backtracking intrusions. In Proceedings of the 2003 Symposium on Operating Systems Principles (SOSP'03). ACM, New York. Google Scholar
Digital Library
- King, S. T., Chen, P. M., Wang, Y.-M., Verbowski, C., Wang, H. J., and Lorch, J. R. 2006. SubVirt: Implementing malware with virtual machines. In Proceedings of the 2006 IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA. Google Scholar
Digital Library
- King, S. T., Dunlap, G. W., and Chen, P. M. 2005. Debugging operating systems with time-traveling virtual machines. In Proceedings of the 2005 Annual USENIX Technical Conference. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Klein, T. 2003. Scooby Doo-VMware Fingerprint Suite. http://www.trapkit.de/research/vmm/scoopydoo/.Google Scholar
- Koju, T., Takada, S., and Doi, N. 2005. An efficient and generic reversible debugger using the virtual machine based approach. In Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments. ACM, New York. Google Scholar
Digital Library
- Kortchinsky, K. 2004. Honey-pots: Counter measures to VMware fingerprinting. http://seclists. org/lists/honeypots/2004/Jan-Mar/0015.html.Google Scholar
- Kourai, K. and Chiba, S. 2005. HyperSpector: Virtual distributed monitoring environments for secure intrusion detection. In Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments. ACM, New York. Google Scholar
Digital Library
- Lion. 2001. Lion worm. http://www.sans.com/y2k/lion.htm.Google Scholar
- Liston, T. and Skoudis, E. 2006. On the cutting edge: Thwarting virtual machine detection. http://handlers.sans.org/tliston/ThwartingVMDetection Liston Skoudis.pdf.Google Scholar
- Meushaw, R. and Simard, D. 2000. NetTop: Commercial technology in high assurance applications. Tech Trend Notes.Google Scholar
- Microsoft. 2003. Volume shadow copy service. http://technet2.microsoft.com/WindowsServer/en/library/2b0d2457-b7d8-42c3-b6c9-59c145b7765f1033.mspx?mfr=true.Google Scholar
- Miller, J. V. 2003. SHV4 root-kit analysis. https://tms.symantec.com/members/AnalystReports/030929-Analysis-SHV4Rootkit.pdf.Google Scholar
- NTRootkit. http://www.megasecurity.org/Tools/Nt rootkit all.html.Google Scholar
- Paxson, V. 1999. Bro: A system for detecting network intruders in real-time. Comput. Networks 31, 23-24, 2345--2463. Google Scholar
Digital Library
- Pennington, A. G., Strunk, J. D., Griffin, J. L., Soules, C. A. N., Goodson, G. R., and Ganger., G. R. 2003. Storage-based intrusion detection: Watching storage activity for suspicious behavior. In Proceedings of the 12th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Petroni, N., Fraser, T., Walters, A., and Arbaugh, W. 2006. An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In Proceedings of the 15th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Petroni, N. L., Fraser, T., Molina, J., and Arbaugh, W. A. 2004. Copilot - A coprocessor-based kernel runtime integrity monitor. In Proceedings of the 13th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Provos, N. 2003. Improving host security with system call policies. In Proceedings of the 12th USENIX Security Symposium. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Rbot. http://research.sunbelt-software.com/threatdisplay.aspx?name=Rbot&threatid=14953.Google Scholar
- Riley, R., Jiang, X., and Xu, D. 2008. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of 11th International Symposium on Recent Advances in Intrusion Detection (RAID'08). Springer, Berlin. Google Scholar
Digital Library
- RootKitRevealer. 2007. RootkitRevealer. http://www.microsoft.com/technet/sysinternals/utilities/RootkitRevealer.mspx.Google Scholar
- Rutkowska, J. 2004. Red pill: Detect VMM using (almost) one CPU instruction. http://invisiblethings.org/papers/redpill.html.Google Scholar
- Rutkowska, J. 2006. Subverting vista kernel for fun and profit. http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Rutkowska.pdf.Google Scholar
- Sailer, R., Valdez, E., Jaeger, T., Perez, R., Van Doorn, L., Griffin, J. L., and Berger, S. 2005. sHype: Secure hypervisor approach to trusted virtualized systems. IBM Research Report RC23511.Google Scholar
- Sebek. 2008. http://www.honeynet.org/tools/sebek/.Google Scholar
- Secunia. 2003. Linux kernel Ptrace privilege escalation vulnerability. http://www.secunia.com/advisories/8337/.Google Scholar
- Seshadri, A., Luk, M., Qu, N., and Perrig, A. 2007. SecVisor: A tiny hypervisor to guarantee lifetime kernel code integrity for commodity OSes. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP'07). ACM, New York. Google Scholar
Digital Library
- Snort. 2008. Homepage. http://www.snort.org.Google Scholar
- SucKit Rootkit. 2001. Linux on-the-fly kernel patching without LKM. http://www.phrack.com/issues.html?issue=58&id=7#articleGoogle Scholar
- Trango. 2008. The Real-Time Embedded Hypervisor. http://www.trango-systems.com/.Google Scholar
- UnixBench. 2007. UnixBench. http://www.tux.org/pub/tux/benchmarks/System/unixbench.Google Scholar
- VMware. 2008. Homepage. http://www.vmware.com/.Google Scholar
- Wang, Y.-M., Beck, D., Vo, B., Roussev, R., and Verbowski, C. 2005. Detecting stealth software with Strider GhostBuster. In Proceedings of the 2005 International Conference on Dependable Systems and Networks. IEEE, Los Alamitos, CA. Google Scholar
Digital Library
- Weaver, N., Paxson, V., and Gonzalez, J. 2007. The Shunt: An FPGA-based accelerator for network intrusion prevention. In Proceedings of the International Symposium on Field Programmable Gate Arrays (FPGA'07). ACM, New York. Google Scholar
Digital Library
- Whitaker, A., Cox, R. S., and Gribble, S. D. 2004. Configuration debugging as search: Finding the needle in the haystack. In Proceedings of USENIX OSDI 2004. USENIX, Berkeley, CA. Google Scholar
Digital Library
- Xen. 2004. Interface manual. http://www.xensource.com/files/xen interface.pdf, 2004.Google Scholar
- Zovi, D. D. 2006. Hardware virtualization based rootkits. http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Zovi.pdf.Google Scholar
Index Terms
Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction
Recommendations
Stealthy malware detection through vmm-based "out-of-the-box" semantic view reconstruction
CCS '07: Proceedings of the 14th ACM conference on Computer and communications securityAn alarming trend in malware attacks is that they are armed with stealthy techniques to detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based anti-malware systems ...
Automated multi-level malware detection system based on reconstructed semantic view of executables using machine learning techniques at VMM
In order to fulfill the requirements like stringent timing restraints and demand on resources, CyberPhysical System (CPS) must deploy on the virtualized environment such as cloud computing. To protect Virtual Machines (VMs) in which CPSs are functioning ...
Automated containment of rootkits attacks
Rootkit attacks are a serious threat to computer systems. Packaged with other malwares such as worms, viruses and spyware, rootkits pose a more potent threat than ever before by allowing malware to evade detection. In the absence of appropriate tools to ...






Comments