skip to main content
research-article

Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction

Published:05 March 2010Publication History
Skip Abstract Section

Abstract

An alarming trend in recent malware incidents is that they are armed with stealthy techniques to detect, evade, and subvert malware detection facilities of the victim. On the defensive side, a fundamental limitation of traditional host-based antimalware systems is that they run inside the very hosts they are protecting (“in-the-box”), making them vulnerable to counter detection and subversion by malware. To address this limitation, recent solutions based on virtual machine (VM) technologies advocate placing the malware detection facilities outside of the protected VM (“out-of-the-box”). However, they gain tamper resistance at the cost of losing the internal semantic view of the host, which is enjoyed by “in-the-box” approaches. This poses a technical challenge known as the semantic gap.

In this article, we present the design, implementation, and evaluation of VMwatcher—an “out-of-the-box” approach that overcomes the semantic gap challenge. A new technique called guest view casting is developed to reconstruct internal semantic views (e.g., files, processes, and kernel modules) of a VM nonintrusively from the outside. More specifically, the new technique casts semantic definitions of guest OS data structures and functions on virtual machine monitor (VMM)-level VM states, so that the semantic view can be reconstructed. Furthermore, we extend guest view casting to reconstruct details of system call events (e.g., the process that makes the system call as well as the system call number, parameters, and return value) in the VM, enriching the semantic view. With the semantic gap effectively narrowed, we identify three unique malware detection and monitoring capabilities: (i) view comparison-based malware detection and its demonstration in rootkit detection; (ii) “out-of-the-box” deployment of off-the-shelf anti malware software with improved detection accuracy and tamper-resistance; and (iii) nonintrusive system call monitoring for malware and intrusion behavior observation. We have implemented a proof-of-concept VMwatcher prototype on a number of VMM platforms. Our evaluation experiments with real-world malware, including elusive kernel-level rootkits, demonstrate VMwatcher's practicality and effectiveness.

References

  1. Adore-ng Rootkit. 2004. Homepage. http://stealth.openwall.net/rootkits/.Google ScholarGoogle Scholar
  2. Agobot. 2004. Description. http://www.f-secure.com/v-descs/agobot.shtml.Google ScholarGoogle Scholar
  3. Anagnostakis, K. G., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., and Keromytis, A. D. 2005. Detecting targeted attacks using shadow honey-pots. In Proceedings of the 14th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Apache. 2007. The Apache HTTP Server Project. http://httpd.apache.org.Google ScholarGoogle Scholar
  5. Arbaugh, W. A., Farbert, D. J., and Smith, J. M. 1997. A secure and reliable bootstrap architecture. In Proceedings of the 1997 IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., A. Ho, R. N., Pratt, I., and Warfield, A. 2003. Xen and the art of virtualization. In Proceedings of the 19th ACM Symposium on Operating Systems Principles. ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bellard, F. 2005. QEMU, a fast and portable dynamic translator. In Proceedings of USENIX Annual Technical Conference 2005 (FREENIX Track). USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Bellard, F. 2006. QEMU accelerator user documentation. http://fabrice.bellard.free.fr/qemu/kqemudoc.html.Google ScholarGoogle Scholar
  9. Blacklight. 2007. Homepage. http://www.f-secure.com/blacklight/.Google ScholarGoogle Scholar
  10. Bryant, E., Early, J., Gopalakrishna, R., Roth, G., Spafford, E. H., Watson, K., Williams, P., and Yost, S. 2003. Poly2 Paradigm: A secure network service architecture. In Proceedings of the 19th Annual Computer Security Applications Conference. IEEE, Los Alamitos, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Bugcheck. 2006. GREPEXEC: Grepping executive objects from pool memory. http://www. uninformed.org/?v=4&a=2&t=sumry.Google ScholarGoogle Scholar
  12. Chen, P. M. and Noble, B. D. 2001. When virtual is better than real. HotOS VIII, Schoss Elmau, Germany. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Corey, J. 2004. Local honey-pot identification. Phrack 62, 7.Google ScholarGoogle Scholar
  14. Dike, J. 2002. User mode Linux. http://user-mode-linux.sourceforge.net.Google ScholarGoogle Scholar
  15. Dornseif, M., Holz, T., and Klein, C. 2004. NoSEBrEaK - Attacking honey-nets. In Proceedings of the 5th Annual IEEE Information Assurance Workshop. IEEE, Los Alamitos, CA.Google ScholarGoogle Scholar
  16. Dunlap, G. W., King, S. T., Cinar, S., Basrai, M. A., and Chen, P. M. 2002. ReVirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI). USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Fu. 2005. Rootkit. http://www.rootkit.com/board_project_fused.php?did=proj12.Google ScholarGoogle Scholar
  18. Garfinkel, T., Pfaff, B., Chow, J., Rosenblum, M., and Boneh, D. 2003. Terra: A virtual machine-based platform for trusted computing. In Proceedings of the 2003 Symposium on Operating Systems Principles (SOSP'03). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Garfinkel, T. and Rosenblum, M. 2003. A virtual machine introspection-based architecture for intrusion detection. In Proceedings of the 2003 Network and Distributed System Security Symposium. IEEE, Los Alamitos, CA.Google ScholarGoogle Scholar
  20. Honeynet. 2008. Homepage. http://www.honeynet.org.Google ScholarGoogle Scholar
  21. hxdef. http://hxdef.czweb.org.Google ScholarGoogle Scholar
  22. Jiang, X., Wang, X., and Xu, D. 2007. Stealthy malware detection through VMM-based “out-of-the-box” semantic view reconstruction. In Proceedings of the 14th ACM Conference on Computer and Communications Security (CCS'07). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Jiang, X. and Xu, D. 2004. Collapsar: A VM-based architecture for network attack detention center. In Proceedings of the 13th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Jiang, X., Xu, D., Wang, H. J., and Spafford, E. H. 2005. Virtual playgrounds for worm behavior investigation. In Proceedings of 8th International Symposium on Recent Advances in Intrusion Detection (RAID'05). Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Joshi, A., King, S. T., Dunlap, G. W., and Chen, P. M. 2005. Detecting past and present intrusions through vulnerability-specific predicates. In Proceedings of the 2005 Symposium on Operating Systems Principles (SOSP'05). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kim, G. H. and Spafford, E. H. 1994. Experiences with tripwire: Using integrity checkers for intrusion detection. In Proceedings of the Systems Administration, Networking and Security Conference III. USENIX, Berkeley, CA.Google ScholarGoogle Scholar
  27. King, S. T. and Chen, P. M. 2003. Backtracking intrusions. In Proceedings of the 2003 Symposium on Operating Systems Principles (SOSP'03). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. King, S. T., Chen, P. M., Wang, Y.-M., Verbowski, C., Wang, H. J., and Lorch, J. R. 2006. SubVirt: Implementing malware with virtual machines. In Proceedings of the 2006 IEEE Symposium on Security and Privacy. IEEE, Los Alamitos, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. King, S. T., Dunlap, G. W., and Chen, P. M. 2005. Debugging operating systems with time-traveling virtual machines. In Proceedings of the 2005 Annual USENIX Technical Conference. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Klein, T. 2003. Scooby Doo-VMware Fingerprint Suite. http://www.trapkit.de/research/vmm/scoopydoo/.Google ScholarGoogle Scholar
  31. Koju, T., Takada, S., and Doi, N. 2005. An efficient and generic reversible debugger using the virtual machine based approach. In Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments. ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Kortchinsky, K. 2004. Honey-pots: Counter measures to VMware fingerprinting. http://seclists. org/lists/honeypots/2004/Jan-Mar/0015.html.Google ScholarGoogle Scholar
  33. Kourai, K. and Chiba, S. 2005. HyperSpector: Virtual distributed monitoring environments for secure intrusion detection. In Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments. ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Lion. 2001. Lion worm. http://www.sans.com/y2k/lion.htm.Google ScholarGoogle Scholar
  35. Liston, T. and Skoudis, E. 2006. On the cutting edge: Thwarting virtual machine detection. http://handlers.sans.org/tliston/ThwartingVMDetection Liston Skoudis.pdf.Google ScholarGoogle Scholar
  36. Meushaw, R. and Simard, D. 2000. NetTop: Commercial technology in high assurance applications. Tech Trend Notes.Google ScholarGoogle Scholar
  37. Microsoft. 2003. Volume shadow copy service. http://technet2.microsoft.com/WindowsServer/en/library/2b0d2457-b7d8-42c3-b6c9-59c145b7765f1033.mspx?mfr=true.Google ScholarGoogle Scholar
  38. Miller, J. V. 2003. SHV4 root-kit analysis. https://tms.symantec.com/members/AnalystReports/030929-Analysis-SHV4Rootkit.pdf.Google ScholarGoogle Scholar
  39. NTRootkit. http://www.megasecurity.org/Tools/Nt rootkit all.html.Google ScholarGoogle Scholar
  40. Paxson, V. 1999. Bro: A system for detecting network intruders in real-time. Comput. Networks 31, 23-24, 2345--2463. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Pennington, A. G., Strunk, J. D., Griffin, J. L., Soules, C. A. N., Goodson, G. R., and Ganger., G. R. 2003. Storage-based intrusion detection: Watching storage activity for suspicious behavior. In Proceedings of the 12th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Petroni, N., Fraser, T., Walters, A., and Arbaugh, W. 2006. An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In Proceedings of the 15th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Petroni, N. L., Fraser, T., Molina, J., and Arbaugh, W. A. 2004. Copilot - A coprocessor-based kernel runtime integrity monitor. In Proceedings of the 13th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Provos, N. 2003. Improving host security with system call policies. In Proceedings of the 12th USENIX Security Symposium. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Rbot. http://research.sunbelt-software.com/threatdisplay.aspx?name=Rbot&threatid=14953.Google ScholarGoogle Scholar
  46. Riley, R., Jiang, X., and Xu, D. 2008. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of 11th International Symposium on Recent Advances in Intrusion Detection (RAID'08). Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. RootKitRevealer. 2007. RootkitRevealer. http://www.microsoft.com/technet/sysinternals/utilities/RootkitRevealer.mspx.Google ScholarGoogle Scholar
  48. Rutkowska, J. 2004. Red pill: Detect VMM using (almost) one CPU instruction. http://invisiblethings.org/papers/redpill.html.Google ScholarGoogle Scholar
  49. Rutkowska, J. 2006. Subverting vista kernel for fun and profit. http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Rutkowska.pdf.Google ScholarGoogle Scholar
  50. Sailer, R., Valdez, E., Jaeger, T., Perez, R., Van Doorn, L., Griffin, J. L., and Berger, S. 2005. sHype: Secure hypervisor approach to trusted virtualized systems. IBM Research Report RC23511.Google ScholarGoogle Scholar
  51. Sebek. 2008. http://www.honeynet.org/tools/sebek/.Google ScholarGoogle Scholar
  52. Secunia. 2003. Linux kernel Ptrace privilege escalation vulnerability. http://www.secunia.com/advisories/8337/.Google ScholarGoogle Scholar
  53. Seshadri, A., Luk, M., Qu, N., and Perrig, A. 2007. SecVisor: A tiny hypervisor to guarantee lifetime kernel code integrity for commodity OSes. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP'07). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Snort. 2008. Homepage. http://www.snort.org.Google ScholarGoogle Scholar
  55. SucKit Rootkit. 2001. Linux on-the-fly kernel patching without LKM. http://www.phrack.com/issues.html?issue=58&id=7#articleGoogle ScholarGoogle Scholar
  56. Trango. 2008. The Real-Time Embedded Hypervisor. http://www.trango-systems.com/.Google ScholarGoogle Scholar
  57. UnixBench. 2007. UnixBench. http://www.tux.org/pub/tux/benchmarks/System/unixbench.Google ScholarGoogle Scholar
  58. VMware. 2008. Homepage. http://www.vmware.com/.Google ScholarGoogle Scholar
  59. Wang, Y.-M., Beck, D., Vo, B., Roussev, R., and Verbowski, C. 2005. Detecting stealth software with Strider GhostBuster. In Proceedings of the 2005 International Conference on Dependable Systems and Networks. IEEE, Los Alamitos, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Weaver, N., Paxson, V., and Gonzalez, J. 2007. The Shunt: An FPGA-based accelerator for network intrusion prevention. In Proceedings of the International Symposium on Field Programmable Gate Arrays (FPGA'07). ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Whitaker, A., Cox, R. S., and Gribble, S. D. 2004. Configuration debugging as search: Finding the needle in the haystack. In Proceedings of USENIX OSDI 2004. USENIX, Berkeley, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Xen. 2004. Interface manual. http://www.xensource.com/files/xen interface.pdf, 2004.Google ScholarGoogle Scholar
  63. Zovi, D. D. 2006. Hardware virtualization based rootkits. http://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Zovi.pdf.Google ScholarGoogle Scholar

Index Terms

  1. Stealthy malware detection and monitoring through VMM-based “out-of-the-box” semantic view reconstruction

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Information and System Security
          ACM Transactions on Information and System Security  Volume 13, Issue 2
          February 2010
          230 pages
          ISSN:1094-9224
          EISSN:1557-7406
          DOI:10.1145/1698750
          Issue’s Table of Contents

          Copyright © 2010 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 5 March 2010
          • Accepted: 1 November 2008
          • Revised: 1 June 2008
          • Received: 1 February 2008
          Published in tissec Volume 13, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!