skip to main content
research-article

Fine-grained user-space security through virtualization

Published:09 March 2011Publication History
Skip Abstract Section

Abstract

This paper presents an approach to the safe execution of applications based on software-based fault isolation and policy-based system call authorization. A running application is encapsulated in an additional layer of protection using dynamic binary translation in user-space. This virtualization layer dynamically recompiles the machine code and adds multiple dynamic security guards that verify the running code to protect and contain the application.

The binary translation system redirects all system calls to a policy-based system call authorization framework. This interposition framework validates every system call based on the given arguments and the location of the system call. Depending on the user-loadable policy and an extensible handler mechanism the framework decides whether a system call is allowed, rejected, or redirect to a specific user-space handler in the virtualization layer.

This paper offers an in-depth analysis of the different security guarantees and a performance analysis of libdetox, a prototype of the full protection platform. The combination of software-based fault isolation and policy-based system call authorization imposes only low overhead and is therefore an attractive option to encapsulate and sandbox applications to improve host security.

References

  1. Acharya, A., and Raje, M. MAPbox: using parameterized behavior classes to confine untrusted applications. In SSYM'00: Proceedings of the 9th conference on USENIX Security Symposium (2000). Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Alexandrov, A., Kmiec, P., and Schauser, K. Consh: Confined execution environment for internet computations, 1999.Google ScholarGoogle Scholar
  3. Baratloo, A., Singh, N., and Tsai, T. Transparent run-time defense against stack smashing attacks. In ATEC '00: Proceedings of the annual conference on USENIX Annual Technical Conference (2000). Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. Xen and the art of virtualization. In SOSP '03 (New York, NY, USA, 2003), pp. 164--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Bauer, M. Paranoid penguin: an introduction to novell apparmor. Linux J. 2006, 148 (2006), 13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Bellard, F. QEMU, a fast and portable dynamic translator. In ATEC '05 (Berkeley, CA, USA, 2005), pp. 41--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bhatkar, E., Duvarney, D. C., and Sekar, R. Address obfuscation: an efficient approach to combat a broad range of memory error exploits. In Proceedings of the 12th USENIX Security Symposium (2003), pp. 105--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Bhatkar, S., Bhatkar, E., Sekar, R., and Duvarney, D. C. Efficient techniques for comprehensive protection from memory error exploits. In Proceedings of the 14th USENIX Security Symposium (2005). Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Bruening, D., Duesterwald, E., and Amarasinghe, S. Design and implementation of a dynamic optimization framework for Windows. In ACM Workshop Feedback-directed Dyn. Opt. (FDDO-4) (2001).Google ScholarGoogle Scholar
  10. Bruening, D., Garnett, T., and Amarasinghe, S. An infrastructure for adaptive dynamic optimization. In CGO '03 (Washington, DC, USA, 2003), pp. 265--275. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Bugnion, E. Dynamic binary translator with a system and method for updating and maintaining coherency of a translation cache. US Patent 6704925, March 2004.Google ScholarGoogle Scholar
  12. Cowan, C., Barringer, M., Beattie, S., Kroah-Hartman, G., Frantzen, M., and Lokier, J. Formatguard: automatic protection from printf format string vulnerabilities. In SSYM'01: Proceedings of the 10th conference on USENIX Security Symposium (2001). Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Cowan, C., Beattie, S., Kroah-Hartman, G., Pu, C., Wagle, P., and Gligor, V. Subdomain: Parsimonious server security. In LISA '00: Proceedings of the 14th USENIX conference on System administration (2000). Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Cowan, C., Pu, C., Maier, D., Hintony, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., and Zhang, Q. Stackguard: automatic adaptive detection and prevention of buffer-overflow attacks. In SSYM'98: Proceedings of the 7th conference on USENIX Security Symposium (1998). Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Devine, S. W., Bugnion, E., and Rosenblum, M. Virtualization system including a virtual machine monitor for a computer with a segmented architecture. US Patent 6397242.Google ScholarGoogle Scholar
  16. Fetzer, C., and Suesskraut, M. Switchblade: enforcing dynamic personalized system call models. In Eurosys '08: Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems 2008 (New York, NY, USA, 2008), ACM, pp. 273--286. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Ford, B., and Cox, R. Vx32: lightweight user-level sandboxing on the x86. In ATC'08: USENIX 2008 Annual Technical Conference on Annual Technical Conference (Berkeley, CA, USA, 2008), USENIX Association, pp. 293--306. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Garfinkel, T. Traps and pitfalls: Practical problems in system call interposition based security tools. In In Proc. Network and Distributed Systems Security Symposium (2003), pp. 163--176.Google ScholarGoogle Scholar
  19. Garfinkel, T., Pfaff, B., and Rosenblum, M. Ostia: A delegating architecture for secure system call interposition. In Proc. Network and Distributed Systems Security Symposium (February 2004).Google ScholarGoogle Scholar
  20. Garfinkel, T., and Rosenblum, M. A virtual machine introspection based architecture for intrusion detection. In Proc. Network and Distributed Systems Security Symposium (February 2003).Google ScholarGoogle Scholar
  21. Garg, M. Sysenter based system call mechanism in linux 2.6 (http://manugarg.googlepages.com/systemcallinlinux2\_6.html).Google ScholarGoogle Scholar
  22. Goldberg, I., Wagner, D., Thomas, R., and Brewer, E. A. A secure environment for untrusted helper applications: Confining the wily hacker. In Proceedings of the 6th Usenix Security Symposium (1996). Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Hazelwood, K., and Smith, M. D. Managing bounded code caches in dynamic binary optimization systems. TACO '06 3, 3 (2006), 263--294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Hiroaki, E., and Kunikazu, Y. propolice : Improved stack-smashing attack detection. IPSJ SIG Notes 2001, 75 (2001-07--25), 181--188.Google ScholarGoogle Scholar
  25. Ho, A., Fetterman, M., Clark, C., Warfield, A., and Hand, S. Practical taint-based protection using demand emulation. vol. 40, pp. 29--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Kiriansky, V., Bruening, D., and Amarasinghe, S. P. Secure execution via program shepherding. In Proceedings of the 11th USENIX Security Symposium (Berkeley, CA, USA, 2002), USENIX Association, pp. 191--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Liang, Z., Sun, W., Venkatakrishnan, V. N., and Sekar, R. Alcatraz: An isolated environment for experimenting with untrusted software. ACM Trans. Inf. Syst. Secur. 12, 3 (2009), 1--37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Luk, C.-K., Cohn, R., Muth, R., Patil, H., Klauser, A., Lowney, G., Wallace, S., Reddi, V. J., and Hazelwood, K. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI '05 (New York, NY, USA, 2005), pp. 190--200. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. McCamant, S., and Morrisett, G. Evaluating SFI for a CISC architecture. In 15th USENIX Security Symposium (Vancouver, BC, Canada, August 2-4, 2006), pp. 209--224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. PaX-Team. PaX ASLR (Address Space Layout Randomization). http://pax.grsecurity.net/docs/aslr.txt.Google ScholarGoogle Scholar
  31. Payer, M., and Gross, T. Requirements for fast binary translation. In 2nd Workshop on Architectural and Microarchitectural Support for Binary Translation (2009).Google ScholarGoogle Scholar
  32. Payer, M., and Gross, T. R. Generating low-overhead dynamic binary translators. In SYSTOR'10 (2010). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Provos, N. Improving host security with system call policies. In SSYM'03: Proceedings of the 12th conference on USENIX Security Symposium (Berkeley, CA, USA, 2003), USENIX Association, pp. 18--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Scott, K., and Davidson, J. Strata: A software dynamic translation infrastructure. Tech. rep., Charlottesville, VA, USA, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Scott, K., and Davidson, J. Safe virtual execution using software dynamic translation. Computer Security Applications Conference, Annual 0 (2002), 209. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Shacham, H., Page, M., Pfaff, B., Goh, E.-J., Modadugu, N., and Boneh, D. On the effectiveness of address-space randomization. In CCS'04 (2004), pp. 298--307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Sridhar, S., Shapiro, J. S., and Bungale, P. P. HDTrans: a low-overhead dynamic translator. SIGARCH Comput. Archit. News 35, 1 (2007), 135--140. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Sridhar, S., Shapiro, J. S., Northup, E., and Bungale, P. P. HDTrans: an open source, low-level dynamic instrumentation system. In VEE '06 (New York, NY, USA, 2006), pp. 175--185. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Wahbe, R., Lucco, S., Anderson, T. E., and Graham, S. L. Efficient software-based fault isolation. In SOSP'93 (New York, NY, USA, 1993), ACM, pp. 203--216. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Watson, R. N. M. Exploiting concurrency vulnerabilities in system call wrappers. In WOOT '07: Proceedings of the first USENIX workshop on Offensive Technologies (2007). Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Wright, C., Cowan, C., Smalley, S., Morris, J., and Kroah-Hartman, G. Linux security modules: General security support for the linux kernel. In Proceedings of the 11th USENIX Security Symposium (2002). Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Yee, B., Sehr, D., Dardyk, G., Chen, J. B., Muth, R., Ormandy, T., Okasaka, S., Narula, N., and Fullagar, N. Native client: A sandbox for portable, untrusted x86 native code. IEEE Symposium on Security and Privacy (2009), 79--93. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Fine-grained user-space security through virtualization

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!