skip to main content
research-article

Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation

Authors Info & Claims
Published:14 March 2015Publication History
Skip Abstract Section

Abstract

Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a small isolated kernel within a traditional monolithic kernel. The "nested kernel" interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware while allowing the entire operating system, including untrusted components, to operate at the highest hardware privilege level by write-protecting MMU translations and de-privileging the untrusted part of the kernel. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels by introducing write-mediation and write-logging services to protect critical system data structures. Performance of the nested kernel prototype shows modest overheads: <1% average for Apache and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.

References

  1. AMD64 architecture programmers manual volume 2: System programming. Manual, Advancd Micro Devices, 2006.Google ScholarGoogle Scholar
  2. Intel 64 and IA-32 architectures software developers manual. Manual 325384-051US, Intel, June 2014.Google ScholarGoogle Scholar
  3. M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and M. Young. Mach: A new kernel foundation for UNIX development. In Proceedings of the USENIX Annual Technical Conference, USENIX ATC'10, pages 93--112, Altanta, GA, USA, 1986. USENIX Association.Google ScholarGoogle Scholar
  4. M. Aiken, M. Fhndrich, C. Hawblitzel, G. Hunt, and J. Larus. Deconstructing process isolation. In Proceedings of the 2006 Workshop on Memory System Performance and Correctness, MSPC '06, pages 1--10, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. argp and Karl. Exploiting UMA, FreeBSD's kernel memory allocator..:: Phrack Magazine ::., 0x0d(0x42), Nov. 2009.Google ScholarGoogle Scholar
  6. S. Bahram, X. Jiang, Z. Wang, M. Grace, J. Li, D. Srinivasan, J. Rhee, and D. Xu. DKSM: Subverting virtual machine introspection for fun and profit. In Proceedings of the 2010 29th IEEE Symposium on Reliable Distributed Systems, SRDS '10, pages 82--91, Washington, DC, USA, 2010. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 164--177, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. A. Belay, A. Bittau, A. Mashtizadeh, D. Terei, D. Mazires, and C. Kozyrakis. Dune: Safe user-level access to privileged CPU features. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI'12, pages 335--348, Berkeley, CA, USA, 2012. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. Extensibility safety and performance in the SPIN operating system. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 267--283, New York, NY, USA, 1995. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. Castro, M. Costa, J.-P. Martin, M. Peinado, P. Akritidis, A. Donnelly, P. Barham, and R. Black. Fast byte-granularity software fault isolation. In Proceedings of the ACM SIGOPS 22nd symposium on Operating Systems Principles, SOSP '09, pages 45--58, New York, NY, USA, 2009. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. S. Chen, J. Xu, E. C. Sezer, P. Gauriar, and R. K. Iyer. Non-control-data attacks are realistic threats. In Proceedings of the 14th Conference on USENIX Security Symposium - Volume 14, SSYM'05, pages 12--12, Berkeley, CA, USA, 2005. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An empirical study of operating systems errors. In Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, SOSP '01, pages 73--88, New York, NY, USA, 2001. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J. Criswell, N. Dautenhahn, and V. Adve. KCoFI: Complete control-flow integrity for commodity operating system kernels. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 292--307, Washington, DC, USA, 2014. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. J. Criswell, N. Dautenhahn, and V. Adve. Virtual ghost: Protecting applications from hostile operating systems. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '14, pages 81--96, New York, NY, USA, 2014. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. Criswell, N. Geoffray, and V. Adve. Memory safety for low-level software/hardware interactions. In Proceedings of the 18th Conference on USENIX Security Symposium, SSYM'09, pages 83--100, Berkeley, CA, USA, 2009. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. J. Criswell, A. Lenharth, D. Dhurjati, and V. Adve. Secure virtual architecture: A safe execution environment for commodity operating systems. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 351--366, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. D. R. Engler, M. F. Kaashoek, and J. O'Toole, Jr. Exokernel: An operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 251--266, New York, NY, USA, 1995. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. U. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, OSDI '06, pages 75--88, Berkeley, CA, USA, 2006. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. T. Garfinkel and M. Rosenblum. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2003, San Diego, California, USA. The Internet Society, 2003.Google ScholarGoogle Scholar
  20. O. S. Hofmann, S. Kim, A. M. Dunn, M. Z. Lee, and E. Witchel. InkTag: Secure applications on an untrusted operating system. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 265--278, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. A. Hofmeyr, S. Forrest, and A. Somayaji. Intrusion detection using sequences of system calls. J. Comput. Secur., 6(3):151--180, Aug. 1998. Google ScholarGoogle ScholarCross RefCross Ref
  22. N. Honarmand, N. Dautenhahn, J. Torrellas, S. T. King, G. Pokam, and C. Pereira. Cyrus: Unintrusive application-level record-replay for replay parallelism. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 193--206, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. N. Honarmand and J. Torrellas. Replay debugging: Leveraging record and replay for program debugging. In Proceeding of the 41st Annual International Symposium on Computer Architecuture, ISCA '14, pages 445--456, Piscataway, NJ, USA, 2014. IEEE Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. B. Jain, M. B. Baig, D. Zhang, D. E. Porter, and R. Sion. SoK: Introspections on trust and the semantic gap. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 605--620, Washington, DC, USA, 2014. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. V. P. Kemerlis, M. Polychronakis, and A. D. Keromytis. ret2dir: Rethinking kernel isolation. In Proceedings of the 23rd USENIX Conference on Security Symposium, SEC'14, pages 957--972, Berkeley, CA, USA, 2014. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. S. T. King and P. M. Chen. Backtracking intrusions. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 223--236, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. Kong. Designing BSD Rootkits. No Starch Press, San Francisco, CA, USA, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. Liedtke. On micro-kernel construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 237--250, New York, NY, USA, 1995. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Y. Mao, H. Chen, D. Zhou, X. Wang, N. Zeldovich, and M. F. Kaashoek. Software fault isolation with API integrity and multi-principal modules. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 115--128, New York, NY, USA, 2011. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. L. McVoy and C. Staelin. lmbench: Portable tools for performance analysis. In Proceedings of the 1996 Annual Conference on USENIX Annual Technical Conference, ATEC '96, pages 23--23, Berkeley, CA, USA, 1996. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Microsoft. Kernel patch protection: frequently asked questions (windows drivers), 2007.Google ScholarGoogle Scholar
  32. P. Montesinos, M. Hicks, S. T. King, and J. Torrellas. Capo: A software-hardware interface for practical deterministic multiprocessor replay. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, pages 73-- 84, New York, NY, USA, 2009. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. D. Mutz, F. Valeur, G. Vigna, and C. Kruegel. Anomalous system call detection. ACM Trans. Inf. Syst. Secur., 9(1):61--93, Feb. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. E. I. Organick. The Multics System: An Examination of Its Structure. MIT Press, Cambridge, MA, USA, 1972. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. B. D. Payne, M. Carbone, M. Sharif, and W. Lee. Lares: An architecture for secure active monitoring using virtualization. In Proceedings of the 2008 IEEE Symposium on Security and Privacy, SP '08, pages 233--247, Washington, DC, USA, 2008. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. G. Pokam, K. Danne, C. Pereira, R. Kassa, T. Kranich, S. Hu, J. Gottschlich, N. Honarmand, N. Dautenhahn, S. T. King, and J. Torrellas. QuickRec: Prototyping an intel architecture extension for record and replay of multithreaded programs. In Proceedings of the 40th Annual International Symposium on Computer Architecture, ISCA '13, pages 643--654, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. C. Ries. Defeating windows personal firewalls: Filtering methodologies, attacks, and defenses. Technical report, 2005.Google ScholarGoogle Scholar
  38. R. Riley, X. Jiang, and D. Xu. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection, RAID '08, pages 1--20, Berlin, Heidelberg, 2008. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. J. M. Rushby. Design and verification of secure systems. In Proceedings of the Eighth ACM Symposium on Operating Systems Principles, SOSP '81, pages 12--21, New York, NY, USA, 1981. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. J. H. Saltzer. Protection and the control of information sharing in multics. Commun. ACM, 17(7):388--402, July 1974. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. J. H. Saltzer and M. D. Schroeder. The protection of information in computer systems. Proceedings of the IEEE, 63(9):1278--1308, 1975.Google ScholarGoogle ScholarCross RefCross Ref
  42. T. Saulpaugh and C. A. Mirho. Inside the JavaOS operating system. Addison-Wesley Reading, 1999.Google ScholarGoogle Scholar
  43. A. Seshadri, M. Luk, N. Qu, and A. Perrig. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 335--350, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. M. I. Sharif, W. Lee, W. Cui, and A. Lanzi. Secure in-VM monitoring using hardware virtualization. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 477--487, New York, NY, USA, 2009. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. G. E. Suh, D. Clarke, B. Gassend, M. van Dijk, and S. Devadas. AEGIS: Architecture for tamper-evident and tamper-resistant processing. In Proceedings of the 17th Annual International Conference on Supercomputing, ICS '03, pages 160--171, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. M. M. Swift, B. N. Bershad, and H. M. Levy. Improving the reliability of commodity operating systems. ACM Trans. Comput. Syst., 23(1):77--110, Feb. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. A. Tereshkin. Rootkits: Attacking personal firewalls. In Proceedings of the Black Hat USA 2006 Conference, 2006.Google ScholarGoogle Scholar
  48. I. Unified EFI. Unified extensible firmware interface specification: Version 2.2d, November 2010.Google ScholarGoogle Scholar
  49. D. Wagner and P. Soto. Mimicry attacks on host-based intrusion detection systems. In Proceedings of the 9th ACM Conference on Computer and Communications Security, CCS '02, pages 255--264, New York, NY, USA, 2002. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Z. Wang and X. Jiang. HyperSafe: A lightweight approach to provide lifetime hypervisor control-flow integrity. In Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP '10, pages 380--395, Washington, DC, USA, 2010. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Z. Wang, X. Jiang, W. Cui, and P. Ning. Countering kernel rootkits with lightweight hook protection. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 545--554, New York, NY, USA, 2009. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. C. Warrender, S. Forrest, and B. A. Pearlmutter. Detecting intrusions using system calls: Alternative data models. In 1999 IEEE Symposium on Security and Privacy, SP '99, pages 133--145, Oakland, California, USA, May 1999. IEEE Computer Society.Google ScholarGoogle ScholarCross RefCross Ref
  53. D. Wheeler. SLOCCount, 2015. http://www.dwheeler.com/sloccount/.Google ScholarGoogle Scholar
  54. C. Wright. Para-virtualization interfaces, 2006. http://lwn.net/Articles/194340.Google ScholarGoogle Scholar
  55. X. Xiong and P. Liu. SILVER: Fine-grained and transparent protection domain primitives in commodity OS kernel. In S. J. Stolfo, A. Stavrou, and C. V. Wright, editors, Research in Attacks, Intrusions, and Defenses, number 8145 in Lecture Notes in Computer Science, pages 103--122. Springer Berlin Heidelberg, Jan. 2013.Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. M. Xu, X. Jiang, R. Sandhu, and X. Zhang. Towards a VMM-based usage control framework for OS kernel integrity protection. In Proceedings of the 12th ACM Symposium on Access Control Models and Technologies, SACMAT '07, pages 71--80, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!