Abstract
Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a small isolated kernel within a traditional monolithic kernel. The "nested kernel" interposes on all updates to virtual memory translations to assert protections on physical memory, thus significantly reducing the trusted computing base for memory access control enforcement. We incorporated the nested kernel architecture into FreeBSD on x86-64 hardware while allowing the entire operating system, including untrusted components, to operate at the highest hardware privilege level by write-protecting MMU translations and de-privileging the untrusted part of the kernel. Our implementation inherently enforces kernel code integrity while still allowing dynamically loaded kernel modules, thus defending against code injection attacks. We also demonstrate that the nested kernel architecture allows kernel developers to isolate memory in ways not possible in monolithic kernels by introducing write-mediation and write-logging services to protect critical system data structures. Performance of the nested kernel prototype shows modest overheads: <1% average for Apache and 2.7% for kernel compile. Overall, our results and experience show that the nested kernel design can be retrofitted to existing monolithic kernels, providing important security benefits.
- AMD64 architecture programmers manual volume 2: System programming. Manual, Advancd Micro Devices, 2006.Google Scholar
- Intel 64 and IA-32 architectures software developers manual. Manual 325384-051US, Intel, June 2014.Google Scholar
- M. Accetta, R. Baron, W. Bolosky, D. Golub, R. Rashid, A. Tevanian, and M. Young. Mach: A new kernel foundation for UNIX development. In Proceedings of the USENIX Annual Technical Conference, USENIX ATC'10, pages 93--112, Altanta, GA, USA, 1986. USENIX Association.Google Scholar
- M. Aiken, M. Fhndrich, C. Hawblitzel, G. Hunt, and J. Larus. Deconstructing process isolation. In Proceedings of the 2006 Workshop on Memory System Performance and Correctness, MSPC '06, pages 1--10, New York, NY, USA, 2006. ACM. Google Scholar
Digital Library
- argp and Karl. Exploiting UMA, FreeBSD's kernel memory allocator..:: Phrack Magazine ::., 0x0d(0x42), Nov. 2009.Google Scholar
- S. Bahram, X. Jiang, Z. Wang, M. Grace, J. Li, D. Srinivasan, J. Rhee, and D. Xu. DKSM: Subverting virtual machine introspection for fun and profit. In Proceedings of the 2010 29th IEEE Symposium on Reliable Distributed Systems, SRDS '10, pages 82--91, Washington, DC, USA, 2010. IEEE Computer Society. Google Scholar
Digital Library
- P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 164--177, New York, NY, USA, 2003. ACM. Google Scholar
Digital Library
- A. Belay, A. Bittau, A. Mashtizadeh, D. Terei, D. Mazires, and C. Kozyrakis. Dune: Safe user-level access to privileged CPU features. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI'12, pages 335--348, Berkeley, CA, USA, 2012. USENIX Association. Google Scholar
Digital Library
- B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. Extensibility safety and performance in the SPIN operating system. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 267--283, New York, NY, USA, 1995. ACM. Google Scholar
Digital Library
- M. Castro, M. Costa, J.-P. Martin, M. Peinado, P. Akritidis, A. Donnelly, P. Barham, and R. Black. Fast byte-granularity software fault isolation. In Proceedings of the ACM SIGOPS 22nd symposium on Operating Systems Principles, SOSP '09, pages 45--58, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- S. Chen, J. Xu, E. C. Sezer, P. Gauriar, and R. K. Iyer. Non-control-data attacks are realistic threats. In Proceedings of the 14th Conference on USENIX Security Symposium - Volume 14, SSYM'05, pages 12--12, Berkeley, CA, USA, 2005. USENIX Association. Google Scholar
Digital Library
- A. Chou, J. Yang, B. Chelf, S. Hallem, and D. Engler. An empirical study of operating systems errors. In Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, SOSP '01, pages 73--88, New York, NY, USA, 2001. ACM. Google Scholar
Digital Library
- J. Criswell, N. Dautenhahn, and V. Adve. KCoFI: Complete control-flow integrity for commodity operating system kernels. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 292--307, Washington, DC, USA, 2014. IEEE Computer Society. Google Scholar
Digital Library
- J. Criswell, N. Dautenhahn, and V. Adve. Virtual ghost: Protecting applications from hostile operating systems. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '14, pages 81--96, New York, NY, USA, 2014. ACM. Google Scholar
Digital Library
- J. Criswell, N. Geoffray, and V. Adve. Memory safety for low-level software/hardware interactions. In Proceedings of the 18th Conference on USENIX Security Symposium, SSYM'09, pages 83--100, Berkeley, CA, USA, 2009. USENIX Association. Google Scholar
Digital Library
- J. Criswell, A. Lenharth, D. Dhurjati, and V. Adve. Secure virtual architecture: A safe execution environment for commodity operating systems. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 351--366, New York, NY, USA, 2007. ACM. Google Scholar
Digital Library
- D. R. Engler, M. F. Kaashoek, and J. O'Toole, Jr. Exokernel: An operating system architecture for application-level resource management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 251--266, New York, NY, USA, 1995. ACM. Google Scholar
Digital Library
- U. Erlingsson, M. Abadi, M. Vrable, M. Budiu, and G. C. Necula. XFI: Software guards for system address spaces. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, OSDI '06, pages 75--88, Berkeley, CA, USA, 2006. USENIX Association. Google Scholar
Digital Library
- T. Garfinkel and M. Rosenblum. A virtual machine introspection based architecture for intrusion detection. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2003, San Diego, California, USA. The Internet Society, 2003.Google Scholar
- O. S. Hofmann, S. Kim, A. M. Dunn, M. Z. Lee, and E. Witchel. InkTag: Secure applications on an untrusted operating system. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 265--278, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- S. A. Hofmeyr, S. Forrest, and A. Somayaji. Intrusion detection using sequences of system calls. J. Comput. Secur., 6(3):151--180, Aug. 1998. Google Scholar
Cross Ref
- N. Honarmand, N. Dautenhahn, J. Torrellas, S. T. King, G. Pokam, and C. Pereira. Cyrus: Unintrusive application-level record-replay for replay parallelism. In Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '13, pages 193--206, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- N. Honarmand and J. Torrellas. Replay debugging: Leveraging record and replay for program debugging. In Proceeding of the 41st Annual International Symposium on Computer Architecuture, ISCA '14, pages 445--456, Piscataway, NJ, USA, 2014. IEEE Press. Google Scholar
Digital Library
- B. Jain, M. B. Baig, D. Zhang, D. E. Porter, and R. Sion. SoK: Introspections on trust and the semantic gap. In Proceedings of the 2014 IEEE Symposium on Security and Privacy, SP '14, pages 605--620, Washington, DC, USA, 2014. IEEE Computer Society. Google Scholar
Digital Library
- V. P. Kemerlis, M. Polychronakis, and A. D. Keromytis. ret2dir: Rethinking kernel isolation. In Proceedings of the 23rd USENIX Conference on Security Symposium, SEC'14, pages 957--972, Berkeley, CA, USA, 2014. USENIX Association. Google Scholar
Digital Library
- S. T. King and P. M. Chen. Backtracking intrusions. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP '03, pages 223--236, New York, NY, USA, 2003. ACM. Google Scholar
Digital Library
- J. Kong. Designing BSD Rootkits. No Starch Press, San Francisco, CA, USA, 2007. Google Scholar
Digital Library
- J. Liedtke. On micro-kernel construction. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles, SOSP '95, pages 237--250, New York, NY, USA, 1995. ACM. Google Scholar
Digital Library
- Y. Mao, H. Chen, D. Zhou, X. Wang, N. Zeldovich, and M. F. Kaashoek. Software fault isolation with API integrity and multi-principal modules. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, SOSP '11, pages 115--128, New York, NY, USA, 2011. ACM. Google Scholar
Digital Library
- L. McVoy and C. Staelin. lmbench: Portable tools for performance analysis. In Proceedings of the 1996 Annual Conference on USENIX Annual Technical Conference, ATEC '96, pages 23--23, Berkeley, CA, USA, 1996. USENIX Association. Google Scholar
Digital Library
- Microsoft. Kernel patch protection: frequently asked questions (windows drivers), 2007.Google Scholar
- P. Montesinos, M. Hicks, S. T. King, and J. Torrellas. Capo: A software-hardware interface for practical deterministic multiprocessor replay. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIV, pages 73-- 84, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- D. Mutz, F. Valeur, G. Vigna, and C. Kruegel. Anomalous system call detection. ACM Trans. Inf. Syst. Secur., 9(1):61--93, Feb. 2006. Google Scholar
Digital Library
- E. I. Organick. The Multics System: An Examination of Its Structure. MIT Press, Cambridge, MA, USA, 1972. Google Scholar
Digital Library
- B. D. Payne, M. Carbone, M. Sharif, and W. Lee. Lares: An architecture for secure active monitoring using virtualization. In Proceedings of the 2008 IEEE Symposium on Security and Privacy, SP '08, pages 233--247, Washington, DC, USA, 2008. IEEE Computer Society. Google Scholar
Digital Library
- G. Pokam, K. Danne, C. Pereira, R. Kassa, T. Kranich, S. Hu, J. Gottschlich, N. Honarmand, N. Dautenhahn, S. T. King, and J. Torrellas. QuickRec: Prototyping an intel architecture extension for record and replay of multithreaded programs. In Proceedings of the 40th Annual International Symposium on Computer Architecture, ISCA '13, pages 643--654, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- C. Ries. Defeating windows personal firewalls: Filtering methodologies, attacks, and defenses. Technical report, 2005.Google Scholar
- R. Riley, X. Jiang, and D. Xu. Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection, RAID '08, pages 1--20, Berlin, Heidelberg, 2008. Springer-Verlag. Google Scholar
Digital Library
- J. M. Rushby. Design and verification of secure systems. In Proceedings of the Eighth ACM Symposium on Operating Systems Principles, SOSP '81, pages 12--21, New York, NY, USA, 1981. ACM. Google Scholar
Digital Library
- J. H. Saltzer. Protection and the control of information sharing in multics. Commun. ACM, 17(7):388--402, July 1974. Google Scholar
Digital Library
- J. H. Saltzer and M. D. Schroeder. The protection of information in computer systems. Proceedings of the IEEE, 63(9):1278--1308, 1975.Google Scholar
Cross Ref
- T. Saulpaugh and C. A. Mirho. Inside the JavaOS operating system. Addison-Wesley Reading, 1999.Google Scholar
- A. Seshadri, M. Luk, N. Qu, and A. Perrig. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP '07, pages 335--350, New York, NY, USA, 2007. ACM. Google Scholar
Digital Library
- M. I. Sharif, W. Lee, W. Cui, and A. Lanzi. Secure in-VM monitoring using hardware virtualization. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 477--487, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- G. E. Suh, D. Clarke, B. Gassend, M. van Dijk, and S. Devadas. AEGIS: Architecture for tamper-evident and tamper-resistant processing. In Proceedings of the 17th Annual International Conference on Supercomputing, ICS '03, pages 160--171, New York, NY, USA, 2003. ACM. Google Scholar
Digital Library
- M. M. Swift, B. N. Bershad, and H. M. Levy. Improving the reliability of commodity operating systems. ACM Trans. Comput. Syst., 23(1):77--110, Feb. 2005. Google Scholar
Digital Library
- A. Tereshkin. Rootkits: Attacking personal firewalls. In Proceedings of the Black Hat USA 2006 Conference, 2006.Google Scholar
- I. Unified EFI. Unified extensible firmware interface specification: Version 2.2d, November 2010.Google Scholar
- D. Wagner and P. Soto. Mimicry attacks on host-based intrusion detection systems. In Proceedings of the 9th ACM Conference on Computer and Communications Security, CCS '02, pages 255--264, New York, NY, USA, 2002. ACM. Google Scholar
Digital Library
- Z. Wang and X. Jiang. HyperSafe: A lightweight approach to provide lifetime hypervisor control-flow integrity. In Proceedings of the 2010 IEEE Symposium on Security and Privacy, SP '10, pages 380--395, Washington, DC, USA, 2010. IEEE Computer Society. Google Scholar
Digital Library
- Z. Wang, X. Jiang, W. Cui, and P. Ning. Countering kernel rootkits with lightweight hook protection. In Proceedings of the 16th ACM Conference on Computer and Communications Security, CCS '09, pages 545--554, New York, NY, USA, 2009. ACM. Google Scholar
Digital Library
- C. Warrender, S. Forrest, and B. A. Pearlmutter. Detecting intrusions using system calls: Alternative data models. In 1999 IEEE Symposium on Security and Privacy, SP '99, pages 133--145, Oakland, California, USA, May 1999. IEEE Computer Society.Google Scholar
Cross Ref
- D. Wheeler. SLOCCount, 2015. http://www.dwheeler.com/sloccount/.Google Scholar
- C. Wright. Para-virtualization interfaces, 2006. http://lwn.net/Articles/194340.Google Scholar
- X. Xiong and P. Liu. SILVER: Fine-grained and transparent protection domain primitives in commodity OS kernel. In S. J. Stolfo, A. Stavrou, and C. V. Wright, editors, Research in Attacks, Intrusions, and Defenses, number 8145 in Lecture Notes in Computer Science, pages 103--122. Springer Berlin Heidelberg, Jan. 2013.Google Scholar
Digital Library
- M. Xu, X. Jiang, R. Sandhu, and X. Zhang. Towards a VMM-based usage control framework for OS kernel integrity protection. In Proceedings of the 12th ACM Symposium on Access Control Models and Technologies, SACMAT '07, pages 71--80, New York, NY, USA, 2007. ACM. Google Scholar
Digital Library
Index Terms
Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
Recommendations
Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
ASPLOS '15: Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating SystemsMonolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a ...
Nested Kernel: An Operating System Architecture for Intra-Kernel Privilege Separation
ASPLOS'15Monolithic operating system designs undermine the security of computing systems by allowing single exploits anywhere in the kernel to enjoy full supervisor privilege. The nested kernel operating system architecture addresses this problem by "nesting" a ...
Fast Intra-kernel Isolation and Security with IskiOS
RAID '21: Proceedings of the 24th International Symposium on Research in Attacks, Intrusions and DefensesThe kernels of operating systems such as Windows, Linux, and MacOS are vulnerable to control-flow hijacking. Defenses exist, but many require efficient intra-address-space isolation. Execute-only memory, for example, requires read protection on code ...







Comments