Unikraft: fast, specialized unikernels the easy way

ABSTRACT

Unikernels are famous for providing excellent performance in terms of boot times, throughput and memory consumption, to name a few metrics. However, they are infamous for making it hard and extremely time consuming to extract such performance, and for needing significant engineering effort in order to port applications to them. We introduce Unikraft, a novel micro-library OS that (1) fully modularizes OS primitives so that it is easy to customize the unikernel and include only relevant components and (2) exposes a set of composable, performance-oriented APIs in order to make it easy for developers to obtain high performance.

Our evaluation using off-the-shelf applications such as nginx, SQLite, and Redis shows that running them on Unikraft results in a 1.7x-2.7x performance improvement compared to Linux guests. In addition, Unikraft images for these apps are around 1MB, require less than 10MB of RAM to run, and boot in around 1ms on top of the VMM time (total boot time 3ms-40ms). Unikraft is a Linux Foundation open source project and can be found at www.unikraft.org.

References

1. Intel® 64 and IA-32 Architectures Software Developer's Manual. Volume 3A, Section 4.6.2.
2. Newlib: a C library intended for use on embedded systems. https://sourceware.org/newlib/. Online; accessed Jan, 25 2021.
3. Martín Abadi, Mihai Budiu, Úlfar Erlingsson, and Jay Ligatti. Controlflow integrity. In Proceedings of the 12th ACM Conference on Computer and Communications Security, CCS '05, pages 340--353, New York, NY, USA, 2005. Association for Computing Machinery.
4. Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa. Firecracker: Lightweight virtualization for serverless applications. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI'20), pages 419--434, 2020.
5. Andrew Baumann, Jonathan Appavoo, Orran Krieger, and Timothy Roscoe. A fork() in the road. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS'19, page 14--22, New York, NY, USA, 2019. Association for Computing Machinery.
6. Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christos Kozyrakis, and Edouard Bugnion. IX: A protected dataplane operating system for high throughput and low latency. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI'14, pages 49--65, Broomfield, CO, 2014. USENIX Association.
7. Florian Bernd and Joel Höner. Zydis: Fast and lightweight x86/x86-64 disassembler library. https://zydis.re/. Online; accessed Jan, 25 2021.
8. Ivan T. Bowman, Richard C. Holt, and Neil V. Brewster. Linux as a case study: Its extracted software architecture. In Proceedings of the 21st International Conference on Software Engineering, ICSE '99, page 555--563, New York, NY, USA, 1999. Association for Computing Machinery.
9. Alfred Bratterud, Alf-Andre Walla, Harek Haugerud, Paal E. Engelstad, and Kyrre Begnum. IncludeOS: A minimal, resource efficient unikernel for cloud services. In Proceedings of the 7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom'15. IEEE, November 2015.
10. James Cadden, Thomas Unger, Yara Awad, Han Dong, Orran Krieger, and Jonathan Appavoo. SEUSS: Rapid serverless deployment using environment snapshots. CoRR, abs/1910.01558, 2019.
11. Jonathan Corbet. The rapid growth of io_uring. https://lwn.net/Articles/810414/. Online; accessed Jan, 25 2021.
12. Thurston H.Y. Dang, Petros Maniatis, and David Wagner. Oscar: A practical page-permissions-based scheme for thwarting dangling pointers. In Proceedings of the 26th USENIX Security Symposium, USENIX Security'17, pages 815--832, Vancouver, BC, 2017. USENIX Association.
13. Nicolai Davidsson, Andre Pawlowski, and Thorsten Holz. Towards automated application-specific software stacks. In Proceedings of the 24th European Symposium on Research in Computer Security, pages 88--109, 2019.
14. Debian. Debian Popularity Contest. https://popcon.debian.org/. Online; accessed Jan, 25 2021.
15. Jeff Dike. A user-mode port of the linux kernel. In Proceedings of the 4th Annual Linux Showcase and Conference (Volume 4), ALS'00, pages 7--7, Berkeley, CA, USA, 2000. USENIX Association.
16. Docker Docs. Get Docker. https://docs.docker.com/get-docker/. Online; accessed March, 26 2021.
17. Adam Dunkels. Design and implementation of the lwip stack. 2001.
18. Pekka Enberg, Ashwin Rao, and Sasu Tarkoma. I/O Is Faster Than the CPU: Let's Partition Resources and Eliminate (Most) OS Abstractions. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS'19, pages 81--87, New York, NY, USA, 2019. ACM.
19. Dario Faggioli. Virtual-machine scheduling and scheduling in virtual machines. https://lwn.net/Articles/793375/, July 2019. Online; accessed Jan, 25 2021.
20. Galois Inc. The haskell lightweight virtual machine (halvm). https://github.com/GaloisInc/HaLVM, 2008. Online; accessed Jan, 25 2021.
21. GitHub. A Unikernel SDK. Extreme Specialization for Security and Performance. https://github.com/unikraft. Online; accessed March, 26 2021.
22. GitHub. Artifacts, including experiments and graphs, for the paper: "Unikraft: Fast, Specialized Unikernels the Easy Way" (EuroSys'21). https://github.com/unikraft/eurosys21-artifacts. Online; accessed March, 26 2021.
23. GitHub. Erlang on Xen. https://github.com/cloudozer/ling. Online; accessed Jan, 25 2021.
24. GitHub. Firecracker GitHub issue #1034: Slower networking of OSv on firecracker vs QEMU/KVM. https://github.com/firecracker-microvm/firecracker/issues/1034. Online; accessed Jan, 25 2021.
25. GitHub. Hermitux GitHub issue #2: It does not work on qemu. https://github.com/ssrg-vt/hermitux/issues/2. Online; accessed Jan, 25 2021.
26. Github. linux-libos-tools. https://github.com/libos-nuse/linux-libos-tools. Online; accessed Jan, 25 2021.
27. GitHub. Performance issue with Redis on recent Linux kernels. https://github.com/ssrg-vt/hermitux/issues/12. Online; accessed March, 26 2021.
28. Github. Xen Minimal OS - Memory management related functions. https://github.com/sysml/mini-os/blob/master/mm.c. Online; accessed Jan, 25 2021.
29. Github.com. The Unikernel and MicroVM Compilation and Deployment Platform. https://github.com/solo-io/unik. Online; accessed Jan, 25 2021.
30. Google. Cloud TPU - Train and run machine learning models faster than ever before. https://cloud.google.com/tpu. Online; accessed Jan, 25 2021.
31. Google. Protocol Buffers - Google's data interchange format. https://github.com/protocolbuffers/protobuf.
32. Habana. 100% AI. https://habana.ai/. Online; accessed Jan, 25 2021.
33. Intel. Ethernet-Converged-Network-Adapter X520-T2. https://ark.intel.com/content/www/de/de/ark/products/69655/intel-ethernet-converged-network-adapter-x520-t2.html. Online; accessed Mar, 26 2021.
34. Intel. Intel® Movidius™ Vision Processing Units (VPUs). https://www.intel.com/content/www/us/en/products/processors/movidius-vpu.html. Online; accessed Jan, 25 2021.
35. Antti Kantee. Kernel file systems as userspace programs. September 2007.
36. Antti Kantee. Flexible Operating System Internals: The Design and Implementation of the Anykernel and Rump Kernels. PhD thesis, Aalto University, 2012.
37. Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav Har'El, Don Marti, and Vlad Zolotarov. OSv-Optimizing the Operating System for Virtual Machines. In Proceedings of the 2014 USENIX Annual Technical Conference, USENIX ATC'14, pages 61--72, Philadelphia, PA, June 2014. USENIX Association.
38. Simon Kuenzer, Anton Ivanov, Filipe Manco, Jose Mendes, Yuri Volchkov, Florian Schmidt, Kenichi Yasukata, Michio Honda, and Felipe Huici. Unikernels everywhere: The case for elastic cdns. In Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '17, pages 15--29, New York, NY, USA, 2017. ACM.
39. Simon Kuenzer, Joao Martins, Mohamed Ahmed, and Felipe Huici. Towards minimalistic, virtualized content caches with minicache. In Proceedings of the 2013 ACM Workshop on Hot Topics in Middleboxes and Network Function Virtualization, HotMiddlebox'13, pages 13--18. ACM, 2013.
40. Hsuan-Chi Kuo, Dan Williams, Ricardo Koller, and Sibin Mohan. A linux in unikernel clothing. In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys '20, New York, NY, USA, 2020. Association for Computing Machinery.
41. Lars Kurth and Russell Pavlicek. Xen Project Wiki Mini-OS. https://wiki.xenproject.org/wiki/Mini-OS, 2018. Online; accessed Jan, 25 2021.
42. Daan Leijen, Benjamin Zorn, and Leonardo de Moura. Mimalloc: Free list sharding in action. In Asian Symposium on Programming Languages and Systems, pages 244--265. Springer, 2019.
43. Guanyu Li, Dong Du, and Yubin Xia. Iso-unik: lightweight multiprocess unikernel through memory protection keys. Cybersecur., 3(1):11, 2020.
44. James Litton, Deepak Garg, Peter Druschel, and Bobby Bhattacharjee. Composing abstractions using the null-kernel. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS'19, pages 1--6, New York, NY, USA, 2019. ACM.
45. Anil Madhavapeddy, Thomas Leonard, Magnus Skjegstad, Thomas Gazagnaire, David Sheets, Dave Scott, Richard Mortier, Amir Chaudhry, Balraj Singh, Jon Ludlam, Jon Crowcroft, and Ian Leslie. Jitsu: Just-In-Time Summoning of Unikernels. In 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI '15, pages 559--573, Oakland, CA, 2015. USENIX Association.
46. Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. Unikernels: Library operating systems for the cloud. In Proceedings of the 18th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS'13). ACM, 2013.
47. Anil Madhavapeddy and David J. Scott. Unikernels: Rise of the Virtual Library Operating System. Queue, 11(11):30:30--30:44, December 2013.
48. Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. My vm is lighter (and safer) than your container. In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP '17, pages 218--233, New York, NY, USA, 2017. ACM.
49. Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. My VM is lighter (and safer) than your container. In Proceedings of the 26th ACM Symposium on Operating Systems Principles, SOSP'17. ACM, 2017.
50. Ilias Marinos, Robert N.M. Watson, and Mark Handley. Network Stack Specialization for Performance. In Proceedings of the 2014 ACM Conference on Computer Communication, SIGCOMM '14, pages 175--186, New York, NY, USA, 2014. ACM.
51. Joao Martins, Mohamed Ahmed, Costin Raiciu, and Felipe Huici. Enabling fast, dynamic network processing with clickos. In Proceedings of the 2nd ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, HotSDN''13. ACM, 2013.
52. Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. ClickOS and the art of network function virtualization. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation, NSDI'14, pages 459--473. USENIX, 2014.
53. Miguel Masmano, Ismael Ripoll, Alfons Crespo, and Jorge Real. TLSF: A new dynamic memory allocator for real-time systems. In Proceedings of the 16th Euromicro Conference on Real-Time Systems, ECRTS'04, pages 79--88. IEEE, 2004.
54. Pierre Olivier, Daniel Chiba, Stefan Lankes, Changwoo Min, and Binoy Ravindran. A binary-compatible unikernel. In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE 2019, pages 59--73, New York, NY, USA, 2019. ACM.
55. Pierre Olivier, Daniel Chiba, Stefan Lankes, Changwoo Min, and Binoy Ravindran. A binary-compatible unikernel. In Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), VEE'19, pages 59--73. ACM, 2019.
56. Openwall. Implement glibc chk interfaces for ABI compatibility. https://www.openwall.com/lists/musl/2015/06/17/1. Online; accessed Jan, 25 2021.
57. Bo Peng, Haozhong Zhang, Jianguo Yao, Yaozu Dong, Yu Xu, and Haibing Guan. Mdev-nvme: A nvme storage virtualization solution with mediated pass-through. In Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference, USENIX ATC'18, page 665--676, USA, 2018. USENIX Association.
58. Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe. Arrakis: The operating system is the control plane. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 1--16, Broomfield, CO, October 2014. USENIX Association.
59. Donald E. Porter, Silas Boyd-Wickizer, Jon Howell, Reuben Olinsky, and Galen C. Hunt. Rethinking the library os from the top down. In Proceedings of the Sixteenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS'16, page 291--304, New York, NY, USA, 2011. Association for Computing Machinery.
60. Octavian Purdila, Lucian Grijincu, and Nicolae Tapus. Lkl: The linux kernel library. Proceedings of the Roedunet International Conference, pages 328 -- 333, 07 2010.
61. Anh Quach, Rukayat Erinfolami, David Demicco, and Aravind Prakash. A multi-os cross-layer study of bloating in user programs, kernel and managed execution environments. In Proceedings of the 2017 Workshop on Forming an Ecosystem Around Software Transformation, FEAST '17, page 65--70, New York, NY, USA, 2017. Association for Computing Machinery.
62. Alessandro Randazzo and Ilenia Tinnirello. Kata containers: An emerging architecture for enabling mec services in fast and secure way. In Proceedings of the 6th International Conference on Internet of Things: Systems, Management and Security, IOTSMS'19, pages 209--214. IEEE, 2019.
63. Ali Raza, Parul Sohal, James Cadden, Jonathan Appavoo, Ulrich Drepper, Richard Jones, Orran Krieger, Renato Mancuso, and Larry Woodman. Unikernels: The next stage of linux's dominance. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS '19, pages 7--13, New York, NY, USA, 2019. ACM.
64. Luigi Rizzo. netmap: A novel framework for fast packet I/O. In Proceedings of the 21st USENIX Annual Technical Conference, USENIX ATC'12, pages 101--112. USENIX, 2012.
65. runtimejs.org. JavaScript Library Operating System for the Cloud. http://runtimejs.org/. Online; accessed Jan, 25 2021.
66. Joe Savage and Timothy M. Jones. Halo: Post-link heap-layout optimisation. In Proceedings of the 18th ACM/IEEE International Symposium on Code Generation and Optimization, CGO'20, page 94--106, New York, NY, USA, 2020. Association for Computing Machinery.
67. Karsten Schmidt. malloc, free replacement for unmanaged, linear memory situations. https://github.com/thi-ng/tinyalloc. Online; accessed Jan, 25 2021.
68. Shuttle. SH370R6 XCP Cube. http://global.shuttle.com/products/productsDetail?productId=2265. Online; accessed Mar, 26 2021.
69. Mincheol Sung, Pierre Olivier, Stefan Lankes, and Binoy Ravindran. Intra-unikernel isolation with intel memory protection keys. In Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '20, page 143--156, New York, NY, USA, 2020. Association for Computing Machinery.
70. The Linux Kernel Development Community. The kernel address sanitizer (KASAN). https://www.kernel.org/doc/html/v5.10/dev-tools/kasan.html. Online; accessed Jan, 25 2021.
71. The Linux Kernel Development Community. The kernel concurrency sanitizer (KCSAN). https://www.kernel.org/doc/html/v5.10/dev-tools/kcsan.html. Online; accessed Jan, 25 2021.
72. The Linux Kernel Development Community. The undefined behavior sanitizer (UBSAN). https://www.kernel.org/doc/html/v5.10/dev-tools/ubsan.html. Online; accessed Jan, 25 2021.
73. Chia-Che Tsai, Kumar Saurabh Arora, Nehal Bandi, Bhushan Jain, William Jannen, Jitin John, Harry A. Kalodner, Vrushali Kulkarni, Daniela Oliveira, and Donald E. Porter. Cooperation and security isolation of library oses for multi-process applications. In Proceedings of the 9th European Conference on Computer Systems, EuroSys'14, pages 9:1--9:14, New York, NY, USA, 2014. ACM.
74. Chia-Che Tsai, Bhushan Jain, Nafees Ahmed Abdul, and Donald E. Porter. A study of modern linux api usage and compatibility: What to support when you're supporting. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys '16, New York, NY, USA, 2016. Association for Computing Machinery.
75. unikraft.org. Unikraft's Documentation. http://docs.unikraft.org/. Online; accessed March, 26 2021.
76. Arjan Van de Ven. An introduction to Clear Containers. https://lwn.net/Articles/644675/. Online; accessed Jan, 25 2021.
77. Eric Van Hensbergen and Ron Minnich. Grave robbers from outer space using 9p2000 under linux. In Proceedings of the USENIX Annual Technical Conference, ATC'05, page 45, USA, 2005. USENIX Association.
78. Dan Williams and Ricardo Koller. Unikernel Monitors: Extending Minimalism Outside of the Box. In 8th USENIX Workshop on Hot Topics in Cloud Computing, HotCloud '16, Denver, CO, 2016. USENIX Association.
79. Xen Project. Minios-devel - Mini-os development list. https://lists.xenproject.org/cgi-bin/mailman/listinfo/minios-devel. Online; accessed March 26, 2021.
80. Irene Zhang, Jing Liu, Amanda Austin, Michael Lowell Roberts, and Anirudh Badam. I'm not dead yet!: The role of the operating system in a kernel-bypass era. In Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS'19, pages 73--80, New York, NY, USA, 2019. ACM.
81. Yiming Zhang, Jon Crowcroft, Dongsheng Li, Chengfen Zhang, Huiba Li, Yaozheng Wang, Kai Yu, Yongqiang Xiong, and Guihai Chen. KylinX: A Dynamic Library Operating System for Simplified and Efficient Cloud Virtualization. In 2018 USENIX Annual Technical Conference, USENIX ATC'18, pages 173--186. USENIX Association, July 2018.
82. Yiming Zhang, Chengfei Zhang, Yaozheng Wang, Kai Yu, Guangtao Xue, and Jon Crowcroft. Kylinx: Simplified virtualization architecture for specialized virtual appliances with strong isolation. ACM Trans. Comput. Syst., 37(1--4), February 2021.

Index Terms

• Software and its engineering

  • Software organization and properties

    • Contextual software domains

      • Operating systems