Abstract
The traditional "trap and emulate" I/O paravirtualization model conveniently allows for I/O interposition, yet it inherently incurs costly guest-host context switches. The newer "sidecore" model eliminates this overhead by dedicating host (side)cores to poll the relevant guest memory regions and react accordingly without context switching. But the dedication of sidecores on each host might be wasteful when I/O activity is low, or it might not provide enough computational power when I/O activity is high. We propose to alleviate this problem at rack scale by consolidating the dedicated sidecores spread across several hosts onto one server. The hypervisor is then effectively split into two parts: the local hypervisor that hosts the VMs, and the remote hypervisor that processes their paravirtual I/O. We call this model vRIO---paraVirtual Remote I/O. We find that by increasing the latency somewhat, it provides comparable throughput with fewer sidecores and superior throughput with the same number of sidecores as compared to the state of the art. vRIO additionally constitutes a new, cost-effective way to consolidate I/O devices (on the remote hypervisor) while supporting efficient programmable I/O interposition.
- Keith Adams and Ole Agesen. A comparison of software and hardware techniques for x86 virtualization. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 2--13, 2006. http://dx.doi.org/10.1145/1168857.1168860.Google Scholar
Digital Library
- N.R. Adiga, G. Almasi, G.S. Almasi, Y. Aridor, R. Barik, and many others. An overview of the BlueGene/L supercomputer. In ACM/IEEE Supercomputing (SC), pages 1--22, 2002. http://dx.doi.org/10.1109/SC.2002.10017.Google Scholar
- Brian Aker. Memslap - load testing and benchmarking a server. http://docs.libmemcached.org/bin/memslap.html. libmemcached 1.1.0 documentation. Accessed: Jan 2015.Google Scholar
- Nadav Amit, Muli Ben-Yehuda, Dan Tsafrir, and Assaf Schuster. vIOMMU: efficient IOMMU emulation. In USENIX Annual Technical Conference (ATC), pages 73--86, 2011. http://www.usenix.org/events/atc11/tech/final_files/Amit.pdf.Google Scholar
Digital Library
- Nadav Amit, Abel Gordon, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. Bare-metal performance for virtual machines with exitless interrupts. Communications of the ACM (CACM), 59(1):108--116, Jan 2016. http://dx.doi.org/10.1145/2845648.Google Scholar
- Nadav Amit, Dan Tsafrir, and Assaf Schuster. VSwapper: A memory swapper for virtualized environments. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 349--366, 2014. http://dx.doi.org/10.1145/2541940.2541969.Google Scholar
Digital Library
- M.R. Anala, M. Kashyap, and G. Shobha. Application performance analysis during live migration of virtual machines. In IEEE International Advance Computing Conference (IACC), pages 366--372, 2013. http://dx.doi.org/10.1109/IAdCC.2013.6514252.Google Scholar
Cross Ref
- N. Anastopoulos and N. Koziris. Facilitating efficient synchronization of asymmetric threads on hyper-threaded processors. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1--8, 2008. http://dx.doi.org/10.1109/IPDPS.2008.4536358.Google Scholar
Cross Ref
- Apachebench. http://en.wikipedia.org/wiki/ApacheBench. Accessed: Jan 2015.Google Scholar
- Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau. Operating Systems: Three Easy Pieces, chapter 37. Arpaci-Dusseau Books, LLC, 0.90 edition, 2015. http://pages.cs.wisc.edu/ remzi/OSTEP/file-disks.pdf.Google Scholar
- Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. In ACM Symposium on Operating Systems Principles (SOSP), pages 164--177, 2003. http://dx.doi.org/10.1145/945445.945462.Google Scholar
Digital Library
- Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christos Kozyrakis, and Edouard Bugnion. IX: A protected dataplane operating system for high throughput and low latency. In USENIX Symposium on Operating System Design and Implementation (OSDI), pages 49--65, 2014. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-belay.pdf.Google Scholar
- Muli Ben-Yehuda, Eran Borovik, Michael Factor, Eran Rom, Avishay Traeger, and Ben-Ami Yassour. Adding advanced storage controller functionality via low-overhead virtualization. In USENIX Conference on File and Storage Technologies (FAST), pages 187--194, 2012. https://www.usenix.org/legacy/events/fast12/tech/full_papers/Ben-Yehuda2--2--12.pdf.Google Scholar
- Muli Ben-Yehuda, Michael D. Day, Zvi Dubitzky, Michael Factor, Nadav Har'El, Abel Gordon, Anthony Liguori, Orit Wasserman, and Ben-Ami Yassour. The Turtles project: Design and implementation of nested virtualization. In USENIX Symposium on Operating System Design and Implementation (OSDI), pages 423--436, 2010. http://www.usenix.org/events/osdi10/tech/full_papers/Ben-Yehuda.pdf.Google Scholar
- Muli Ben-Yehuda, Jimi Xenidis, Michal Ostrowski, Karl Rister, Alexis Bruemmer, and Leendert van Doorn. The price of safety: Evaluating IOMMU performance. In Ottawa Linux Symposium (OLS), pages 9--20, 2007. https://www.kernel.org/doc/mirror/ols2007v1.pdf#page=9.Google Scholar
- Jason Blosil, David Fair, and David Fair. 10GbE -- key trends, drivers and predictions. SNIA presentation: http://www.snia.org/sites/default/files/SNIA_ESF_10GbE_Webcast_Final_Slides.pdf, July 2012. (Accessed: Aug 2015).Google Scholar
- Vojtech Cima. An analysis of the performance of block live migration in openstack. URL http://tinyurl.com/live-migration-openstack, 2014. Accessed: Jan 2016.Google Scholar
- Peter F. Corbett and Dror G. Feitelson. The vesta parallel file system. ACM Transactions on Computer Systems (TOCS), 14(3):225--264, Aug 1996. http://dx.doi.org/10.1145/233557.233558.Google Scholar
- Crehan Research. Another year of robust growth and record shipments for branded data center switches. http://www.crehanresearch.com/wp-content/uploads/2015/03/CREHAN-2014-Data-Center-Switching-CR.pdf, Mar 2015. (Accessed: Aug 2015).Google Scholar
- B. Davie and J. Gross. A stateless transport tunneling protocol for network virtualization (STT). http://tools.ietf.org/id/draft-davie-stt-04.txt, Sep 2013. Network Working Group; Internet-Draft; draft-davie-stt-04. Accessed: Jan 2016.Google Scholar
- Dell Inc. Dell PowerEdge R930 4-socket rack server. http://www.dell.com/us/business/p/poweredge-r930/pd, Jul 2015. Prices as of 31 July, 2015.Google Scholar
- Yaozu Dong, Xiaowei Yang, Xiaoyong Li, Jianhui Li, Kun Tian, and Haibing Guan. High performance network virtualization with SR-IOV. In IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 1--10, 2010. http://dx.doi.org/10.1109/HPCA.2010.5416637.Google Scholar
Cross Ref
- Virtual I/O acceleration--the Elvis source code. https://github.com/abelg/virtual_io_acceleration, 2013. (Accessed: Mar 2015).Google Scholar
- The Apache HTTP server project. http://httpd.apache.org. Accessed: Jan 2015.Google Scholar
- Roy T. Fielding and Gail Kaiser. The Apache HTTP server project. IEEE Internet Computing, 1(4):88--90, Jul 1997. http://dx.doi.org/10.1109/4236.612229.Google Scholar
Digital Library
- Brad Fitzpatrick. Distributed caching with memcached. Linux Journal, 2004(124):5, Aug 2004. http://dl.acm.org/citation.cfm?id=1012889.1012894.Google Scholar
Digital Library
- Ada Gavrilovska, Sanjay Kumar, Himanshu Raj, Karsten Schwan, Vishakha Gupta, Ripal Nathuji, Radhika Niranjan, Adit Ranadive, and Purav Saraiya. High-performance hypervisor architectures: Virtualization in HPC systems. In Workshop on System-level Virtualization for HPC (HPCVirt), 2007. http://www.csm.ornl.gov/srt/hpcvirt07/papers/paper10.pdf.Google Scholar
- Robert P Goldberg. Survey of virtual machine research. IEEE Computer, 7(6):34--45, 1974. http://dx.doi.org/10.1109/MC.1974.6323581.Google Scholar
Digital Library
- Abel Gordon, Nadav Amit, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. ELI: Bare-metal performance for I/O virtualization. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 411--422, 2012. http://dx.doi.org/10.1145/2150976.2151020.Google Scholar
Digital Library
- James Hamilton. AWS innovation at scale. https://www.youtube.com/watch?t=113&v=JIQETrFC_SQ, Nov 2014. (Accessed: Aug 2015).Google Scholar
- Nadav Har'El, Abel Gordon, Alex Landau, Muli Ben-Yehuda, Avishay Traeger, and Razya Ladelsky. Efficient and scalable paravirtual I/O system. In USENIX Annual Technical Conference (ATC), pages 231--242, 2013. https://www.usenix.org/conference/atc13/technical-sessions/presentation/har%E2%80%99el.Google Scholar
- R.A. Haring, M. Ohmacht, T.W. Fox, M.K. Gschwind, D.L. Satterfield, and many others. The IBM Blue Gene/Q compute chip. IEEE Micro, 32(2):48--60, March 2012. http://dx.doi.org/10.1109/MM.2011.108.Google Scholar
Digital Library
- Tomas Hruby, Herbert Bos, and Andrew S. Tanenbaum. When slower is faster: On heterogeneous multicores for reliable systems. In USENIX Annual Technical Conference (ATC), pages 255--266, 2013. https://www.usenix.org/conference/atc13/technical-sessions/presentation/hruby.Google Scholar
- Intel. PCI-SIG SR-IOV primer: An introduction to SR-IOV technology. http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html, Jan 2011.Google Scholar
- Intel Corporation. Intel processor pricing effective June 07, 2015. http://tinyurl.com/intel-cpu-prices-2015-07, June 2015. (Accessed: Aug 2015, via http://www.intel.com/go/processor_pricing).Google Scholar
- Hai Jin, Li Deng, Song Wu, Xuanhua Shi, and Xiaodong Pan. Live virtual machine migration with adaptive, memory compression. In IEEE International Conference on Cluster Computing (CLUSTER), pages 1--10, 2009. http://dx.doi.org/10.1109/CLUSTR.2009.5289170.Google Scholar
Cross Ref
- Rick A. Jones. A network performance benchmark (Revision 2.0). Technical report, Hewlett Packard, 1995. http://www.netperf.org/netperf/training/Netperf.html. Accessed: Jan 2015.Google Scholar
- Asim Kadav and Michael M. Swift. Live migration of direct-access devices. In USENIX Workshop on I/O Virtualization (WIOV), pages 95--104, 2008. http://usenix.org/events/wiov08/tech/full_papers/kadav/kadav.pdf.Google Scholar
- Sanjay Kumar, Himanshu Raj, Karsten Schwan, and Ivan Ganev. Re-architecting VMMs for multicore systems: The phsidecore approach. In Workshop on Interaction between Operating Systems and Computer Architecture (WIOSCA), 2007. http://www.ideal.ece.ufl.edu/workshops/wiosca07/Paper3.pdf.Google Scholar
- Alex Landau, Muli Ben-Yehuda, and Abel Gordon. SplitX: Split guest/hypervisor execution on multi-core. In USENIX Workshop on I/O Virtualization (WIOV), 2011. http://www.usenix.org/events/wiov11/tech/final_files/Landau.pdf.Google Scholar
Digital Library
- Joshua LeVasseur, Volkmar Uhlig, Jan Stoess, and Stefan Götz. Unmodified device driver reuse and improved system dependability via virtual machines. In USENIX Symposium on Operating System Design and Implementation (OSDI), pages 17--30, 2004. https://www.usenix.org/legacy/publications/library/proceedings/osdi04/tech/full_papers/levasseur/levasseur.pdf.Google Scholar
Digital Library
- Jianwei Li, Wei-Keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O interface. In ACM/IEEE Supercomputing (SC), page No. 39, 2003. http://dx.doi.org/10.1109/SC.2003.10053.Google Scholar
- Guangdeng Liao, Danhua Guo, Laxmi Bhuyan, and Steve R King. Software techniques to improve virtualized I/O performance on multi-core systems. In ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS), pages 161--170, 2008. http://dx.doi.org/10.1145/1477942.1477971.Google Scholar
Digital Library
- Jiuxing Liu. Evaluating standard-based self-virtualizing devices: A performance study on 10 GbE NICs with SR-IOV support. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1--12, 2010. http://dx.doi.org/10.1109/IPDPS.2010.5470365.Google Scholar
Cross Ref
- Jiuxing Liu and Bulen Abali. Virtualization polling engine (VPE): Using dedicated CPU cores to accelerate I/O virtualization. In ACM International Conference on Supercomputing (ICS), pages 225--23, 2009. http://dx.doi.org/10.1145/1542275.1542309.Google Scholar
Digital Library
- Jiuxing Liu, Wei Huang, Bulent Abali, and Dhabaleswar K. Panda. High performance VMM-bypass I/O in virtual machines. In USENIX Annual Technical Conference (ATC), pages 29--42, 2006. http://usenix.org/legacy/event/usenix06/tech/liu.html.Google Scholar
Digital Library
- Dra\vzen Lu\vcanin, Foued Jrad, Ivona Brandic, and Achim Streit. Energy-aware cloud management through progressive SLA specification. In International Conference on Economics of Grids, Clouds, Systems, and Services (GEOCON), pages 83--98. Springer, 2014. http://dx.doi.org/10.1007/978--3--319--14609--6_6.Google Scholar
- J. Mauro, S. Shepler, and V. Tarasov. Filebench. http://sourceforge.net/projects/filebench/. Accessed: Oct, 2012.Google Scholar
- Eyal Moscovici, Yossi Kuperman, Joel Nider, Razya Ladelsky, Abel Gordon, and Dan Tsafrir. Dynamic sidecore allocation. In preparation.Google Scholar
- David Mytton. Network performance at AWS, Google, Rackspace and Softlayer. https://blog.serverdensity.com/network-performance-aws-google-rackspace-softlayer, Apr 2014. (Accessed: Aug 2015).Google Scholar
- Zhenhao Pan, Yaozu Dong, Yu Chen, Lei Zhang, and Zhijiao Zhang. CompSC: Live migration with pass-through devices. In ACM International Conference on Virtual Execution Environments (VEE), pages 109--120, 2012. http://dx.doi.org/10.1145/2151024.2151040.Google Scholar
- PCI-SIG. Single root I/O virtualization and sharing 1.1 specification. http://www.pcisig.com/specifications/iov/single_root/, Jan 2010.Google Scholar
- Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe. Arrakis: The operating system is the control plane. In USENIX Symposium on Operating System Design and Implementation (OSDI), pages 1--16, 2014. https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-peter_simon.pdf.Google Scholar
Digital Library
- Gerald J. Popek and Robert P. Goldberg. Formal requirements for virtualizable third generation architectures. Communications of the ACM (CACM), 17(7):412--421, Jul 1974. http://dx.doi.org/10.1145/361011.361073.Google Scholar
- Jesse Proudman. Live migration is a perk, not a panacea. URL https://www.blueboxcloud.com/insight/blog-article/live-migration-is-a-perk-not-a-panacea, Oct 2014.Google Scholar
- Himanshu Raj and Karsten Schwan. High performance and scalable I/O virtualization via self-virtualized devices. In IEEE International Symposium on High Performance Distributed Computing (HPDC), pages 179--188, 2007. http://dx.doi.org/10.1145/1272366.1272390.Google Scholar
Digital Library
- Kaushik Kumar Ram, Alan L. Cox, Mehul Chadha, and Scott Rixner. Hyper-Switch: A scalable software virtual switching architecture. In USENIX Annual Technical Conference (ATC), pages 13--24, 2013. https://www.usenix.org/conference/atc13/technical-sessions/presentation/ram.Google Scholar
- Rusty Russell. virtio: towards a de-facto standard for virtual I/O devices. ACM Operating System Review (OSR), 42(5):95--103, 2008. http://dx.doi.org/10.1145/1400097.1400108.Google Scholar
- Felix Salfner, Peter Tröger, and Andreas Polze. Downtime analysis of virtual machine live migration. In International Conference on Dependability (DEPEND), pages 100--105, 2011. http://tinyurl.com/DEPEND11-vm-downtime.Google Scholar
- SanDisk Corp. Fusion ioMemory SX300 spec. http://www.fusionio.com/load/-media-/302x3j/docsLibrary/SX300_DS_Final_v3.pdf, 2014. (Accessed: Aug 2015).Google Scholar
- K.E. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett. Server-directed collective I/O in Panda. In ACM/IEEE Supercomputing (SC), page No. 57, 1995. http://dx.doi.org/10.1145/224170.224371.Google Scholar
- Arvind Seshadri, Mark Luk, Ning Qu, and Adrian Perrig. SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In ACM Symposium on Operating Systems Principles (SOSP), pages 335--350, 2007. http://dx.doi.org/10.1145/1294261.1294294.Google Scholar
Digital Library
- Mark Silberstein, Bryan Ford, Idit Keidar, and Emmett Witchel. GPUfs: Integrating a file system with GPUs. In ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 485--498, 2003. http://dx.doi.org/10.1145/2451116.2451169.Google Scholar
- R. Takano, H. Nakada, T. Hirofuchi, Y. Tanaka, and T. Kudoh. Cooperative VM migration for a virtualized HPC cluster with VMM-bypass I/O devices. In IEEE International Conference on E-Science (e-Science), pages 1--8, 2012. http://dx.doi.org/10.1109/eScience.2012.6404487.Google Scholar
Digital Library
- Cheng-Chun Tu, Chao-tang Lee, and Tzi-cker Chiueh. Secure I/O device sharing among virtual machines on multiple hosts. In ACM International Symposium on Computer Architecture (ISCA), pages 108--119, 2013. http://dx.doi.org/10.1145/2485922.2485932.Google Scholar
- VMware, Inc. VMware ESX server 2 - architecture and performance implications. Technical Report ESX-ENG-Q305--122, VMware, 2005. https://www.vmware.com/pdf/esx2_performance_implications.pdf.Google Scholar
- VMware, Inc. Performance evaluation of VMXNET3 virtual network device. http://www.vmware.com/pdf/vsp_4_vmxnet3_perf.pdf, 2009.Google Scholar
- William Voorsluys, James Broberg, Srikumar Venugopal, and Rajkumar Buyya. Cost of virtual machine live migration in clouds: A performance evaluation. In International Conference on Cloud Computing, pages 254--265. Springer-Verlag, 2009. http://dx.doi.org/10.1007/978--3--642--10665--1_23.Google Scholar
Digital Library
- Carl Waldspurger and Mendel Rosenblum. I/O virtualization. Communications of the ACM (CACM), 55(1):66--73, Jan 2012. http://dx.doi.org/10.1145/2063176.2063194.Google Scholar
- Andrew Whitaker, Marianne Shaw, and Steven D. Gribble. Denali: Lightweight virtual machines for distributed and networked applications. Technical Report 02-02-01, University of Washington, 2002. http://www.cs.washington.edu/tr/2002/02/UW-CSE-02-02-01.pdf.Google Scholar
- Paul Willmann, Scott Rixner, and Alan L. Cox. Protection strategies for direct access to virtualized I/O devices. In USENIX Annual Technical Conference (ATC), pages 15--28, 2008. https://www.usenix.org/legacy/event/usenix08/tech/full_papers/willmann/willmann.pdf.Google Scholar
Digital Library
- Paul Willmann, Jeffrey Shafer, David Carr, Aravind Menon, Scott Rixner, Alan L. Cox, and Willy Zwaenepoel. Concurrent direct network access for virtual machine monitors. In IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 306--317, 2007. http://dx.doi.org/10.1109/HPCA.2007.346208.Google Scholar
- Timothy Wood, Prashant Shenoy, Arun Venkataramani, and Mazin Yousif. Black-box and gray-box strategies for virtual machine migration. In USENIX Symposium on Networked Systems Design and Implementation (NSDI), pages 17--17, 2007. http://usenix.org/event/nsdi07/tech/full_papers/wood/wood.pdf.Google Scholar
- Cong Xu, Sahan Gamage, Hui Lu, Ramana Kompella, and Dongyan Xu. vTurbo: Accelerating virtual machine I/O processing using designated turbo-sliced core. In USENIX Annual Technical Conference (ATC), pages 243--254, 2013. https://www.usenix.org/conference/atc13/technical-sessions/presentation/xu.Google Scholar
- Frank Yang. 10GBASE-T economics. http://www.commscope.com/Blog/10gbase-t-economics/, 2012. (Accessed: Aug 2015).Google Scholar
- Ben-Ami Yassour, Muli Ben-Yehuda, and Orit Wasserman. Direct device assignment for untrusted fully-virtualized virtual machines. Technical Report H-0263, IBM Research, 2008. http://tinyurl.com/IBM-TR-H-0263.Google Scholar
- Edwin Zhai, Gregory D. Cummings, and Yaozu Dong. Live migration with pass-through device for Linux VM. In Ottawa Linux Symposium (OLS), pages 261--268, 2008. https://www.kernel.org/doc/ols/2008/ols2008v2-pages-261--267.pdf.Google Scholar
Index Terms
Paravirtual Remote I/O
Recommendations
Paravirtual Remote I/O
ASPLOS '16: Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating SystemsThe traditional "trap and emulate" I/O paravirtualization model conveniently allows for I/O interposition, yet it inherently incurs costly guest-host context switches. The newer "sidecore" model eliminates this overhead by dedicating host (side)cores to ...
Paravirtual Remote I/O
ASPLOS'16The traditional "trap and emulate" I/O paravirtualization model conveniently allows for I/O interposition, yet it inherently incurs costly guest-host context switches. The newer "sidecore" model eliminates this overhead by dedicating host (side)cores to ...
Virtual Machine Migration Method between Different Hypervisor Implementations and Its Evaluation
WAINA '12: Proceedings of the 2012 26th International Conference on Advanced Information Networking and Applications WorkshopsVirtualization technologies are an important building block for cloud services. Each service will run on virtual machines (VMs) deployed over different hyper visors in the future. Therefore, a VM migration method between different hyper visor ...







Comments