Abstract
Virtualization is becoming increasingly common in data centers due to its various advantages. However, how to choose among different platforms, including both software and hardware, is a considerable challenge. In this context, evaluating the virtualization capabilities of different platforms is critically important. Regrettably, the existing benchmarks are not qualified for meeting this requirement. Different hardware mechanisms and hypervisor designs introduce many different hypervisor-level events, such as transitions between VMs and the hypervisor, two-dimensional page walk, and binary translation. These events are key factors affecting virtualization performance. Existing benchmarks either overlook these changes or are tightly coupled to a particular hypervisor. In this paper, we present HyperBench, a benchmark suite that focuses on the capabilities of different virtualization platforms. Currently, we design 15 hypervisor benchmarks covering CPU, memory, and I/O. The virtualization-sensitive operation in each benchmark triggers hypervisor-level events, which examines the platform's ability in the target area. HyperBench is designed as a custom kernel which can adapt to different hypervisors and architectures. What's more, adding a new benchmark is pretty easy. Finally, we perform a series of experiments on the host machine and several popular hypervisors, such as QEMU, KVM, and Xen, demonstrating that HyperBench is capable of revealing the performance implications of the hardware mechanism and hypervisor design.
- Keith Adams and Ole Agesen. 2006. A comparison of software and hardware techniques for x86 virtualization. ACM SIGARCH Computer Architecture News, Vol. 34, 5 (2006), 2--13. Google Scholar
Digital Library
- Ole Agesen, Jim Mattson, Radu Rugina, and Jeffrey Sheldon. 2012. Software Techniques for Avoiding Hardware Virtualization Exits. In USENIX Annual Technical Conference. 373--385. Google Scholar
Digital Library
- Fabrice Bellard. 2005. QEMU, a fast and portable dynamic translator. In USENIX Annual Technical Conference, FREENIX Track, Vol. 41. 46. Google Scholar
Digital Library
- Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques. ACM, 72--81. Google Scholar
Digital Library
- Jeffrey P Casazza, Michael Greenfield, and Kan Shi. 2006. Redefining server performance characterization for virtualization benchmarking. Intel Technology Journal, Vol. 10, 3 (2006).Google Scholar
Cross Ref
- Tianshi Chen, Qi Guo, Olivier Temam, Yue Wu, Yungang Bao, Zhiwei Xu, and Yunji Chen. 2015. Statistical performance comparisons of computers. IEEE Trans. Comput., Vol. 64, 5 (2015), 1442--1455.Google Scholar
Digital Library
- KVM contributors. 2017. KVM-unit-tests. https://www.linux-kvm.org/index.php?title=KVM-unit-tests&oldid=173824 Retrieved February 18, 2019 fromGoogle Scholar
- Unixbench contributors. 2018. Unixbench. https://github.com/kdlucas/byte-unixbench Retrieved February 18, 2019 fromGoogle Scholar
- Intel Corporation. 2017. Intel 64 and IA-32 Architectures Software Developer Manual.Google Scholar
- Intel Corporation. 2017. Intel Virtualization Technology for Directed I/O .Google Scholar
- Christoffer Dall, Shih-Wei Li, Jin Tack Lim, Jason Nieh, and Georgios Koloventzos. 2016. ARM virtualization: performance and architectural implications. ACM SIGARCH Computer Architecture News, Vol. 44, 3 (2016), 304--316. Google Scholar
Digital Library
- Advanced Micro Devices. 2005. AMD64 Virtualization Codenamed "acifica" Technology: Secure Virtual Machine Architecture Reference Manual .Google Scholar
- Artem Dinaburg, Paul Royal, Monirul Sharif, and Wenke Lee. 2008. Ether: malware analysis via hardware virtualization extensions. In Proceedings of the 15th ACM conference on Computer and communications security. ACM, 51--62. Google Scholar
Digital Library
- Tal Garfinkel, Ben Pfaff, Jim Chow, Mendel Rosenblum, and Dan Boneh. 2003. Terra: A virtual machine-based platform for trusted computing. In ACM SIGOPS Operating Systems Review, Vol. 37. ACM, 193--206. Google Scholar
Digital Library
- Abel Gordon, Nadav Amit, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. 2012. ELI: bare-metal performance for I/O virtualization. ACM SIGPLAN Notices, Vol. 47, 4 (2012), 411--422. Google Scholar
Digital Library
- Abel Gordon, Nadav Har'El, Alex Landau, Muli Ben-Yehuda, and Avishay Traeger. 2012. Towards exitless and efficient paravirtual I/O. In Proceedings of the 5th Annual International Systems and Storage Conference. ACM, 10. Google Scholar
Digital Library
- Diwaker Gupta, Rob Gardner, and Ludmila Cherkasova. 2005. Xenmon: Qos monitoring and performance profiling tool. Hewlett-Packard Labs, Tech. Rep. HPL-2005--187 (2005), 1--13.Google Scholar
- John L Henning. 2006. SPEC CPU2006 benchmark descriptions. ACM SIGARCH Computer Architecture News, Vol. 34, 4 (2006), 1--17. Google Scholar
Digital Library
- Chi-Yao Hong, Matthew Caesar, and P Godfrey. 2012. Finishing flows quickly with preemptive scheduling. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM, 127--138. Google Scholar
Digital Library
- Rishi Kapoor, George Porter, Malveeka Tewari, Geoffrey M Voelker, and Amin Vahdat. 2012. Chronos: Predictable low latency for data center applications. In Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 9. Google Scholar
Digital Library
- Alex Landau, Muli Ben-Yehuda, and Abel Gordon. 2011. SplitX: Split Guest/Hypervisor Execution on Multi-Core. In WIOV . Google Scholar
Digital Library
- Sangmin Lee, Rina Panigrahy, Vijayan Prabhakaran, Venugopalan Ramasubramanian, Kunal Talwar, Lincoln Uyeda, and Udi Wieder. 2011. Validating heuristics for virtual machines consolidation. Microsoft Research, MSR-TR-2011--9 (2011), 1--14.Google Scholar
- Wanpeng Li. 2018. Torwards a more Scalable KVM Hypervisor. KVM Forum.Google Scholar
- Vikram Makhija, Bruce Herndon, Paula Smith, Lisa Roderick, Eric Zamost, and Jennifer Anderson. 2006. VMmark: A scalable benchmark for virtualized systems. Technical Report. Technical Report TR 2006-002, VMware.Google Scholar
- Larry W McVoy, Carl Staelin, et almbox. 1996. lmbench: Portable Tools for Performance Analysis. In USENIX annual technical conference. San Diego, CA, USA, 279--294. Google Scholar
Digital Library
- Aravind Menon, Jose Renato Santos, Yoshio Turner, G John Janakiraman, and Willy Zwaenepoel. 2005. Diagnosing performance overheads in the xen virtual machine environment. In Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments. ACM, 13--23.Google Scholar
Digital Library
- Timothy Merrifield and H Reza Taheri. 2016. Performance implications of extended page tables on virtualized x86 processors. ACM SIGPLAN Notices, Vol. 51, 7 (2016), 25--35. Google Scholar
Digital Library
- Jun Nakajima. 2012. Enabling optimized interrupt/APIC virtualization in KVM. KVM Forum.Google Scholar
- Michael Nelson, Beng-Hong Lim, Greg Hutchins, et almbox. 2005. Fast Transparent Migration for Virtual Machines. In USENIX Annual technical conference, general track. 391--394. Google Scholar
Digital Library
- Gabriele Paoloni. 2010. How to Benchmark Code Execution Times on Intel? IA-32 and IA-64 Instruction Set Architectures .Google Scholar
- Gerald J Popek and Robert P Goldberg. 1974. Formal requirements for virtualizable third generation architectures. Commun. ACM, Vol. 17, 7 (1974), 412--421. Google Scholar
Digital Library
- Rusty Russell. 2008. virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Systems Review, Vol. 42, 5 (2008), 95--103. Google Scholar
Digital Library
- Xiang Song, Jicheng Shi, Haibo Chen, and Binyu Zang. 2013. Schedule processes, not VCPUs. In Proceedings of the 4th Asia-Pacific Workshop on Systems. ACM, 1. Google Scholar
Digital Library
- SPEC. 2017. SPEC virt_sc 2013. http://www.spec.org/virt_sc2013/index.html Retrieved September 28, 2018 fromGoogle Scholar
- Michael S. Tsirkin. 2010. vhost-net and virtio-net: Need for Speed. KVM Forum.Google Scholar
- Cheng-Chun Tu, Michael Ferdman, Chao-tung Lee, and Tzi-cker Chiueh. 2015. A comprehensive implementation and evaluation of direct interrupt delivery. In ACM SIGPLAN Notices, Vol. 50. ACM, 1--15. Google Scholar
Digital Library
- Werner Vogels. 2008. Beyond server consolidation. Queue, Vol. 6, 1 (2008), 20--26. Google Scholar
Digital Library
- Harry Wagstaff, Bruno Bodin, Tom Spink, and Björn Franke. 2017. SimBench: A portable benchmarking methodology for full-system simulators. In Performance Analysis of Systems and Software (ISPASS), 2017 IEEE International Symposium on. IEEE, 217--226.Google Scholar
Cross Ref
- xv6 contributors. 2018. xv6. https://github.com/mit-pdos/xv6-public Retrieved February 18, 2019 fromGoogle Scholar
- David Zats, Tathagata Das, Prashanth Mohan, Dhruba Borthakur, and Randy Katz. 2012. DeTail: reducing the flow completion time tail in datacenter networks. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM, 139--150. Google Scholar
Digital Library
- Binbin Zhang, Xiaolin Wang, Rongfeng Lai, Liang Yang, Zhenlin Wang, Yingwei Luo, and Xiaoming Li. 2010. Evaluating and optimizing I/O virtualization in kernel-based virtual machine (KVM). In IFIP International Conference on Network and Parallel Computing. Springer, 220--231. Google Scholar
Digital Library
Index Terms
HyperBench: A Benchmark Suite for Virtualization Capabilities
Recommendations
HyperBench: A Benchmark Suite for Virtualization Capabilities
Virtualization is ubiquitous in modern data centers. By deploying applications on separate virtual machines hosted in a shared physical machine, it brings benefits over traditional systems in resources utilization[5, 10], system security[2, 3], and ...
NEVE: Nested Virtualization Extensions for ARM
SOSP '17: Proceedings of the 26th Symposium on Operating Systems PrinciplesNested virtualization, the ability to run a virtual machine inside another virtual machine, is increasingly important because of the need to deploy virtual machines running software stacks on top of virtualized cloud infrastructure. As ARM servers make ...
HyperBench: A Benchmark Suite for Virtualization Capabilities
SIGMETRICS '19: Abstracts of the 2019 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer SystemsVirtualization is becoming increasingly common in data centers due to its various advantages. However, how to choose among different platforms, including both software and hardware, is a considerable challenge. In this context, evaluating the ...






Comments