skip to main content
research-article
Open Access

HyperBench: A Benchmark Suite for Virtualization Capabilities

Authors Info & Claims
Published:19 June 2019Publication History
Skip Abstract Section

Abstract

Virtualization is becoming increasingly common in data centers due to its various advantages. However, how to choose among different platforms, including both software and hardware, is a considerable challenge. In this context, evaluating the virtualization capabilities of different platforms is critically important. Regrettably, the existing benchmarks are not qualified for meeting this requirement. Different hardware mechanisms and hypervisor designs introduce many different hypervisor-level events, such as transitions between VMs and the hypervisor, two-dimensional page walk, and binary translation. These events are key factors affecting virtualization performance. Existing benchmarks either overlook these changes or are tightly coupled to a particular hypervisor. In this paper, we present HyperBench, a benchmark suite that focuses on the capabilities of different virtualization platforms. Currently, we design 15 hypervisor benchmarks covering CPU, memory, and I/O. The virtualization-sensitive operation in each benchmark triggers hypervisor-level events, which examines the platform's ability in the target area. HyperBench is designed as a custom kernel which can adapt to different hypervisors and architectures. What's more, adding a new benchmark is pretty easy. Finally, we perform a series of experiments on the host machine and several popular hypervisors, such as QEMU, KVM, and Xen, demonstrating that HyperBench is capable of revealing the performance implications of the hardware mechanism and hypervisor design.

References

  1. Keith Adams and Ole Agesen. 2006. A comparison of software and hardware techniques for x86 virtualization. ACM SIGARCH Computer Architecture News, Vol. 34, 5 (2006), 2--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ole Agesen, Jim Mattson, Radu Rugina, and Jeffrey Sheldon. 2012. Software Techniques for Avoiding Hardware Virtualization Exits. In USENIX Annual Technical Conference. 373--385. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Fabrice Bellard. 2005. QEMU, a fast and portable dynamic translator. In USENIX Annual Technical Conference, FREENIX Track, Vol. 41. 46. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In Proceedings of the 17th international conference on Parallel architectures and compilation techniques. ACM, 72--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Jeffrey P Casazza, Michael Greenfield, and Kan Shi. 2006. Redefining server performance characterization for virtualization benchmarking. Intel Technology Journal, Vol. 10, 3 (2006).Google ScholarGoogle ScholarCross RefCross Ref
  6. Tianshi Chen, Qi Guo, Olivier Temam, Yue Wu, Yungang Bao, Zhiwei Xu, and Yunji Chen. 2015. Statistical performance comparisons of computers. IEEE Trans. Comput., Vol. 64, 5 (2015), 1442--1455.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. KVM contributors. 2017. KVM-unit-tests. https://www.linux-kvm.org/index.php?title=KVM-unit-tests&oldid=173824 Retrieved February 18, 2019 fromGoogle ScholarGoogle Scholar
  8. Unixbench contributors. 2018. Unixbench. https://github.com/kdlucas/byte-unixbench Retrieved February 18, 2019 fromGoogle ScholarGoogle Scholar
  9. Intel Corporation. 2017. Intel 64 and IA-32 Architectures Software Developer Manual.Google ScholarGoogle Scholar
  10. Intel Corporation. 2017. Intel Virtualization Technology for Directed I/O .Google ScholarGoogle Scholar
  11. Christoffer Dall, Shih-Wei Li, Jin Tack Lim, Jason Nieh, and Georgios Koloventzos. 2016. ARM virtualization: performance and architectural implications. ACM SIGARCH Computer Architecture News, Vol. 44, 3 (2016), 304--316. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Advanced Micro Devices. 2005. AMD64 Virtualization Codenamed "acifica" Technology: Secure Virtual Machine Architecture Reference Manual .Google ScholarGoogle Scholar
  13. Artem Dinaburg, Paul Royal, Monirul Sharif, and Wenke Lee. 2008. Ether: malware analysis via hardware virtualization extensions. In Proceedings of the 15th ACM conference on Computer and communications security. ACM, 51--62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Tal Garfinkel, Ben Pfaff, Jim Chow, Mendel Rosenblum, and Dan Boneh. 2003. Terra: A virtual machine-based platform for trusted computing. In ACM SIGOPS Operating Systems Review, Vol. 37. ACM, 193--206. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Abel Gordon, Nadav Amit, Nadav Har'El, Muli Ben-Yehuda, Alex Landau, Assaf Schuster, and Dan Tsafrir. 2012. ELI: bare-metal performance for I/O virtualization. ACM SIGPLAN Notices, Vol. 47, 4 (2012), 411--422. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Abel Gordon, Nadav Har'El, Alex Landau, Muli Ben-Yehuda, and Avishay Traeger. 2012. Towards exitless and efficient paravirtual I/O. In Proceedings of the 5th Annual International Systems and Storage Conference. ACM, 10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Diwaker Gupta, Rob Gardner, and Ludmila Cherkasova. 2005. Xenmon: Qos monitoring and performance profiling tool. Hewlett-Packard Labs, Tech. Rep. HPL-2005--187 (2005), 1--13.Google ScholarGoogle Scholar
  18. John L Henning. 2006. SPEC CPU2006 benchmark descriptions. ACM SIGARCH Computer Architecture News, Vol. 34, 4 (2006), 1--17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Chi-Yao Hong, Matthew Caesar, and P Godfrey. 2012. Finishing flows quickly with preemptive scheduling. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM, 127--138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Rishi Kapoor, George Porter, Malveeka Tewari, Geoffrey M Voelker, and Amin Vahdat. 2012. Chronos: Predictable low latency for data center applications. In Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Alex Landau, Muli Ben-Yehuda, and Abel Gordon. 2011. SplitX: Split Guest/Hypervisor Execution on Multi-Core. In WIOV . Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Sangmin Lee, Rina Panigrahy, Vijayan Prabhakaran, Venugopalan Ramasubramanian, Kunal Talwar, Lincoln Uyeda, and Udi Wieder. 2011. Validating heuristics for virtual machines consolidation. Microsoft Research, MSR-TR-2011--9 (2011), 1--14.Google ScholarGoogle Scholar
  23. Wanpeng Li. 2018. Torwards a more Scalable KVM Hypervisor. KVM Forum.Google ScholarGoogle Scholar
  24. Vikram Makhija, Bruce Herndon, Paula Smith, Lisa Roderick, Eric Zamost, and Jennifer Anderson. 2006. VMmark: A scalable benchmark for virtualized systems. Technical Report. Technical Report TR 2006-002, VMware.Google ScholarGoogle Scholar
  25. Larry W McVoy, Carl Staelin, et almbox. 1996. lmbench: Portable Tools for Performance Analysis. In USENIX annual technical conference. San Diego, CA, USA, 279--294. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Aravind Menon, Jose Renato Santos, Yoshio Turner, G John Janakiraman, and Willy Zwaenepoel. 2005. Diagnosing performance overheads in the xen virtual machine environment. In Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments. ACM, 13--23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Timothy Merrifield and H Reza Taheri. 2016. Performance implications of extended page tables on virtualized x86 processors. ACM SIGPLAN Notices, Vol. 51, 7 (2016), 25--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Jun Nakajima. 2012. Enabling optimized interrupt/APIC virtualization in KVM. KVM Forum.Google ScholarGoogle Scholar
  29. Michael Nelson, Beng-Hong Lim, Greg Hutchins, et almbox. 2005. Fast Transparent Migration for Virtual Machines. In USENIX Annual technical conference, general track. 391--394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Gabriele Paoloni. 2010. How to Benchmark Code Execution Times on Intel? IA-32 and IA-64 Instruction Set Architectures .Google ScholarGoogle Scholar
  31. Gerald J Popek and Robert P Goldberg. 1974. Formal requirements for virtualizable third generation architectures. Commun. ACM, Vol. 17, 7 (1974), 412--421. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Rusty Russell. 2008. virtio: towards a de-facto standard for virtual I/O devices. ACM SIGOPS Operating Systems Review, Vol. 42, 5 (2008), 95--103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Xiang Song, Jicheng Shi, Haibo Chen, and Binyu Zang. 2013. Schedule processes, not VCPUs. In Proceedings of the 4th Asia-Pacific Workshop on Systems. ACM, 1. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. SPEC. 2017. SPEC virt_sc 2013. http://www.spec.org/virt_sc2013/index.html Retrieved September 28, 2018 fromGoogle ScholarGoogle Scholar
  35. Michael S. Tsirkin. 2010. vhost-net and virtio-net: Need for Speed. KVM Forum.Google ScholarGoogle Scholar
  36. Cheng-Chun Tu, Michael Ferdman, Chao-tung Lee, and Tzi-cker Chiueh. 2015. A comprehensive implementation and evaluation of direct interrupt delivery. In ACM SIGPLAN Notices, Vol. 50. ACM, 1--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Werner Vogels. 2008. Beyond server consolidation. Queue, Vol. 6, 1 (2008), 20--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Harry Wagstaff, Bruno Bodin, Tom Spink, and Björn Franke. 2017. SimBench: A portable benchmarking methodology for full-system simulators. In Performance Analysis of Systems and Software (ISPASS), 2017 IEEE International Symposium on. IEEE, 217--226.Google ScholarGoogle ScholarCross RefCross Ref
  39. xv6 contributors. 2018. xv6. https://github.com/mit-pdos/xv6-public Retrieved February 18, 2019 fromGoogle ScholarGoogle Scholar
  40. David Zats, Tathagata Das, Prashanth Mohan, Dhruba Borthakur, and Randy Katz. 2012. DeTail: reducing the flow completion time tail in datacenter networks. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM, 139--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Binbin Zhang, Xiaolin Wang, Rongfeng Lai, Liang Yang, Zhenlin Wang, Yingwei Luo, and Xiaoming Li. 2010. Evaluating and optimizing I/O virtualization in kernel-based virtual machine (KVM). In IFIP International Conference on Network and Parallel Computing. Springer, 220--231. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. HyperBench: A Benchmark Suite for Virtualization Capabilities

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!