Abstract
VMware ESXi leverages hardware support for MMU virtualization available in modern Intel/AMD CPUs. To optimize address translation performance when running on such CPUs, ESXi preferably uses host large pages (2MB in x86-64 systems) to back VM's guest memory. While using host large pages provides best performance when host has sufficient free memory, it increases host memory pressure and effectively defeats page sharing. Hence, the host is more likely to hit the point where ESXi has to reclaim VM memory through much more expensive techniques such as ballooning or host swapping. As a result, using host large pages may significantly hurt consolidation ratio.
To deal with this problem, we propose a new host large page management policy that allows to: a) identify 'cold' large pages and break them even when host has plenty of free memory; b) break all large pages proactively when host free memory becomes scarce, but before the host starts ballooning or swapping; c) reclaim the small pages within the broken large pages through page sharing. With the new policy, the shareable small pages can be shared much earlier and the amount of memory that needs to be ballooned or swapped can be largely reduced when host memory pressure is high. We also propose an algorithm to dynamically adjust the page sharing rate when proactively breaking large pages using a VM large page shareability estimator for higher efficiency.
Experimental results show that the proposed large page management policy can improve the performance of various workloads up to 2.1x by significantly reducing the amount of ballooned or swapped memory when host memory pressure is high. Applications still fully benefit from host large pages when memory pressure is low.
- AMD-VTM Nested Paging. http://developer.amd.com/wordpress/media/ 2012/10/NPT-WP-1%201-final-TM. pdf.Google Scholar
- Dell DVD Store. http://en.community.dell.com/techcenter/extras/ w/wiki/dvd-store.aspx.Google Scholar
- SPECjbb2005. http://www.spec.org/jbb2005/.Google Scholar
- Swingbench. http://www.dominicgiles.com/swingbench.html.Google Scholar
- Vmmark2.x.http://www.vmware.com/products/vmmark/.Google Scholar
- Xen transcendent memory. https://oss.oracle.com/projects/tmem/,2010.Google Scholar
- Hyper-v dynamic memory overview. http://technet.microsoft.com/en-us/library/hh831766.aspx.Google Scholar
- Xcp faq dynamic memory control. http://wiki.xenproject.org/wiki/XCP_FAQ_Dynamic_Memory_Control, 2013.Google Scholar
- B. Agrawal, R. Bidarkar, S. Satnur, L. Spracklen, T. M. Ismail, U. Kurkure, and V. Makhija. Vmware view planner: Measuring true virtual desktops experience at scale. VMware Tech Journal, December 2012.Google Scholar
- A. Arcangeli, I. Eidus, and C. Wright. Increasing memory density by using KSM. In Proceedings of the Linux Symposium, pages 313--328, 2009.Google Scholar
- I. Banerjee, F. Guo, K. Tati, and R. Venkatasubramanian. Memory overcommitment in the esx server. https://labs.vmware.com/vmtj/memory-overcommitment-in-the-esx-server, 2013.Google Scholar
- S. Barker, T. Wood, P. Shenoy, and R. Sitaraman. An Empirical Study of Memory Sharing in Virtual Machines. In Proceedings of the USENIX Annual Technical Conference, 2012. Google Scholar
Digital Library
- N. Bhatia. Performance evaluation of intel ept hardware assit. https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval. pdf, 2009.Google Scholar
- C. Chang, J. Wu, and P. Liu. An empirical study on memory sharing of virtual machines for server consolidation. In IEEE 9th International Symposium on Parallel and Distributed Processing with Applications (ISPA), pages 244--249, 2011. Google Scholar
Digital Library
- Z. Fang, L. Zhang, J. B. Carter,W. C. Hsieh, and S. A. Mckee. Reevaluating online superpage promotion with hardware support. In Proceedings of the International Symposium on High Performance Computer Architecture, pages 63--72, 2001. Google Scholar
Digital Library
- N. Ganapathy and C. Schimmel. General purpose operating system support for multiple page sizes. In Proceedings of the USENIX Annual Technical Conference, 1998. Google Scholar
Digital Library
- F. Gaud, B. Lepers, J. Decouchant, J. Funston, A. Fedorova, and V. Quema. Large pages may be harmful on numa systems. In Proceedings of the USENIX Annual Technical Conference, pages 231--242, 2014. Google Scholar
Digital Library
- K. Govil, D. Teodosiu, Y. Huang, and M. Rosenblum. Cellular disco: resource management using virtual clusters on sharedmemory multiprocessors. In Proceedings of the seventeenth ACM symposium on Operating systems principles, SOSP '99, pages 154--169, 1999. Google Scholar
Digital Library
- F. Guo. Understand memory resource management in VMware vsphere 5.0. http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5. pdf, 2010.Google Scholar
- D. Gupta, S. Lee, M. Vrable, S. Savage, A. C. Snoeren, G. Varghese, G. M. Voelker, and A. Vahdat. Difference engine: harnessing memory redundancy in virtual machines. Commun. ACM, 53(10):85--93, Oct. 2010. Google Scholar
Digital Library
- S. Jiang, F. Chen, and X. Zhang. Clock-pro: An effective improvement of the clock replacement. In Proceedings of the USENIX Annual Technical Conference, 2005. Google Scholar
Digital Library
- J. F. Kloster, J. Kritensen, and A. Mejlholm. Efficient memory sharing in the xen virtual machine monitor. http://mejlholm.org/uni/pdfs/dat5.pdf.Google Scholar
- K. Miller, F. Franz, M. Rittinghaus, M. Hillenbrand, and F. Bellosa. XLH: More Effective Memory Deduplication Scanners Through Cross-layer Hints. In Proceedings of the USENIX Annual Technical Conference, pages 279--290, 2013. Google Scholar
Digital Library
- G. Miłós, D. Murray, S. Hand, and M. Fetterman. Satori: Enlightened page sharing. In Proceedings of the USENIX Annual Technical Conference, 2009. Google Scholar
Digital Library
- J. Navarro, S. Iyer, P. Druschel, and A. Cox. Practical transparent operating system support for superpages. SIGOPS Operating System Review, 36(SI):89--104, Dec. 2002. Google Scholar
Digital Library
- P. Sharma and P. Kulkarni. Singleton: system-wide page deduplication in virtual environments. In Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing, HPDC '12, 2012. Google Scholar
Digital Library
- I. Subramanian, C. Mather, K. Peterson, and B. Raghunath. Implementation of multiple pagesize support in hp-ux. In Proceedings of the USENIX Annual Technical Conference, 1998. Google Scholar
Digital Library
- C. A. Waldspurger. Memory resource management in VMware ESX server. SIGOPS Operating System Review, 36(SI):181--194, Dec. 2002. Google Scholar
Digital Library
Index Terms
Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi
Recommendations
Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi
VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution EnvironmentsVMware ESXi leverages hardware support for MMU virtualization available in modern Intel/AMD CPUs. To optimize address translation performance when running on such CPUs, ESXi preferably uses host large pages (2MB in x86-64 systems) to back VM's guest ...
Memory buddies: exploiting page sharing for smart colocation in virtualized data centers
VEE '09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environmentsMany data center virtualization solutions, such as VMware ESX, employ content-based page sharing to consolidate the resources of multiple servers. Page sharing identifies virtual machine memory pages with identical content and consolidates them into a ...
Memory buddies: exploiting page sharing for smart colocation in virtualized data centers
Many data center virtualization solutions, such as VMware ESX, employ content-based page sharing to consolidate the resources of multiple servers. Page sharing identifies virtual machine memory pages with identical content and consolidates them into a ...







Comments