skip to main content
research-article

Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi

Published:14 March 2015Publication History
Skip Abstract Section

Abstract

VMware ESXi leverages hardware support for MMU virtualization available in modern Intel/AMD CPUs. To optimize address translation performance when running on such CPUs, ESXi preferably uses host large pages (2MB in x86-64 systems) to back VM's guest memory. While using host large pages provides best performance when host has sufficient free memory, it increases host memory pressure and effectively defeats page sharing. Hence, the host is more likely to hit the point where ESXi has to reclaim VM memory through much more expensive techniques such as ballooning or host swapping. As a result, using host large pages may significantly hurt consolidation ratio.

To deal with this problem, we propose a new host large page management policy that allows to: a) identify 'cold' large pages and break them even when host has plenty of free memory; b) break all large pages proactively when host free memory becomes scarce, but before the host starts ballooning or swapping; c) reclaim the small pages within the broken large pages through page sharing. With the new policy, the shareable small pages can be shared much earlier and the amount of memory that needs to be ballooned or swapped can be largely reduced when host memory pressure is high. We also propose an algorithm to dynamically adjust the page sharing rate when proactively breaking large pages using a VM large page shareability estimator for higher efficiency.

Experimental results show that the proposed large page management policy can improve the performance of various workloads up to 2.1x by significantly reducing the amount of ballooned or swapped memory when host memory pressure is high. Applications still fully benefit from host large pages when memory pressure is low.

References

  1. AMD-VTM Nested Paging. http://developer.amd.com/wordpress/media/ 2012/10/NPT-WP-1%201-final-TM. pdf.Google ScholarGoogle Scholar
  2. Dell DVD Store. http://en.community.dell.com/techcenter/extras/ w/wiki/dvd-store.aspx.Google ScholarGoogle Scholar
  3. SPECjbb2005. http://www.spec.org/jbb2005/.Google ScholarGoogle Scholar
  4. Swingbench. http://www.dominicgiles.com/swingbench.html.Google ScholarGoogle Scholar
  5. Vmmark2.x.http://www.vmware.com/products/vmmark/.Google ScholarGoogle Scholar
  6. Xen transcendent memory. https://oss.oracle.com/projects/tmem/,2010.Google ScholarGoogle Scholar
  7. Hyper-v dynamic memory overview. http://technet.microsoft.com/en-us/library/hh831766.aspx.Google ScholarGoogle Scholar
  8. Xcp faq dynamic memory control. http://wiki.xenproject.org/wiki/XCP_FAQ_Dynamic_Memory_Control, 2013.Google ScholarGoogle Scholar
  9. B. Agrawal, R. Bidarkar, S. Satnur, L. Spracklen, T. M. Ismail, U. Kurkure, and V. Makhija. Vmware view planner: Measuring true virtual desktops experience at scale. VMware Tech Journal, December 2012.Google ScholarGoogle Scholar
  10. A. Arcangeli, I. Eidus, and C. Wright. Increasing memory density by using KSM. In Proceedings of the Linux Symposium, pages 313--328, 2009.Google ScholarGoogle Scholar
  11. I. Banerjee, F. Guo, K. Tati, and R. Venkatasubramanian. Memory overcommitment in the esx server. https://labs.vmware.com/vmtj/memory-overcommitment-in-the-esx-server, 2013.Google ScholarGoogle Scholar
  12. S. Barker, T. Wood, P. Shenoy, and R. Sitaraman. An Empirical Study of Memory Sharing in Virtual Machines. In Proceedings of the USENIX Annual Technical Conference, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. N. Bhatia. Performance evaluation of intel ept hardware assit. https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval. pdf, 2009.Google ScholarGoogle Scholar
  14. C. Chang, J. Wu, and P. Liu. An empirical study on memory sharing of virtual machines for server consolidation. In IEEE 9th International Symposium on Parallel and Distributed Processing with Applications (ISPA), pages 244--249, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Z. Fang, L. Zhang, J. B. Carter,W. C. Hsieh, and S. A. Mckee. Reevaluating online superpage promotion with hardware support. In Proceedings of the International Symposium on High Performance Computer Architecture, pages 63--72, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. N. Ganapathy and C. Schimmel. General purpose operating system support for multiple page sizes. In Proceedings of the USENIX Annual Technical Conference, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. F. Gaud, B. Lepers, J. Decouchant, J. Funston, A. Fedorova, and V. Quema. Large pages may be harmful on numa systems. In Proceedings of the USENIX Annual Technical Conference, pages 231--242, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. K. Govil, D. Teodosiu, Y. Huang, and M. Rosenblum. Cellular disco: resource management using virtual clusters on sharedmemory multiprocessors. In Proceedings of the seventeenth ACM symposium on Operating systems principles, SOSP '99, pages 154--169, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. F. Guo. Understand memory resource management in VMware vsphere 5.0. http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5. pdf, 2010.Google ScholarGoogle Scholar
  20. D. Gupta, S. Lee, M. Vrable, S. Savage, A. C. Snoeren, G. Varghese, G. M. Voelker, and A. Vahdat. Difference engine: harnessing memory redundancy in virtual machines. Commun. ACM, 53(10):85--93, Oct. 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. Jiang, F. Chen, and X. Zhang. Clock-pro: An effective improvement of the clock replacement. In Proceedings of the USENIX Annual Technical Conference, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. J. F. Kloster, J. Kritensen, and A. Mejlholm. Efficient memory sharing in the xen virtual machine monitor. http://mejlholm.org/uni/pdfs/dat5.pdf.Google ScholarGoogle Scholar
  23. K. Miller, F. Franz, M. Rittinghaus, M. Hillenbrand, and F. Bellosa. XLH: More Effective Memory Deduplication Scanners Through Cross-layer Hints. In Proceedings of the USENIX Annual Technical Conference, pages 279--290, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. G. Miłós, D. Murray, S. Hand, and M. Fetterman. Satori: Enlightened page sharing. In Proceedings of the USENIX Annual Technical Conference, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. J. Navarro, S. Iyer, P. Druschel, and A. Cox. Practical transparent operating system support for superpages. SIGOPS Operating System Review, 36(SI):89--104, Dec. 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. P. Sharma and P. Kulkarni. Singleton: system-wide page deduplication in virtual environments. In Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing, HPDC '12, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. I. Subramanian, C. Mather, K. Peterson, and B. Raghunath. Implementation of multiple pagesize support in hp-ux. In Proceedings of the USENIX Annual Technical Conference, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. C. A. Waldspurger. Memory resource management in VMware ESX server. SIGOPS Operating System Review, 36(SI):181--194, Dec. 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Proactively Breaking Large Pages to Improve Memory Overcommitment Performance in VMware ESXi

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!