Sai Sha
ABSTRACT
The memory demand of virtual machines (VMs) is increasing, while the traditional DRAM-only memory system has limited capacity and high power consumption. The tiered memory system can effectively expand the memory capacity and increase the cost efficiency. Virtualization introduces new challenges for memory tiering, specifically enforcing performance isolation, minimizing context switching, and providing resource overcommit. However, none of the state-of-the-art designs consider virtualization and thus address these challenges; we observe that a VM with tiered memory incurs up to a 2× slowdown compared to a DRAM-only VM.
This paper proposes vTMM, a tiered memory management system specifically designed for virtualization. vTMM automatically determines page hotness and migrates pages between fast and slow memory to achieve better performance. A key insight in vTMM is to leverage the unique system characteristics in virtualization to meet the above challenges. Specifically, vTMM tracks memory accesses with page-modification logging (PML) and a multi-level queue design. Next, vTMM quantifies the page "temperature" and makes a fine-grained page classification with bucket-sorting. vTMM performs page migration with PML while providing resource overcommit by transparently resizing VM memory through the two-dimensional page tables. In combination, the above techniques minimize overhead, ensure performance isolation and provide dynamic memory partitioning to improve the overall system performance.
We evaluate vTMM on a real DRAM+NVM system and a simulated CXL-Memory system. The results show that vTMM outperforms NUMA balancing, Intel Optane memory mode and Nimble (an OS-level tiered memory management system) for VM tiered memory management. Multi-VM co-running results show that vTMM improves the performance of a DRAM+NVM system by 50%--140% and a CXL-Memory system by 16% -- 40%, respectively.
- Neha Agarwal and Thomas F. Wenisch. 2017. Thermostat: Application-transparent Page Management for Two-tiered Main Memory. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems. 631--644.Google Scholar
- S. Beamer, K. Asanovi, and D. Patterson. 2015. http://arxiv.org/abs/1508.03619. The GAP Benchmark Suite. arXiv e-prints (2015. http://arxiv.org/abs/1508.03619).Google Scholar
- Stella Bitchebe, Djob Mvondo, Laurent Réveillère, Noël De Palma, and Alain Tchana. 2021. Extending Intel PML for Hardware-assisted Working Set Size Estimation of VMs. In Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 111--124.Google ScholarDigital Library
- Daniel P Bovet and Marco Cesati. 2005. Understanding the Linux Kernel. " O'Reilly Media, Inc.". (Section 17.3.1.1 of page 828).Google Scholar
- Apostolos Burnetas, Daniel Solow, and Rishi Agarwal. 1997. An Analysis and Implementation of An Efficient In-place Bucket Sort. Acta Informatica 34, 9 (1997), 687--700.Google ScholarCross Ref
- Linux community. 2021. Intel PML Buffer Flush. https://elixir.bootlin.com/linux/v5.4/source/arch/x86/kvm/vmx/vmx.c.Google Scholar
- Linux community. 2022. migrate_pages() Function of Linux Kernel. https://elixir.bootlin.com/linux/v5.4/source/mm/migrate.c#L1399.Google Scholar
- Qemu community. 2021. QEMU. https://www.qemu.org/.Google Scholar
- CXL Consortium. 2022. CXL. https://www.computeexpresslink.org/.Google Scholar
- B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. 2010. Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010, Indianapolis, Indiana, USA, June 10--11, 2010.Google Scholar
- Intel Corporation. 2022. Page Modification Logging for Virtual Machine Monitor White Paper. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/page-modification-logging-vmm-white-paper.pdf.Google Scholar
- Standard Performance Evaluation Corporation. 2022. SPEC CPU 2006 benchmarks. http://www.spec.org/cpu2006.Google Scholar
- Standard Performance Evaluation Corporation. 2022. SPEC CPU 2017 benchmarks. http://www.spec.org/cpu2017.Google Scholar
- Graph500 developers. 2022. Graph500. http://graph500.org/.Google Scholar
- NUMA developers. 2022. A NUMA API for Linux. http://developer.amd.com/wordpress/media/2012/10/LibNUMA-WP-fv1.pdf.Google Scholar
- G. Dhiman, R. Ayoub, and T. Rosing. 2009. PDRAM: A hybrid PRAM and DRAM main memory system. In IEEE Design Automation Conference.Google Scholar
- Mingkai Dong, Heng Bu, Jifei Yi, Benchao Dong, and Haibo Chen. 2019. Performance and Protection in the ZoFS User-space NVM File System. In Proceedings of the 27th ACM Symposium on Operating Systems Principles. 478--493.Google ScholarDigital Library
- Jayneel Gandhi, Arkaprava Basu, Mark D. Hill, and Michael M. Swift. 2014. BadgerTrap: a tool to instrument x86-64 TLB misses. Acm Sigarch Computer Architecture News 42, 2 (2014), 20--23.Google ScholarDigital Library
- Fabien Gaud, Baptiste Lepers, Jeremie Decouchant, Justin Funston, Alexandra Fedorova, and Vivien Quéma. 2014. Large Pages May Be Harmful on NUMA Systems. In 2014 {USENIX} Annual Technical Conference ({USENIX} {ATC} 14). 231--242.Google Scholar
- Abel Gordon, Michael Hines, Dilma Da Silva, Muli Ben-Yehuda, Marcio Silva, and Gabriel Lizarraga. 2011. Ginkgo: Automated, application-driven memory overcommitment for cloud computing. Proc. RESoLVE (2011).Google Scholar
- Takahiro Hirofuchi and Ryousei Takano. 2016. RAMinate: Hypervisor-based Virtualization for Hybrid Main Memory Systems. In Proceedings of the Seventh ACM Symposium on Cloud Computing. 112--125.Google ScholarDigital Library
- Jingyuan Hu, Xiaokuang Bai, Sai Sha, Yingwei Luo, Xiaolin Wang, and Zhenlin Wang. 2018. HUB: Hugepage Ballooning in Kernel-based Virtual Machines. In Proceedings of the International Symposium on Memory Systems. 31--37.Google ScholarDigital Library
- Xiameng Hu, Xiaolin Wang, Lan Zhou, Yingwei Luo, Chen Ding, and Zhenlin Wang. 2016. Kinetic Modeling of Data Eviction in Cache. In 2016 USENIX Annual Technical Conference (USENIX ATC 16). 351--364.Google ScholarDigital Library
- Intel Inc. 2021. Intel Optane DC Persistent Memory. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html.Google Scholar
- Intel Inc. 2022. Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3B. https://www.intel.cn/content/www/cn/zh/architecture-and-technology/64-ia-32-architectures-software-developer-vol-3b-part-2-manual.html.Google Scholar
- Intel Inc. 2022. Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3C. https://www.intel.cn/content/www/cn/zh/architecture-and-technology/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.html.Google Scholar
- Intel Inc. 2022. PerfMon Events. https://perfmon-events.intel.com/#.Google Scholar
- Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R Dulloor, et al. 2019. Basic Performance Measurements of the Intel Optane DC Persistent Memory Module. arXiv preprint arXiv:1903.05714 (2019).Google Scholar
- Sudarsun Kannan, Ada Gavrilovska, Vishal Gupta, and Karsten Schwan. 2017. Heteroos: OS Design for Heterogeneous Memory Management in Datacenter. In Proceedings of the 44th Annual International Symposium on Computer Architecture. 521--534.Google ScholarDigital Library
- The kernel development community. 2022. DAMON. https://damonitor.github.io/doc/html/latest-damon/vm/damon/index.html.Google Scholar
- Jonghyeon Kim, Wonkyo Choe, and Jeongseob Ahn. 2021. Exploring the Design Space of Page Management for Multi-Tiered Memory Systems. In 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14--16, 2021, Irina Calciu and Geoff Kuenning (Eds.). USENIX Association, 715--728. https://www.usenix.org/conference/atc21/presentation/kim-jonghyeonGoogle Scholar
- C. Lefurgy, K. Rajamani, F. Rawson, W. Felter, M. Kistler, and T. W. Keller. 2003. Energy Management for Commercial Servers. Computer 36, 12 (2003), 39--48.Google ScholarDigital Library
- Redis Ltd. 2022. Redis. https://redis.io/.Google Scholar
- Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit O. Kanaujia, and Prakash Chauhan. 2022. TPP: Transparent Page Placement for CXL-Enabled Tiered Memory. CoRR abs/2206.02878 (2022).Google Scholar
- Timothy Merrifield and H Reza Taheri. 2016. Performance Implications of Extended Page Tables on Virtualized X86 Processors. In Proceedings of the12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 25--35.Google ScholarDigital Library
- Vlad Nitu, Boris Teabe, Alain Tchana, Canturk Isci, and Daniel Hagimont. 2018. Welcome to Zombieland: Practical and Energy-Efficient Memory Disaggregation in a Datacenter. In Proceedings of the Thirteenth EuroSys Conference (Porto, Portugal) (EuroSys '18). Association for Computing Machinery, New York, NY, USA, Article 16, 12 pages. Google ScholarDigital Library
- Cheng Pan, Xiameng Hu, Lan Zhou, Yingwei Luo, Xiaolin Wang, and Zhenlin Wang. 2018. PACE: Penalty Aware Cache Modeling with Enhanced AET. In Proceedings of the 9th Asia-Pacific Workshop on Systems. 1--8.Google ScholarDigital Library
- Moinuddin K. Qureshi, Vijayalakshmi Srinivasan, and Jude A. Rivers. 2009. Scalable high performance main memory system using phase-change memory technology. Computer architecture news (2009).Google Scholar
- Amanda Raybuck, Tim Stamler, Wei Zhang, Mattan Erez, and Simon Peter. 2021. HeMem: Scalable Tiered Memory Management for Big Data Applications and Real NVM. In Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles. 392--407.Google ScholarDigital Library
- Sai Sha, Yi Zhang, Yingwei Luo, Xiaolin Wang, and Zhenlin Wang. 2021. Swift Shadow Paging (SSP): No Write-protection but Following TLB Flushing. In Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments. 29--42.Google ScholarDigital Library
- Rik van Riel and Vinod Chegu. 2022. Automatic NUMA Balancing. https://www.redhat.com/files/summit/2014/summit2014_riel_chegu_w_0340_automatic_numa_balancing.pdf..Google Scholar
- X. Wang, X. Liao, H. Liu, and H. Jin. 2018. Big Data Oriented Hybrid Memory Systems. Big Data Research (2018).Google Scholar
- Zi Yan, Daniel Lustig, David Nellans, and Abhishek Bhattacharjee. 2019. Nimble Page Management for Tiered Memory Systems. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems. 331--345.Google ScholarDigital Library
- Ting Yao, Yiwen Zhang, Jiguang Wan, Qiu Cui, Liu Tang, Hong Jiang, Changsheng Xie, and Xubin He. 2020. atrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with Matrix Container in NVM. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). 17--31.Google Scholar
Index Terms
vTMM: Tiered Memory Management for Virtual Machines
Recommendations
Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension
We are developing an efficient resource management system with aggressive virtual machine (VM) relocation among physical nodes in a data center. Existing live migration technology, however, requires a long time to change the execution host of a VM, it ...
Traffic-sensitive live migration of virtual machines
In this paper we address the problem of network contention between the migration traffic and the Virtual Machine (VM) application traffic for the live migration of co-located Virtual Machines. When VMs are migrated with pre-copy, they run at the source ...
Fast Server Deprovisioning through Scatter-Gather Live Migration of Virtual Machines
Traditional metrics for live migration of virtual machines (VM) include total migration time, downtime, network overhead, and application degradation. In this paper, we introduce a new metric, "eviction time", defined as the time to evict the entire ...
Comments