Abstract
Virtualization has the potential to dramatically increase the usability and reliability of high performance computing (HPC) systems. However, this potential will remain unrealized unless overheads can be minimized. This is particularly challenging on large scale machines that run carefully crafted HPC OSes supporting tightly-coupled, parallel applications. In this paper, we show how careful use of hardware and VMM features enables the virtualization of a large-scale HPC system, specifically a Cray XT4 machine, with < = 5% overhead on key HPC applications, microbenchmarks, and guests at scales of up to 4096 nodes. We describe three techniques essential for achieving such low overhead: passthrough I/O, workload-sensitive selection of paging mechanisms, and carefully controlled preemption. These techniques are forms of symbiotic virtualization, an approach on which we elaborate.
- K. Adams and O. Agesen. A comparison of software and hardware techniques for x86 virtualization. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), October 2006. Google Scholar
Digital Library
- S. R. Alam, J. A. Kuehn, R. F. Barrett, J. M. Larkin, M. R. Fahey, R. Sankaran, and P. H. Worley. Cray XT4: an early evaluation for petascale scientific simulation. In SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing, pages 1--12, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-764-3. http://doi.acm.org/10.1145/1362622.1362675. Google Scholar
Digital Library
- PACIFICAAMD Corporation. AMD64 virtualization codenamed "Pacifica" technology: Secure Virtual Machine Architecture reference manual, May 2005.Google Scholar
- J. Appavoo, V. Uhlig, and A. Waterland. Project kittyhawk: building a global-scale computer: Blue gene/p as a generic computing platform. SIGOPS Oper. Syst. Rev., 42: 77--84, January 2008. ISSN 0163-5980. http://doi.acm.org/10.1145/1341312.1341326. URL http://doi.acm.org/10.1145/1341312.1341326. Google Scholar
Digital Library
- C. Bae, J. Lange, and P. Dinda. Comparing approaches to virtualized page translation in modern VMMs. Technical Report NWU-EECS-10-07, Department of Electrical Engineering and Computer Science, Northwestern University, April 2010.Google Scholar
- R. Bhargava, B. Serebrin, F. Spanini, and S. Manne. Accelerating two-dimensional page walks for virtualized systems. In Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), March 2008. Google Scholar
Digital Library
- R. Brightwell, T. Hudson, K. T. Pedretti, and K. D. Underwood. SeaStar Interconnect: Balanced bandwidth for scalable performance. IEEE Micro, 26 (3): 41--57, May/June 2006. Google Scholar
Digital Library
- J. E.S. Hertel, R. Bell, M. Elrick, A. Farnsworth, G. Kerley, J. McGlaun, S. Petney, S. Silling, P. Taylor, and L. Yarrington. CTH: A Software Family for Multi-Dimensional Shock Physics Analysis. In 19th International Symposium on Shock Waves, held at Marseille, France, pages 377--382, July 1993.Google Scholar
- K. B. Ferreira, R. Brightwell, and P. G. Bridges. Characterizing application sensitivity to OS interference using kernel-level noise injection. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, November 2008. Google Scholar
Digital Library
- R. Figueiredo, P. A. Dinda, and J. Fortes. A case for grid computing on virtual machines. In 23rd IEEE Conference on Distributed Computing (ICDCS 2003, pages 550--559, May 2003. Google Scholar
Digital Library
- A. Gavrilovska, S. Kumar, H. Raj, K. Schwan, V. Gupta, R. Nathuji, R. Niranjan, A. Ranadive, and P. Saraiya. High performance hypervisor architectures: Virtualization in HPC systems. In 1st Workshop on System-level Virtualization for High Performance Computing (HPCVirt), 2007.Google Scholar
- M. Heroux. HPCCG MicroApp. https://software.sandia.gov/mantevo/downloads/HPCCG-0.5.tar.gz, July 2007.Google Scholar
- W. Huang, J. Liu, B. Abali, and D. K. Panda. A case for high performance computing with virtual machines. In 20th Annual International Conference on Supercomputing (ICS), pages 125--134, 2006. Google Scholar
Digital Library
- Intel Corporation. Intel virtualization technology specification for the IA-32 Intel architecture, April 2005.Google Scholar
- Intel GmbH. Intel MPI benchmarks: Users guide and methodology description, 2004.Google Scholar
- L. Kaplan. Cray CNL. In FastOS PI Meeting and Workshop, June 2007. URL http://www.cs.unm.edu/fastos/07meeting/CNL_FASTOS.pdf.Google Scholar
- S. Kelly and R. Brightwell. Software architecture of the lightweight kernel, Catamount. In 2005 Cray Users' Group Annual Technical Conference. Cray Users' Group, May 2005.Google Scholar
- D. Kerbyson, H. Alme, A. Hoisie, F. Petrini, H. Wasserman, and M. Gittings. Predictive performance and scalability modeling of a large-scale application. In Proceedings of ACM/IEEE Supercomputing, November 2001. Google Scholar
Digital Library
- J. Lange and P. Dinda. SymCall: Symbiotic virtualization through VMM-to-guest upcalls. In Proceedings of the 2011 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2011), Newport Beach, CA, March 2011. Google Scholar
Digital Library
- J. Lange, K. Pedretti, T. Hudson, P. Dinda, Z. Cui, L. Xia, P. Bridges, A. Gocke, S. Jaconette, M. Levenhagen, and R. Brightwell. Palacios and kitten: New high performance operating systems for scalable virtualized and native supercomputing. In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS 2010), April 2010.Google Scholar
Cross Ref
- J. Liu, W. Huang, B. Abali, and D. Panda. High Performance VMM-Bypass I/O in Virtual Machines. In Proceedings of the USENIX Annual Technical Conference, May 2006. Google Scholar
Digital Library
- P. Luszczek, J. Dongarra, and J. Kepner. Design and implementation of the HPCC benchmark suite. CT Watch Quarterly, 2 (4A), Nov. 2006.Google Scholar
- M. F. Mergen, V. Uhlig, O. Krieger, and J. Xenidis. Virtualization for high-performance computing. Operating Systems Review, 40 (2): 8--11, 2006. Google Scholar
Digital Library
- H. Nishimura, N. Maruyama, and S. Matsuoka. Virtual clusters on the fly - fast, scalable, and flexible installation. In 7th IEEE International Symposium on Cluster Computing and the Grid (CCGRID), pages 549--556, 2007. Google Scholar
Digital Library
- F. Petrini, D. Kerbyson, and S. Pakin. The case of the missing supercomputer performance: Achieving optimal performance on the 8,192 processors of ASCI Q. In Proceedings of SC'03, 2003. Google Scholar
Digital Library
- H. Raj and K. Schwan. High performance and scalable I/O virtualization via self-virtualized devices. In 16th IEEE International Symposium on High Performance Distributed Computing, July 2007. Google Scholar
Digital Library
- S. Song, R. Ge, X. Feng, and K. W. Cameron. Energy profiling and analysis of the HPC Challenge benchmarks. International Journal of High Performance Computing Applications, Vol. 23, No. 3: 265--276, 2009. Google Scholar
Digital Library
- Top500. Top 500 Supercomputing Sites. URL http://www.top500.org/.Google Scholar
- D. Williams, P. Reynolds, K. Walsh, E. G. Sirer, and F. B. Schneider. Device driver safety through a reference validation mechanism. In Proceedings of the 8th Symposium on Operating Systems Design and Implementation (OSDI'08), 2008. Google Scholar
Digital Library
Index Terms
Minimal-overhead virtualization of a large scale supercomputer
Recommendations
Minimal-overhead virtualization of a large scale supercomputer
VEE '11: Proceedings of the 7th ACM SIGPLAN/SIGOPS international conference on Virtual execution environmentsVirtualization has the potential to dramatically increase the usability and reliability of high performance computing (HPC) systems. However, this potential will remain unrealized unless overheads can be minimized. This is particularly challenging on ...
Virtualizing HPC applications using modern hypervisors
FederatedClouds '12: Proceedings of the 2012 workshop on Cloud services, federation, and the 8th open cirrus summitIn this paper we explore the prospects of virtualization technologies being applied to high performance computing tasks. We use an extensive set of HPC benchmarks to evaluate virtualization overhead, including HPC Challenge, NAS Parallel Benchmarks and ...
A case for dual stack virtualization: consolidating HPC and commodity applications in the cloud
SoCC '12: Proceedings of the Third ACM Symposium on Cloud ComputingWith the growth of Infrastructure as a Service (IaaS) cloud providers, many have begun to seriously consider cloud services as a substrate for HPC applications. While the cloud promises many benefits for the HPC community, it currently does not come ...







Comments