skip to main content
tutorial

Designing Locality and NUMA Aware MPI Runtime for Nested Virtualization based HPC Cloud with SR-IOV Enabled InfiniBand

Authors Info & Claims
Published:08 April 2017Publication History
Skip Abstract Section

Abstract

Hypervisor-based virtualization solutions reveal good security and isolation, while container-based solutions make applications and workloads more portable and distributed in an effective, standardized and repeatable way. Therefore, nested virtualization based computing environments (e.g., container over virtual machine), which inherit the capabilities from both solutions, are becoming more and more attractive in clouds (e.g., running Docker over Amazon EC2 VMs). Recent studies have shown that running applications in either VMs or containers still has significant overhead, especially for I/O intensive workloads. This motivates us to investigate whether the nested virtualization based solution can be adopted to build high-performance computing (HPC) clouds for running MPI applications efficiently and where the bottlenecks lie. To eliminate performance bottlenecks, we propose a high-performance two-layer locality and NUMA aware MPI library, which is able to dynamically detect co-resident containers inside one VM as well as detect co-resident VM inside one host at MPI runtime. Thus the MPI processes across different containers and VMs can communicate to each other by shared memory or Cross Memory Attach (CMA) channels instead of network channel if they are co-resident. We further propose an enhanced NUMA aware hybrid design to utilize InfiniBand loopback based channel to optimize large message transfer across containers when they are running on different sockets. Performance evaluations show that compared with the performance of the state-of-art (1Layer) design, our proposed enhance-hybrid design can bring up to 184%, 81% and 12% benefit on point-to-point, collective operations, and end applications. Compared with the default performance, our enhanced-hybrid design delivers up to 184%, 85% and 16% performance improvement.

References

  1. Amazon EC2. http://aws.amazon.com/ec2/.Google ScholarGoogle Scholar
  2. M. Ben-Yehuda, M. D. Day, Z. Dubitzky, M. Factor, N. Har'El, A. Gordon, A. Liguori, O. Wasserman, and B.-A. Yassour. The turtles project: Design and implementation of nested virtualization. In OSDI, volume 10, pages 423--436, 2010.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Chameleon Cloud. https://www.chameleoncloud.org/.Google ScholarGoogle Scholar
  4. Cross Memory Attach (CMA). http://kernelnewbies.org/Linuxi_3.2.Google ScholarGoogle Scholar
  5. Docker. https://www.docker.com/.Google ScholarGoogle Scholar
  6. Y. Dong, X. Yang, J. Li, G. Liao, K. Tian, and H. Guan. High Performance Network Virtualization with SR-IOV. Journal of Parallel and Distributed Computing, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Performance Comparison of Virtual Machines and Linux Containers. Technical Report RC25482 (AUS1407-001), 2014.Google ScholarGoogle Scholar
  8. Google Compute Engine (GCE). https://cloud.google.com/compute/.Google ScholarGoogle Scholar
  9. W. Huang, J. Liu, B. Abali, and D. K. Panda. A Case for High Performance Computing with Virtual Machines. In Proceedings of the 20th Annual International Conference on Supercomputing, ICS '06, New York, NY, USA, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. W. Huang, M. J. Koop, Q. Gao, and D. K. Panda. Virtual Machine Aware Communication Libraries for High Performance Computing. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (SC), Reno, USA, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. W. Huang, J. Liu, M. Koop, B. Abali, and D. Panda. Nomad: Migrating OS-bypass Networks in Virtual Machines. In Proceedings of the 3rd International Conference on Virtual Execution Environments, VEE '07, New York, NY, USA, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Z. Huang, R. Ma, J. Li, Z. Chang, and H. Guan. Adaptive and Scalable Optimizations for High Performance SR-IOV. In Proceeding of 2012 IEEE International Conference Cluster Computing (CLUSTER), pages 459--467. IEEE, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J. Zhang, X. Lu, J. Jose, M. Li, R. Shi, D. K. Panda. High Performance MPI Library over SR-IOV Enabled InfiniBand Clusters. In Proceedings of International Conference on High Performance Computing (HiPC), Goa, India, December 17-20 2014. Google ScholarGoogle ScholarCross RefCross Ref
  14. J. Zhang, X. Lu, J. Jose, R. Shi, D. K. Panda. Can Inter-VM Shmem Benefit MPI Applications on SR-IOV based Virtualized InfiniBand Clusters? In Proceedings of 20th International Conference Euro-Par 2014 Parallel Processing, Porto, Portugal, August 25-29 2014. Google ScholarGoogle ScholarCross RefCross Ref
  15. Kernel-based Virtual Machine (KVM). http://www.linux-kvm.org/page/Main_Page.Google ScholarGoogle Scholar
  16. K. Kim, C. Kim, S.-I. Jung, H.-S. Shin, and J.-S. Kim. Inter-domain Socket Communications Supporting High Performance and Full Binary Compatibility on Xen. In Proceedings of the 4th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), Seattle, USA, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Linux Containers. https://linuxcontainers.org.Google ScholarGoogle Scholar
  18. Linux VServer. http://linux-vserver.org.Google ScholarGoogle Scholar
  19. J. Liu. Evaluating Standard-Based Self-Virtualizing Devices: A Performance Study on 10 GbE NICs with SR-IOV Support. In Proceeding of 2010 IEEE International Symposium Parallel & Distributed Processing (IPDPS), pages 1--12. IEEE, 2010. Google ScholarGoogle ScholarCross RefCross Ref
  20. J. Liu, W. Huang, B. Abali, and D. K. Panda. High Performance VMM-bypass I/O in Virtual Machines. In Proceedings of the Annual Conference on USENIX '06 Annual Technical Conference, ATC '06, Berkeley, CA, USA, 2006.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. A. C. Macdonell. Shared-Memory Optimizations for Virtual Machines. PhD Thesis. University of Alberta, Edmonton, Alberta, Fall 2011.Google ScholarGoogle Scholar
  22. Microsft. Nested virtualization. https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting.Google ScholarGoogle Scholar
  23. MVAPICH2-Virt. http://mvapich.cse.ohio-state.edu/.Google ScholarGoogle Scholar
  24. Oracle. Nested virtualization: Achieving up to 2x better aws performance! https://www.ravellosystems.com/blog/nested-virtualization-achieving-up-to-2x-better-aws-performance/.Google ScholarGoogle Scholar
  25. OSU Micro-benchmarks. http://mvapich.cse.ohio-state.edu/benchmarks/.Google ScholarGoogle Scholar
  26. Photon OS. https://vmware.github.io/photon/.Google ScholarGoogle Scholar
  27. S. Soltesz, H. Pötzl, M. E. Fiuczynski, A. Bavier, and L. Peterson. Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors. In Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems (EuroSys '07), Lisbon, Portugal, 2007.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. VMware ESX/ESXi. https://www.vmware.com/products/esxiand-esx/overview.Google ScholarGoogle Scholar
  29. VMware vCloud Air. http://vcloud.vmware.com/.Google ScholarGoogle Scholar
  30. J. Wang, K.-L. Wright, and K. Gopalan. XenLoop: A Transparent High Performance Inter-vm Network Loopback. In Proceedings of the 17th International Symposium on High Performance Distributed Computing (HPDC), Boston, USA, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. M. Xavier, M. Neves, F. Rossi, T. Ferreto, T. Lange, and C. De Rose. Performance Evaluation of Container-Based Virtualization for High Performance Computing Environments. In Parallel, Distributed and Network-Based Processing (PDP), 2013 21st Euromicro International Conference on, pages 233--240, Belfast, Northern Ireland, Feb 2013. doi: 10.1109/PDP.2013.41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Xen. http://www.xen.org/.Google ScholarGoogle Scholar
  33. J. Zhang, X. Lu, and D. K. Panda. High Performance MPI Library for Container-Based HPC Cloud on InfiniBand Clusters. In 2016 45th International Conference on Parallel Processing (ICPP), Aug 2016. Google ScholarGoogle ScholarCross RefCross Ref
  34. J. Zhang, X. Lu, and D. K. Panda. Performance Characterization of Hypervisor-and Container-Based Virtualization for HPC on SR-IOV Enabled InfiniBand Clusters. In 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), May 2016. Google ScholarGoogle ScholarCross RefCross Ref
  35. X. Zhang, S. McIntosh, P. Rohatgi, and J. L. Griffin. XenSocket: A High-throughput Interdomain Transport for Virtual Machines. In Proceedings of the ACM/IFIP/USENIX 2007 International Conference on Middleware (Middleware), Newport Beach, USA, 2007. Google ScholarGoogle ScholarCross RefCross Ref
  36. Y. Zhou, B. Subramaniam, K. Keahey, and J. Lange. Comparison of Virtualization and Containerization Techniques for High Performance Computing. In Proceedings of the 2015 ACM/IEEE conference on Supercomputing, Austin, USA, Nov 2015.Google ScholarGoogle Scholar

Index Terms

  1. Designing Locality and NUMA Aware MPI Runtime for Nested Virtualization based HPC Cloud with SR-IOV Enabled InfiniBand

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 52, Issue 7
        VEE '17
        July 2017
        256 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/3140607
        Issue’s Table of Contents
        • cover image ACM Conferences
          VEE '17: Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments
          April 2017
          261 pages
          ISBN:9781450349482
          DOI:10.1145/3050748

        Copyright © 2017 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 8 April 2017

        Check for updates

        Qualifiers

        • tutorial
        • Research
        • Refereed limited

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!