skip to main content
research-article

SPIRIT: a runtime system for distributed irregular tree applications

Published:27 February 2016Publication History
Skip Abstract Section

Abstract

Repeated, depth-first traversal of trees is a common algorithmic pattern in an important set of applications from diverse domains such as cosmological simulations, data mining, and computer graphics. As these applications operate over massive data sets, it is often necessary to distribute the trees to process all of the data.

In this work, we introduce SPIRIT, a runtime system to ease the writing of distributed tree applications. SPIRIT automates the challenging tasks of tree distribution, optimizing communication and parallelizing independent computations. The common algorithmic pattern in tree traversals is exploited to effectively schedule parallel computations and improve locality. As a result, pipeline parallelism in distributed traversals is identified, which is complemented by load-balancing, and locality-enhancing, message aggregation optimizations. Evaluation of SPIRIT on tree traversal in Point Correlation (PC) shows a scalable system, achieving speedups upto 38x on a 16-node, 64 process system compared to a 1-node, baseline configuration. We also find that SPIRIT results in substantially less communication and achieves significant performance improvements over implementations in other distributed graph systems.

References

  1. J. Chhugani, C. Kim, H. Shukla, J. Park, P. Dubey, J. Shalf, and H. D. Simon. Billion-particle simd-friendly two-point correlation on large-scale hpc cluster systems. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, page 1, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. D. Gregor and A. Lumsdaine. The parallel bgl: A generic library for distributed graph computations. Parallel Object-Oriented Scientific Computing (POOSC), 2:1--18, 2005.Google ScholarGoogle Scholar
  3. Y. Jo and M. Kulkarni. Automatically enhancing locality for tree traversals with traversal splicing. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications, OOPSLA '12, pages 355--374, New York, NY, USA, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. M. Kulkarni. The Galois System: optimistic parallelization of irregular programs. PhD thesis, Ithaca, NY, USA, 2008. Adviser-Pingali, Keshav. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed graphlab: A framework for machine learning and data mining in the cloud. Proc. VLDB Endow., 5(8):716--727, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pages 135--146, New York, NY, USA, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. J. H. Reif. Depth-first search is inherently sequential. Information Processing Letters, 20(5):229--234, 1985.Google ScholarGoogle Scholar

Index Terms

  1. SPIRIT: a runtime system for distributed irregular tree applications

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGPLAN Notices
          ACM SIGPLAN Notices  Volume 51, Issue 8
          PPoPP '16
          August 2016
          405 pages
          ISSN:0362-1340
          EISSN:1558-1160
          DOI:10.1145/3016078
          Issue’s Table of Contents
          • cover image ACM Conferences
            PPoPP '16: Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming
            February 2016
            420 pages
            ISBN:9781450340922
            DOI:10.1145/2851141

          Copyright © 2016 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 27 February 2016

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!