skip to main content
research-article

Extended Abstract: Productive Parallel Programming with Parsl

Published:27 April 2021Publication History
Skip Abstract Section

Abstract

Parsl is a parallel programming library for Python that aims to make it easy to specify parallelism in programs and to realize that parallelism on arbitrary parallel and distributed computing systems. Parsl relies on developers annotating Python functions-wrapping either Python or external applications-to indicate that these functions may be executed concurrently. Developers can then link together functions via the exchange of data. Parsl establishes a dynamic dependency graph and sends tasks for execution on connected resources when dependencies are resolved. Parsl's runtime system enables different compute resources to be used, from laptops to supercomputers, without modification to the Parsl program.

References

  1. Airflow Project. 2019. Airflow. https://airflow.apache.org/. Accessed Sep 1, 2020.Google ScholarGoogle Scholar
  2. Y. Babuji, K. Chard, I. Foster, D.S. Katz, M. Wilde, A. Woodard, and J. Wozniak. 2018. Parsl: Scalable Parallel Scripting in Python. In 10th International Workshop on Science Gateways (IWSG).Google ScholarGoogle Scholar
  3. Y. Babuji, A. Woodard, Z. Li, D.S. Katz, B. Clifford, R. Kumar, L. Lacinski, R. Chard, J. Wozniak, I. Foster, M. Wilde, and K. Chard. 2019. Parsl: Pervasive Parallel Programming in Python. In 28th ACM International Symposium on High- Performance Parallel and Distributed Computing (HPDC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Yadu Babuji, AnnaWoodard, Zhuozhao Li, Daniel S. Katz, Ben Clifford, Ian Foster, Michael Wilde, and Kyle Chard. 2019. Scalable Parallel Programming in Python with Parsl. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning) (Chicago, IL, USA) (PEARC '19). ACM, Article 22, 8 pages. https://doi.org/10.1145/3332186.3332231 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. M.G. Burke, K. Knobe, R. Newton, and V. Sarkar. 2011. Concurrent Collections Programming Model. Springer US, Boston, MA, 364--371.Google ScholarGoogle Scholar
  6. R. Chard, Y. Babuji, Z. Li, T. Skluzacek, A. Woodard, B. Blaiszik, I. Foster, and K. Chard. 2020. funcX: A Federated Function Serving Fabric for Science. Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. https://doi.org/10.1145/3369583.3392683 Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. L. Dagum and R. Menon. 1998. OpenMP: an industry standard API for sharedmemory programming. Computational Science & Engineering, IEEE 5, 1 (1998), 46--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Dask Development Team. 2016. Dask: Library for dynamic task scheduling. https: //dask.orgGoogle ScholarGoogle Scholar
  9. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P.J. Maechling, R. Mayani, W. Chen, R.F. da Silva, et al. 2015. Pegasus, a workflow management system for science automation. Future Generation Computer Systems 46 (2015), 17--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. P. Di Tommaso, M. Chatzou, E.W. Floden, P.P. Barja, E. Palumbo, and C. Notredame. 2017. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 4 (2017), 316.Google ScholarGoogle ScholarCross RefCross Ref
  11. A. Jain, S.P. Ong, W. Chen, B. Medasani, X. Qu, M. Kocher, M. Brafman, Gu. Petretto, G. Rignanese, et al. 2015. FireWorks: A dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience 27, 17 (2015), 5037--5059. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Luigi Team. [n.d.]. Luigi. https://github.com/spotify/luigi. Accessed Sep 1, 2020.Google ScholarGoogle Scholar
  13. M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster. 2011. Swift: A language for distributed parallel scripting. Parallel Comput. 37, 9 (2011), 633--652. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

(auto-classified)
  1. Extended Abstract: Productive Parallel Programming with Parsl

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGAda Ada Letters
          ACM SIGAda Ada Letters  Volume 40, Issue 2
          December 2020
          55 pages
          ISSN:1094-3641
          DOI:10.1145/3463478
          • Editor:
          • Alok Srivastava
          Issue’s Table of Contents

          Copyright © 2021 Copyright is held by the owner/author(s)

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 27 April 2021

          Check for updates

          Qualifiers

          • research-article
        • Article Metrics

          • Downloads (Last 12 months)13
          • Downloads (Last 6 weeks)0

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!