research-article

Public Access

Parsl: Pervasive Parallel Programming in Python

Authors:

Daniel S. Katz,

Lukasz Lacinski,

Justin M. Wozniak,

Kyle ChardAuthors Info & Claims

HPDC '19: Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

Pages 25 - 36

https://doi.org/10.1145/3307681.3325400

Published: 17 June 2019 Publication History

Abstract

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.

References

[1]

Common Workflow Language Specifications, v1.0.2. https://www.commonwl. org/v1.0/. Accessed Apr 24, 2019.

[2]

Computational Data Analysis Workflow Systems. https://s.apache.org/ existing-workflow-systems. Accessed Apr 24, 2019.

[3]

Dask. http://docs.dask.org/en/latest/. Accessed Apr 24, 2019.

[4]

Luigi. https://github.com/spotify/luigi. Accessed Apr 24, 2019.

[5]

Pickle. https://docs.python.org/3/library/pickle.html. Accessed Apr 24, 2019.

[6]

Airflow. https://airflow.apache.org/. Accessed Apr 24, 2019.

[7]

Apache Hadoop. https://hadoop.apache.org/. Accessed Apr 24, 2019.

[8]

Apache Spark. https://spark.apache.org/. Accessed Apr 24, 2019.

[9]

Dask distributed. http://distributed.dask.org/en/latest/. Accessed Apr 24, 2019.

[10]

IPython.parallel. https://github.com/ipython/ipyparallel. Accessed Apr 24, 2019.

[11]

Midway at University of Chicago Research Computing Center. https://rcc. uchicago.edu/docs/using-midway/index.html. Accessed Apr 24, 2019.

[12]

M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, et al. 2016. TensorFlow: A system for large-scale machine learning. In 12th USENIX Conf. on Operating Systems Design and Implementation. 265--283.

[13]

R. Anathankrishnan, K. Chard, I. Foster, M. Lidman, B. McCollam, S. Rosen, and S. Tuecke. 2016. Globus Auth: A research identity and access management platform. In 16th Intl Conf. on e-Science. 203--212.

[14]

T. Armstrong. 2011. Integrating task parallelism into the Python programming language. Master's thesis. University of Chicago.

[15]

T. G. Armstrong, Z. Zhang, D. S. Katz, M. Wilde, and I. T. Foster. 2010. Scheduling many-task workloads on supercomputers: Dealing with trailing tasks. In IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS). 1--10.

[16]

Y. Babuji, K. Chard, I. Foster, D. S. Katz, M. Wilde, A. Woodard, and J. Wozniak. 2018. Parsl: Scalable Parallel Scripting in Python. In 10th International Workshop on Science Gateways.

[17]

B. Blaiszik, K. Chard, J. Pruyne, R. Ananthakrishnan, S. Tuecke, and I. Foster. 2016. The Materials Data Facility: Data services to advance materials science research. JOM 68, 8 (2016), 2045--2052.

[18]

B. Bode, M. Butler, T. Dunning, T. Hoefler,W. Kramer,W. Gropp, andW.-m. Hwu. 2013. The Blue Waters super-system for super-science. In Contemporary High Performance Computing. Chapman and Hall/CRC, 339--366.

[19]

K. M. Chandy and C. Kesselman. 1993. Compositional C++: Compositional parallel programming. In Languages & Compilers for Parallel Computing. Springer, 124--144.

Digital Library

[20]

K. Chard, S. Tuecke, and I. Foster. 2014. Efficient and secure transfer, synchronization, and sharing of big data. IEEE Cloud Computing 1, 3 (2014), 46--55.

[21]

R. Chard, Z. Li, K. Chard, L. T. Ward, Y. N. Babuji, A. Woodard, S. Tuecke, B. Blaiszik, M. J. Franklin, and I. T. Foster. 2019. DLHub: Model and data serving for science. In 33rd IEEE International Parallel and Distributed Processing Symposium.

[22]

R. Chard, R. Vescovi, M. Du, H. Li, K. Chard, S. Tuecke, N. Kasthuri, and I. Foster. 2018. High-throughput neuroanatomy and trigger-action programming: a case study in research automation. In 1st Intl.Work. on Autonomous Infra. for Sci. ACM, 1:1--1:7.

Digital Library

[23]

L. Dalcín, R. Paz, and M. Storti. 2005. MPI for Python. J. Parallel and Distrib. Comput. 65, 9 (2005), 1108--1115.

Digital Library

[24]

E. Deelman, K. Vahi, G. Juve, M. Rynge, et al. 2015. Pegasus, a workflow management system for science automation. Future Gen. Comp. Sys. 46 (2015), 17--35.

Digital Library

[25]

P. Di Tommaso, M. Chatzou, E. W. Floden, P. P. Barja, E. Palumbo, and C. Notredame. 2017. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 4 (2017), 316.

[26]

M. Dorier, J. M.Wozniak, and R. Ross. 2017. Supporting task-level fault-tolerance in HPC workflows by launching MPI jobs inside MPI jobs. In 12th Workshop on Workflows in Support of Large-Scale Science. 5:1--5:11.

Digital Library

[27]

I. Foster, R. Olson, and S. Tuecke. 1992. Productive parallel programming: The PCN approach. Scientific Programming 1, 1 (1992), 51--66.

Digital Library

[28]

I. Foster and S. Taylor. 1990. Strand: New concepts in parallel programming. Prentice Hall (1990).

Digital Library

[29]

GFD-R-P.231 2016. Distributed Resource Management Application API Version 2.2 (DRMAA). Recommendation. Open Grid Forum.

[30]

GFD-R-P.90 2013. A Simple API for Grid Applications (SAGA). Specification. Open Grid Forum.

[31]

J. Goecks, A. Nekrutenko, and J. Taylor. 2010. Galaxy: A comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biology 11, 8 (2010), R86.

[32]

P. Hintjens. 2013. ZeroMQ: Messaging for Many Applications. O'Reilly.

[33]

A. Jain, S. P. Ong,W. Chen, B. Medasani, X. Qu, et al. 2015. FireWorks: A dynamic workflow system designed for high-throughput applications. Concurrency and Computation: Practice and Experience 27, 17 (2015), 5037--5059.

Digital Library

[34]

M. M. McKerns, L. Strand, T. Sullivan, A. Fang, and M. A. Aivazis. 2012. Building a framework for predictive science. arXiv preprint arXiv:1202.1056 (2012).

[35]

P. Moritz, R. Nishihara, S.Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, et al. 2018. Ray: A distributed framework for emerging AI applications. In 13th USENIX Conf. on Operating Systems Design and Implementation. 561--577.

Digital Library

[36]

R. S. Nikhil. 1993. An overview of the parallel language Id (a foundation for pH, a parallel dialect of Haskell). Technical Report. Digital Equipment Corporation, Cambridge Research Laboratory.

[37]

J. J. Pitt. 2017. Deciphering Cancer Development and Progression through Large- Scale Computational Analyses of Germline and Somatic Genomes. Ph.D. Dissertation. University of Chicago.

[38]

E. Tejedor, Y. Becerra, G. Alomar, A. Queralt, R. M. Badia, J. Torres, T. Cortes, and J. Labarta. 2017. PyCOMPSs: Parallel computational workflows in Python. Intl Journal of High Performance Computing Applications 31, 1 (2017), 66--82.

Digital Library

[39]

M. Wilde, M. Hategan, J.Wozniak, B. Clifford, D. Katz, and I. Foster. 2011. Swift: A language for distributed parallel scripting. Parallel Comput. 37, 9 (2011), 633--652.

Digital Library

Cited By

Callaghan SMaechling PSilva FSu MMilner KGraves ROlsen KCui YVahi KKottke AGoulet CDeelman EJordan TBen-Zion Y(2024)Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard modelsFrontiers in High Performance Computing10.3389/fhpcp.2024.13607202Online publication date: 1-May-2024
https://doi.org/10.3389/fhpcp.2024.1360720
Rosen AGallant MGeorge JRiebesell JSahasrabuddhe HShen JWen MEvans MPetretto GWaroquiers DRignanese GPersson KJain AGanose A(2024)Jobflow: Computational Workflows Made SimpleJournal of Open Source Software10.21105/joss.059959:93(5995)Online publication date: Jan-2024
https://doi.org/10.21105/joss.05995
Ward LPauloski JHayot-Sasson VBabuji YBrace AChard RChard KThakur RFoster I(2024)Employing artificial intelligence to steer exascale workflows with colmenaThe International Journal of High Performance Computing Applications10.1177/10943420241288242Online publication date: 8-Oct-2024
https://doi.org/10.1177/10943420241288242
Show More Cited By

Index Terms

Parsl: Pervasive Parallel Programming in Python
1. Computing methodologies
  1. Distributed computing methodologies
  2. Parallel computing methodologies
2. Software and its engineering
  1. Software notations and tools
    1. Software libraries and repositories

Recommendations

Parallel Scripting with Python

The combination of the Python language and the BSP parallel computing model make developing and testing parallel programs a much more pleasurable experience.
Web development with python and django (abstract only)
SIGCSE '12: Proceedings of the 43rd ACM technical symposium on Computer Science Education

Many instructors have already discovered the joy of teaching programming using the Python programming language. Now it's time to take Python to the next level. This workshop will introduce Django, an open source Python web framework that saves you time ...
PGAS for Distributed Numerical Python Targeting Multi-core Clusters
IPDPS '12: Proceedings of the 2012 IEEE 26th International Parallel and Distributed Processing Symposium

In this paper we propose a parallel programming model that combines two well-known execution models: Single Instruction, Multiple Data (SIMD) and Single Program, Multiple Data (SPMD). The combined model supports SIMD-style data parallelism in global ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HPDC '19: Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing

June 2019

278 pages

ISBN:9781450366700

DOI:10.1145/3307681

General Chair:
Jon Weissman
University of Minnesota, USA
,
Program Chairs:
Ali R. Butt
Virginia Tech, USA
,
Evgenia Smirni
College of William and Mary, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

University of Arizona: University of Arizona
SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing
SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

HPDC '19

Sponsor:

University of Arizona
SIGHPC
SIGARCH

HPDC '19: The 28th International Symposium on High-Performance Parallel and Distributed Computing

June 22 - 29, 2019

AZ, Phoenix, USA

Acceptance Rates

HPDC '19 Paper Acceptance Rate 22 of 106 submissions, 21%;

Overall Acceptance Rate 166 of 966 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

170
Total Citations
View Citations
2,798
Total Downloads

Downloads (Last 12 months)562
Downloads (Last 6 weeks)66

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Callaghan SMaechling PSilva FSu MMilner KGraves ROlsen KCui YVahi KKottke AGoulet CDeelman EJordan TBen-Zion Y(2024)Using open-science workflow tools to produce SCEC CyberShake physics-based probabilistic seismic hazard modelsFrontiers in High Performance Computing10.3389/fhpcp.2024.13607202Online publication date: 1-May-2024
https://doi.org/10.3389/fhpcp.2024.1360720
Rosen AGallant MGeorge JRiebesell JSahasrabuddhe HShen JWen MEvans MPetretto GWaroquiers DRignanese GPersson KJain AGanose A(2024)Jobflow: Computational Workflows Made SimpleJournal of Open Source Software10.21105/joss.059959:93(5995)Online publication date: Jan-2024
https://doi.org/10.21105/joss.05995
Ward LPauloski JHayot-Sasson VBabuji YBrace AChard RChard KThakur RFoster I(2024)Employing artificial intelligence to steer exascale workflows with colmenaThe International Journal of High Performance Computing Applications10.1177/10943420241288242Online publication date: 8-Oct-2024
https://doi.org/10.1177/10943420241288242
Yokelson DTitov MRamesh SKilic OTurilli MJha SMalony A(2024)Enabling Performance Observability for Heterogeneous HPC Workflows with SOMAProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673100(220-230)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673100
Ananthakrishnan RBabuji YBaughman MBryan JChard KChard RClifford BFoster IKatz DHunter Kesling KJanidlo CMello RWang L(2024)Enabling Remote Management of FaaS Endpoints with Globus Compute Multi-User EndpointsPractice and Experience in Advanced Research Computing 2024: Human Powered Computing10.1145/3626203.3670612(1-5)Online publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1145/3626203.3670612
Phung TThomas CWard LChard KThain DMencagli GDazzi PLowenthal DBadia R(2024)Accelerating Function-Centric Applications by Discovering, Distributing, and Retaining Reusable Context in Workflow SystemsProceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing10.1145/3625549.3658663(122-134)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3625549.3658663
Pauloski JHayot-Sasson VGonthier MHudson NPan HZhou SFoster IChard K(2024)TaPS: A Performance Evaluation Suite for Task-based Execution Frameworks2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678702(1-10)Online publication date: 16-Sep-2024
https://doi.org/10.1109/e-Science62913.2024.10678702
Nicolae BWozniak JBicer TNguyen HPatel PPan HGueroudji AGonthier MHayot-Sasson VHuerta EChard KChard RDorier MRao NAl-Najjar ACorsi AFoster I(2024)Diaspora: Resilience-Enabling Services for Real-Time Distributed Workflows2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678669(1-9)Online publication date: 16-Sep-2024
https://doi.org/10.1109/e-Science62913.2024.10678669
Fink ZParasyris KRathi PGeorgakoudis GMenon HBremer P(2024)HPAC-ML: A Programming Model for Embedding ML Surrogates in Scientific ApplicationsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00078(1-16)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00078
Sly-Delgado BTovar BZhou JThain D(2024)Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVineProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00068(1-13)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00068
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents