extended-abstract

HyperShell v2: Distributed Task Execution for HPC

Authors:

Geoffrey Lentner,

Lev GorensteinAuthors Info & Claims

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

Article No.: 80, Pages 1 - 3

https://doi.org/10.1145/3491418.3535138

Published: 08 July 2022 Publication History

Get Access

Abstract

HyperShell is an elegant, cross-platform, high-performance computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios. There are several existing tools that serve a similar purpose, but lack some aspect that HyperShell provides (e.g., distributed, detailed logging, automated retries, super scale). Novel aspects of HyperShell include but are not limited to (1) cross-platform, (2) client-server design, (3) staggered launch for large scales, (4) persistent hosting of the server, and optionally (5) a database in-the-loop for restarts and persisting task metadata. HyperShell was originally created to support researchers at Purdue University, out of a specific unmet need. It has been in use for several years now. With this next release, we’ve completely re-implemented HyperShell as both an application and a library to provide new features, scalability, flexibility, robustness, and wider support. (https://github.com/glentner/hyper-shell)

References

[1]

GNU Software Foundation. 2022. GNU Make. https://www.gnu.org/software/make/

Google Scholar

[2]

NERSC. 2022. TaskFarmer. https://docs.nersc.gov/jobs/workflow/taskfarmer/ TaskFarmer is a workflow manager developed in-house at NERSC to coordinate single or multicore tasks.

Google Scholar

[3]

SchedMD. 2022. Slurm Workload Manager. https://slurm.schedmd.com

Google Scholar

[4]

Carol Song, Preston Smith, Xiao Zhu, and Rajesh Kalyanam. 2020. NSF Award 2005632 - Category I: Anvil - A National Composable Advanced Computational Resource for the Future of Science and Engineering. (2020). https://www.nsf.gov/awardsearch/showAward?AWD_ID=2005632

Google Scholar

[5]

Ole Tange. 2021. GNU Parallel 20220322. https://doi.org/10.5281/zenodo.6377950 GNU Parallel is a general parallelizer to run multiple serial command line programs in parallel without changing them.

Crossref

Google Scholar

[6]

Lucas Wilson, John Fonner, Oscar Esteban, Jason Allison, Marshall Lerner, and Harry Kenya. 2017. Launcher: A simple tool for executing high throughput computing workloads. Journal of Open Source Software (Aug. 2017). https://doi.org/

Crossref

Google Scholar

Cited By

View all

Abhinit IAdams EAlam KChase BDeelman EGorenstein LHudson SIslam TLarson JLentner GMandal ANavarro JNicolae BPouchard LRoss RRoy BRynge MSerebrenik AVahi KWild SXin Yda Silva RFilgueira R(2022)Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS56498.2022.00016(84-92)Online publication date: Nov-2022
https://doi.org/10.1109/WORKS56498.2022.00016

Recommendations

Distributed Computing Education, Part 4: Training Infrastructure

Teaching distributed computing requires environments that provide adequate training infrastructure, or t-infrastructure. In practice, t-infrastructure includes the computing equipment, digital communications, software, data, and support staff necessary ...
Experiences with resource provisioning for scientific workflows using Corral

The development of grid and workflow technologies has enabled complex, loosely coupled scientific applications to be executed on distributed resources. Many of these applications consist of large numbers of short-duration tasks whose runtimes are ...
Middleware support for many-task computing

Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Many-task computing denotes high-performance computations comprising multiple distinct activities, coupled via file ...

Comments

Information & Contributors

Information

Published In

PEARC '22: Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You

July 2022

455 pages

ISBN:9781450391610

DOI:10.1145/3491418

Editors:
Julie Wernert
Indiana University
,
Alan Chalker
Ohio Supercomputer Center
,
Shava Smallen
San Diego Supercomputer Center, University of California, San Diego
,
Tabitha Samuel
University of Tennessee, Knoxville
,
JP Navarro
Argonne National Laboratory

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Check for updates

Badges

Best Poster

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

PEARC '22

Sponsor:

PEARC '22: Practice and Experience in Advanced Research Computing

July 10 - 14, 2022

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 133 of 202 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)3

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Abhinit IAdams EAlam KChase BDeelman EGorenstein LHudson SIslam TLarson JLentner GMandal ANavarro JNicolae BPouchard LRoss RRoy BRynge MSerebrenik AVahi KWild SXin Yda Silva RFilgueira R(2022)Novel Proposals for FAIR, Automated, Recommendable, and Robust Workflows2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS56498.2022.00016(84-92)Online publication date: Nov-2022
https://doi.org/10.1109/WORKS56498.2022.00016

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

Distributed Computing Education, Part 4: Training Infrastructure

Experiences with resource provisioning for scientific workflows using Corral

Middleware support for many-task computing

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Badges

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations