research-article

Public Access

An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers

Authors:

Sudharshan S. Vazhkudai,

Youngjae KimAuthors Info & Claims

HPCAsia '21: The International Conference on High Performance Computing in Asia-Pacific Region

Pages 11 - 22

https://doi.org/10.1145/3432261.3432263

Published: 20 January 2021 Publication History

All formats PDF

Abstract

Supercomputer design is a complex, multi-dimensional optimization process, wherein several subsystems need to be reconciled to meet a desired figure of merit performance for a portfolio of applications and a budget constraint. However, overall, the HPC community has been gravitating towards ever more Flops, at the expense of many other subsystems. To draw attention to overall system balance, in this paper, we analyze balance ratios and architectural trends in the world’s most powerful supercomputers. Specifically, we have collected the performance characteristics of systems between 1993 and 2019 based on the Top500 lists and then analyzed their architectures from diverse system design perspectives. Notably, our analysis studies the performance balance of the machines, across a variety of subsystems such as compute, memory, I/O, interconnect, intra-node connectivity and power. Our analysis reveals that balance ratios of the various subsystems need to be considered carefully alongside the application workload portfolio to provision the subsystem capacity and bandwidth specifications, which can help achieve optimal performance.

References

[1]

2020. 3D XPoint Technology - Micron Technology, Inc.https://www.micron.com/products/advanced-solutions/3d-xpoint-technology.

[2]

2020. 3D XPoint™: A Breakthrough in Non-Volatile Memory Technology. https://www.intel.com/content/www/us/en/architecture-and-technology/intel-micron-3d-xpoint-webcast.html.

[3]

2020. GPUDirect | NVIDIA Developer. https://developer.nvidia.com/gpudirect.

[4]

2020. Green500 | TOP500 Supercomputer Sites. https://www.top500.org/green500/.

[5]

2020. HPCG Benchmark. https://www.hpcg-benchmark.org.

[6]

2020. IO-500 [Virtual Institute for I/O]. https://www.vi4io.org/std/io500/start.

[7]

2020. Radeon Vega 20 Will Have XGMI. https://www.phoronix.com/scan.php?page=news_item&px=AMDGPU-XGMI-Vega20-Patches.

[8]

2020. Sierra. https://computing.llnl.gov/computers/sierra.

[9]

2020. Summit - Oak Ridge Leadership Computing Facility. https://www.olcf.ornl.gov/summit/.

[10]

2020. Titan - Cray XK7, Opteron 6274 16C 2.200GHz, Cray Gemini interconnect, NVIDIA K20x | TOP500 Supercomputer Sites. http://www.top500.org/system/177975.

[11]

2020. TOP500 Lists. http://www.top500.org/lists/.

[12]

2020. ZettaScaler - WikiChip. https://en.wikichip.org/wiki/zettascaler.

[13]

[13] ACM Gordon Bell Prize.2020. https://awards.acm.org/bell.

[14]

S. R. Alam, J. A. Kuehn, R. F. Barrett, J. M. Larkin, M. R. Fahey, R. Sankaran, and P. H. Worley. 2007. Cray XT4: An Early Evaluation for Petascale Scientific Simulation. In SC ’07: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing.

Digital Library

[15]

B. Austin, C. Daley, D. Doerfler, J. Deslippe, B. Cook, B. Friesen, T. Kurth, C. Yang, and N. J. Wright. 2018. A Metric for Evaluating Supercomputer Performance in the Era of Extreme Heterogeneity. In 2018 IEEE/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS).

[16]

Keren Bergman, Shekhar Borkar, Dan Campbell, William Carlson, William Dally, Monty Denneau, Paul Franzon, William Harrod, Kerry Hill, Jon Hiller, 2008. Exascale Computing Study: Technology Challenges in Achieving Exascale Systems. Defense Advanced Research Projects Agency Information Processing Techniques Office (DARPA IPTO), Tech. Rep 15 (2008).

[17]

A. Bhatele, N. Jain, Y. Livnat, V. Pascucci, and P. Bremer. 2016. Analyzing Network Health and Congestion in Dragonfly-Based Supercomputers. In 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS). 93–102.

[18]

Arthur S Bland, Jack C Wells, Otis E Messer, Oscar R Hernandez, and James H Rogers. 2012. Titan: Early Experience with the Cray XK6 at Oak Ridge National Laboratory. In Proceedings of cray user group conference (CUG 2012). Cray User Group Stuttgart, Germany.

[19]

D. Chen, P. Heidelberger, C. Stunkel, Y. Sugawara, C. Minkenberg, B. Prisacari, and G. Rodriguez. 2016. An Evaluation of Network Architectures for Next Generation Supercomputers. In 2016 7th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS). 11–21.

[20]

Neal E Davis, Robert W Robey, Charles R Ferenbaugh, David Nicholaeff, and Dennis P Trujillo. 2012. Paradigmatic Shifts for Exascale Supercomputing. The Journal of Supercomputing 62, 2 (2012).

Digital Library

[21]

Douglas W Doerfler, Mahesh Rajan, Marcus Epperson, Courtenay T Vaughan, Kevin Pedretti, Richard Frederick Barrett, and Brian Barrett. 2011. A Comparison of the Performance Characteristics of Capability and Capacity Class HPC Systems.Technical Report. Sandia National Lab., Albuquerque, NM.

[22]

Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. 2003. The LINPACK Benchmark: Past, Present and Future. Concurrency and Computation: practice and experience 15, 9(2003).

[23]

D. Foley and J. Danskin. 2017. Ultra-Performance Pascal GPU and NVLink Interconnect. IEEE Micro 37, 2 (Mar 2017).

Digital Library

[24]

P. M. Kogge and T. J. Dysart. 2011. Using the TOP500 to Trace and Project Technology and Architecture Trends. In SC ’11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis(SC ’11).

Digital Library

[25]

[25] Graph 500 | large-scale benchmarks.2020. https://graph500.org/.

[26]

M. Li, S. S. Vazhkudai, A. R. Butt, F. Meng, X. Ma, Y. Kim, C. Engelmann, and G. Shipman. 2010. Functional Partitioning to Optimize End-to-End Performance on Many-core Architectures. In SC ’10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis.

[27]

Ning Liu, Jason Cope, Philip Carns, Christopher Carothers, Robert Ross, Gary Grider, Adam Crume, and Carlos Maltzahn. 2012. On the role of burst buffers in leadership-class storage systems. In IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST).

[28]

John McCalpin. 1995. Memory bandwidth and machine balance in high performance computers. IEEE Technical Committee on Computer Architecture Newsletter (12 1995).

[29]

John McCalpin. 2006. STREAM: Sustainable Memory Bandwidth in High Performance Computers. http://www.cs.virginia.edu/stream/. (2006).

[30]

Hans Werner Meuer. 2008. The Top500 Project. Looking Back over 15 Years of Supercomputing Experience. PIK-Praxis der Informationsverarbeitung und Kommunikation 31, 2(2008).

[31]

Sarp Oral, Sudharshan S. Vazhkudai, Feiyi Wang, Christopher Zimmer, Christopher Brumgard, Jesse Hanley, George Markomanolis, Ross Miller, Dustin Leverman, Scott Atchley, and Veronica Vergara Larrea. 2019. End-to-End I/O Portfolio for the Summit Supercomputing Ecosystem. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Denver, Colorado) (SC ’19). Article 63, 14 pages.

Digital Library

[32]

Yoshio Oyanagi. 2002. Future of Supercomputing. Journal of Computational and Applied Mathematics 149, 1(2002).

[33]

R. R. Schaller. 1997. Moore’s law: past, present and future. IEEE Spectrum 34, 6 (June 1997).

Digital Library

[34]

Amar Shan. 2006. Heterogeneous Processing: a Strategy for Augmenting Moore’s Law. Linux Journal 2006, 142 (2006).

[35]

Dan Stanzione, John West, R. Todd Evans, Tommy Minyard, Omar Ghattas, and Dhabaleswar K. Panda. 2020. Frontera: The Evolution of Leadership Computing at the National Science Foundation. In Practice and Experience in Advanced Research Computing (Portland, OR, USA) (PEARC ’20). Association for Computing Machinery, New York, NY, USA, 106–111. https://doi.org/10.1145/3311790.3396656

Digital Library

[36]

Erich Strohmaier, Hans W Meuer, Jack Dongarra, and Horst D Simon. 2015. The Top500 List and Progress in High-Performance Computing. Computer 48, 11 (2015).

[37]

C. B. Stunkel, R. L. Graham, G. Shainer, M. Kagan, S. S. Sharkawi, B. Rosenburg, and G. A. Chochia. 2020. The high-speed networks of the Summit and Sierra supercomputers. IBM Journal of Research and Development 64, 3/4 (2020).

[38]

Sudharshan S. Vazhkudai and Bronis R. et. al. de Supinski. 2018. The Design, Deployment, and Evaluation of the CORAL Pre-exascale Systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis(SC ’18).

[39]

F. Wang, S. Oral, S. Sen, and N. Imam. 2019. Learning from Five-year Resource-Utilization Data of Titan System. In Workshop on Monitoring and Analysis for High Performance Computing Systems Plus Applications(HPCMASPA ’19).

[40]

Weixia Xu, Yutong Lu, Qiong Li, Enqiang Zhou, Zhenlong Song, Yong Dong, Wei Zhang, Dengping Wei, Xiaoming Zhang, Haitao Chen, Jianying Xing, and Yuan Yuan. 2014. Hybrid Hierarchy Storage System in MilkyWay-2 Supercomputer. Front. Comput. Sci. 8, 3 (June 2014).

Digital Library

Cited By

Perminov ETestov V(2024)Mathematisation of specialised disciplines as the basis for fundamentalising IT training in universitiesThe Education and science journal10.17853/1994-5639-2024-7-12-4326:7(12-43)Online publication date: 6-Sep-2024
https://doi.org/10.17853/1994-5639-2024-7-12-43
Byun HJamil SHan JPark SLee MKim CChoi BKim Y(2024)An Analytical Model-based Capacity Planning Approach for Building CSD-based Storage SystemsACM Transactions on Embedded Computing Systems10.1145/362367723:6(1-25)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3623677
Wei JHan XYu JJiang JLiu HLin PYu MXu KZhao LWang PZheng WXie JZhou YZhang TZhang FZhang YYu YWang YBai YLi CYu ZDeng HLi YChi X(2024)A Performance-Portable Kilometer-Scale Global Ocean Model on ORISE and New Sunway Heterogeneous SupercomputersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00009(1-12)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00009
Show More Cited By

Index Terms

An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers
1. Computing methodologies
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles

Index terms have been assigned to the content through auto-classification.

Recommendations

Sustainable HPC infrastructures - supercomputers, grids, and clouds: the DEISA experience
HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing
New capabilities in qoscosgrid middleware for advanced job management, advance reservation and co-allocation of computing resources --- quantum chemistry application use case
Building a National Distributed e-Infrastructure - PL-Grid

In this chapter we present the new capabilities of QosCosGrid (QCG) middleware for advanced job and resource management in the grid environment. By connecting many computing clusters together, QosCosGrid offers easy-to-use mapping, execution and ...
HPC Cluster Monitoring System Architecture Design and Implement
ICICTA '09: Proceedings of the 2009 Second International Conference on Intelligent Computation Technology and Automation - Volume 02

High performance computing (HPC) facilities such as HPC clusters, as building blocks of Grid computing, are playing an important role in computational Grid. HPC monitoring in HPC cluster systems presents an important challenge because HPC cluster ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

HPCAsia '21: The International Conference on High Performance Computing in Asia-Pacific Region

January 2021

143 pages

ISBN:9781450388429

DOI:10.1145/3432261

Copyright © 2021 ACM.

© 2021 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

U.S. Department of Energy

Conference

HPC Asia 2021

HPC Asia 2021: The International Conference on High Performance Computing in Asia-Pacific Region

January 20 - 22, 2021

Virtual Event, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 69 of 143 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,976
Total Downloads

Downloads (Last 12 months)1,078
Downloads (Last 6 weeks)145

Reflects downloads up to 23 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Perminov ETestov V(2024)Mathematisation of specialised disciplines as the basis for fundamentalising IT training in universitiesThe Education and science journal10.17853/1994-5639-2024-7-12-4326:7(12-43)Online publication date: 6-Sep-2024
https://doi.org/10.17853/1994-5639-2024-7-12-43
Byun HJamil SHan JPark SLee MKim CChoi BKim Y(2024)An Analytical Model-based Capacity Planning Approach for Building CSD-based Storage SystemsACM Transactions on Embedded Computing Systems10.1145/362367723:6(1-25)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3623677
Wei JHan XYu JJiang JLiu HLin PYu MXu KZhao LWang PZheng WXie JZhou YZhang TZhang FZhang YYu YWang YBai YLi CYu ZDeng HLi YChi X(2024)A Performance-Portable Kilometer-Scale Global Ocean Model on ORISE and New Sunway Heterogeneous SupercomputersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00009(1-12)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00009
Copik MChrapek MSchmid LCalotoiu AHoefler T(2024)Software Resource Disaggregation for HPC with Serverless Computing2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS57955.2024.00021(139-156)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPS57955.2024.00021
Benoit AHerault TPerotin LRobert YVivien F(2024)Revisiting I/O bandwidth-sharing strategies for HPC applicationsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2024.104863(104863)Online publication date: Feb-2024
https://doi.org/10.1016/j.jpdc.2024.104863
Fernandes LKharate PSingh B(2024)The Future of High Performance Computing in Biomimetics and Some ChallengesHigh Performance Computing in Biomimetics10.1007/978-981-97-1017-1_15(287-303)Online publication date: 21-Mar-2024
https://doi.org/10.1007/978-981-97-1017-1_15
Абрамов НАбрамов С(2023)June 2022 Top500 List OverviewProgram Systems: Theory and ApplicationsПрограммные системы: теория и приложения10.25209/2079-3316-2023-14-2-49-9314:2(49-93)Online publication date: 2023
https://doi.org/10.25209/2079-3316-2023-14-2-49-93
Valero-Lara PJorquera ILui FVetter J(2023)Mixed-Precision S/DGEMM Using the TF32 and TF64 Frameworks on Low-Precision AI Tensor CoresProceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis10.1145/3624062.3624084(179-186)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3624062.3624084
Lindquist NLuszczek PDongarra JGallivan KNikolopoulos DBeivide RGallopoulos E(2023)Using Additive Modifications in LU Factorization Instead of PivotingProceedings of the 37th International Conference on Supercomputing10.1145/3577193.3593731(14-24)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3577193.3593731
Papaphilippou PQue ZLuk W(2023)Efficiently Removing Sparsity for High-Throughput Stream Processing2023 International Conference on Field Programmable Technology (ICFPT)10.1109/ICFPT59805.2023.00033(244-249)Online publication date: 12-Dec-2023
https://doi.org/10.1109/ICFPT59805.2023.00033
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents