An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle

ABSTRACT

Mantle convection is the fundamental physical process within earth's interior responsible for the thermal and geological evolution of the planet, including plate tectonics. The mantle is modeled as a viscous, incompressible, non-Newtonian fluid. The wide range of spatial scales, extreme variability and anisotropy in material properties, and severely nonlinear rheology have made global mantle convection modeling with realistic parameters prohibitive. Here we present a new implicit solver that exhibits optimal algorithmic performance and is capable of extreme scaling for hard PDE problems, such as mantle convection. To maximize accuracy and minimize runtime, the solver incorporates a number of advances, including aggressive multi-octree adaptivity, mixed continuous-discontinuous discretization, arbitrarily-high-order accuracy, hybrid spectral/geometric/algebraic multigrid, and novel Schur-complement preconditioning. These features present enormous challenges for extreme scalability. We demonstrate that---contrary to conventional wisdom---algorithmically optimal implicit solvers can be designed that scale out to 1.5 million cores for severely nonlinear, ill-conditioned, heterogeneous, and anisotropic PDEs.

References

1. D. Depaolo, T. Cerling, S. Hemming, A. Knoll, F. Richter, L. Royden, R. Rudnick, L. Stixrude, and J. Trefil, "Origin and Evolution of Earth: Research Questions for a Changing Planet," National Academies Press, Committee on Grand Research Questions in the Solid Earth Sciences, National Research Council of the National Academies, 2008.
2. C. Burstedde, O. Ghattas, M. Gurnis, T. Isaac, G. Stadler, T. Warburton, and L. C. Wilcox, "Extreme-scale AMR," in Proceedings of SC10. ACM/IEEE, 2010.
3. L. Alisic, M. Gurnis, G. Stadler, C. Burstedde, and O. Ghattas, "Multi-scale dynamics and rheology of mantle flow with plates," Journal of Geophysical Research, vol. 117, p. B10402, 2012.
4. G. Stadler, M. Gurnis, C. Burstedde, L. C. Wilcox, L. Alisic, and O. Ghattas, "The dynamics of plate tectonics and mantle flow: From local to global scales," Science, vol. 329, no. 5995, pp. 1033--1038, 2010.
5. L. Alisic, M. Gurnis, G. Stadler, C. Burstedde, L. C. Wilcox, and O. Ghattas, "Slab stress and strain rate as constraints on global mantle flow," Geophysical Research Letters, vol. 37, p. L22308, 2010.
6. C. Burstedde, L. C. Wilcox, and O. Ghattas, "p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees," SIAM Journal on Scientific Computing, vol. 33, no. 3, pp. 1103--1133, 2011.
7. J. Dongarra, J. Hittinger, J. Bell, L. Chacón, R. Falgout, M. Heroux, P. Hovland, E. Ng, C. Webster, and S. Wild, "Applied mathematics research for exascale computing," Report of the DOE/ASCR Exascale Mathematics Working Group, 2014.
8. S. C. Eisenstat and H. F. Walker, "Choosing the forcing terms in an inexact Newton method," SIAM Journal on Scientific Computing, vol. 17, pp. 16--32, 1996.
9. B. Gmeiner, U. Rüde, H. Stengel, C. Waluga, and B. Wohlmuth, "Performance and scalability of Hierarchical Hybrid Multigrid solvers for Stokes systems," SIAM Journal on Scientific Computing, vol. 37, no. 2, pp. C143--C168, 2015.
10. S. Reiter, A. Vogel, I. Heppner, M. Rupp, and G. Wittum, "A massively parallel geometric multigrid solver on hierarchically distributed grids," Computing and Visualization in Science, vol. 16, no. 4, pp. 151--164, 2013.
11. A. H. Baker, R. D. Falgout, T. V. Kolev, and U. M. Yang, "Scaling hypre's multigrid solvers to 100,000 cores," in High-Performance Scientific Computing, M. W. Berry, K. A. Gallivan, E. Gallopoulos, A. Grama, B. Philippe, Y. Saad, and F. Saied, Eds. Springer London, 2012, pp. 261--279.
12. S. Badia, A. F. Martín, and J. Principe, "A highly scalable parallel implementation of balancing domain decomposition by constraints," SIAM Journal on Scientific Computing, vol. 36, no. 2, pp. C190--C218, 2014.
13. O. Ippisch and M. Blatt, "Scalability test of μϕ and the Parallel Algebraic Multigrid solver of DUNE-ISTL," in Jülich Blue Gene/P Extreme Scaling Workshop, no. FZJ-JSC-IB-2011-02. Jülich Supercomputing Centre, 2011.
14. H. Sundar, G. Biros, C. Burstedde, J. Rudi, O. Ghattas, and G. Stadler, "Parallel geometric-algebraic multigrid on unstructured forests of octrees," in Proceedings of SC12. ACM/IEEE, 2012.
15. C. Burstedde, O. Ghattas, M. Gurnis, E. Tan, T. Tu, G. Stadler, L. C. Wilcox, and S. Zhong, "Scalable adaptive mantle convection simulation on petascale supercomputers," in Proceedings of SC08. ACM/IEEE, 2008.
16. M. Kronbichler, T. Heister, and W. Bangerth, "High accuracy mantle convection simulation through modern numerical methods," Geophysical Journal International, vol. 191, no. 1, pp. 12--29, 2012.
17. G. Ranalli, Rheology of the Earth. Springer, 1995.
18. T. Isaac, C. Burstedde, L. C. Wilcox, and O. Ghattas, "Recursive algorithms for distributed forests of octrees," SIAM Journal on Scientific Computing (to appear), 2015, http://arxiv.org/abs/1406.0089.
19. M. O. Deville, P. F. Fischer, and E. H. Mund, High-Order Methods for Incompressible Fluid Flow, ser. Cambridge Monographs on Applied and Computational Mathematics. Cambridge, UK: Cambridge University Press, 2002, vol. 9.
20. M. Benzi, G. H. Golub, and J. Liesen, "Numerical solution of saddle point problems," Acta Numerica, vol. 14, pp. 1--137, 2005.
21. H. Elman, V. Howle, J. Shadid, R. Shuttleworth, and R. Tuminaro, "Block preconditioners based on approximate commutators," SIAM Journal on Scientific Computing, vol. 27, no. 5, pp. 1651--1668, 2006.
22. D. A. May and L. Moresi, "Preconditioned iterative methods for Stokes flow problems arising in computational geodynamics," Physics of the Earth and Planetary Interiors, vol. 171, pp. 33--47, 2008.
23. D. Rossinelli, B. Hejazialhosseini, P. Hadjidoukas, C. Bekas, A. Curioni, A. Bertsch, S. Futral, S. J. Schmidt, N. A. Adams, and P. Koumoutsakos, "11 pflop/s simulations of cloud cavitation collapse," in Proceedings of SC13. ACM/IEEE, 2013.
24. V. Karakasis, T. Gkountouvas, K. Kourtis, G. I. Goumas, and N. Koziris, "An extended compression format for the optimization of sparse matrix-vector multiplication," IEEE Trans. Parallel Distrib. Syst., vol. 24, no. 10, pp. 1930--1940, 2013.
25. J. Milano and P. Lembke, "IBM system Blue Gene solution: Blue Gene/Q hardware overview and installation planning," IBM, Tech. Rep. SG24-7872-01, May 2013.
26. K. Yoshii, K. Iskra, R. Gupta, P. Beckman, V. Vishwanath, C. Yu, and S. Coghlan, "Evaluating power-monitoring capabilities on IBM Blue Gene/P and Blue Gene/Q," in Proc. of the IEEE Int. Conf. Cluster Computing, Beijing, 2012, pp. 36--44.
27. U.S. Energy Information Administration (EIA), "Electric power monthly, with data from january 2015," U.S. Department of Energy, Tech. Rep., March 2015. {Online}. Available: http://www.eia.gov/electricity/monthly/pdf/epm.pdf
28. S. Zhong and M. Gurnis, "Controls on trench topography from dynamic models of subducted slabs," J. Geophys. Res., vol. 99, pp. 15 683--15 695, 1994.
29. T. R. A. Song and M. Simons, "Large trench-parallel gravity variations predict seismogenic behavior in subduction zones," Science, vol. 301, pp. 630--633, 2003.
30. V. Ratnaswamy, G. Stadler, and M. Gurnis, "Adjoint-based estimation of plate coupling in a non-linear mantle flow model: theory and examples," Geophysical Journal International, vol. 202, no. 2, pp. 768--786, 2015.
31. L. Ruff and H. Kanamori, "Seismic coupling and uncoupling at subduction zones," Tectonophysics, vol. 99, no. 2, pp. 99--117, 1983.
32. C. H. Scholz and J. Campos, "The seismic coupling of subduction zones revisited," Journal of Geophysical Research: Solid Earth, vol. 117, no. B5, 2012.
33. D. Forsyth and S. Uyeda, "On the relative importance of the driving forces of plate motion," Geophysical Journal International, vol. 43, no. 1, pp. 163--200, 1975.
34. B. Hager and R. O'Connell, "A simple global model of plate dynamics and mantle convection," J. Geophys. Res., vol. 86, pp. 4843--4867, 1981.

Index Terms

• Applied computing

  • Physical sciences and engineering

    • Earth and atmospheric sciences

• Computer systems organization

  • Dependable and fault-tolerant systems and networks

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Mathematics of computing

  • Mathematical analysis

    • Differential equations

      • Partial differential equations

• Networks

  • Network performance evaluation