skip to main content
research-article
Open Access

Simplifying dependent reductions in the polyhedral model

Published:04 January 2021Publication History
Skip Abstract Section

Abstract

A Reduction – an accumulation over a set of values, using an associative and commutative operator – is a common computation in many numerical computations, including scientific computations, machine learning, computer vision, and financial analytics. Contemporary polyhedral-based compilation techniques make it possible to optimize reductions, such as prefix sums, in which each component of the reduction’s output potentially shares computation with another component in the reduction. Therefore an optimizing compiler can identify the computation shared between multiple components and generate code that computes the shared computation only once.

These techniques, however, do not support reductions that – when phrased in the language of the polyhedral model – span multiple dependent statements. In such cases, existing approaches can generate incorrect code that violates the data dependences of the original, unoptimized program.

In this work, we identify and formalize the optimization of dependent reductions as an integer bilinear program. We present a heuristic optimization algorithm that uses an affine sequential schedule of the program to determine how to simplfy reductions yet still preserve the program’s dependences.

We demonstrate that the algorithm provides optimal complexity for a set of benchmark programs from the literature on probabilistic inference algorithms, whose performance critically relies on simplifying these reductions. The complexities for 10 of the 11 programs improve siginifcantly by factors at least of the sizes of the input data, which are in the range of 104 to 106 for typical real application inputs. We also confirm the significance of the improvement by showing speedups in wall-clock time that range from 1.1x to over 106x.

Skip Supplemental Material Section

Supplemental Material

References

  1. Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. 2010. Multi-dimensional Rankings, Program Termination, and Complexity Bounds of Flowchart Programs. In Static Analysis Symposium. https://hal.inria.fr/inria-00523298Google ScholarGoogle ScholarCross RefCross Ref
  2. Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. In arXiv e-prints.Google ScholarGoogle Scholar
  3. Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro Dalcin, Dag Sverre Seljebotn, and Kurt Smith. 2011. Cython: The Best of Both Worlds. Computing in Science and Engg. 13, 2 (March 2011 ), 31-39.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Mohamed-Walid Benabderrahmane, Louis-Noël Pouchet, Albert Cohen, and Cédric Bastoul. 2010. The Polyhedral Model is More Widely Applicable Than You Think. In European Conference on Theory and Practice of Software, International Conference on Compiler Construction.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research 20, 28 ( 2019 ).Google ScholarGoogle Scholar
  6. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning ( Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg.Google ScholarGoogle Scholar
  7. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3 (Jan. 2003 ), 993-1022.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  9. Jean-François Collard, Denis Barthou, and Paul Feautrier. 1995. Fuzzy Array Dataflow Analysis. In Symposium on Principles and Practice of Parallel Programming.Google ScholarGoogle Scholar
  10. D Collett. 1993. Modelling Survival Data in Medical Research. Chapman & Hall, New York.Google ScholarGoogle Scholar
  11. D. R. Cox. 1972. Regression Models and Life-Tables. Journal of the Royal Statistical Society: Series B (Methodological) 34, 2 ( 1972 ), 187-202.Google ScholarGoogle ScholarCross RefCross Ref
  12. Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019. Gen: a General-purpose Probabilistic Programming System with Programmable Inference. In Conference on Programming Language Design and Implementation.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Greg Morisett Daniel Huang, Jean-Baptiste Tristan. 2017. Compiling Markov Chain Monte Carlo Algorithms for Probabilistic Modeling. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  14. Johannes Doerfert, Kevin Streit, Sebastian Hack, and Zino Benaissa. 2015. Polly's Polyhedral Scheduling in the Presence of Reductions. In International Workshop on Polyhedral Compilation Techniques.Google ScholarGoogle Scholar
  15. E. Ehrhardt. 1967. Sur un problème de Géométrie Diophantienne Linéaire. II. Journal für die Reine und Angewandte Mathematik 1967 ( 1967 ), 25-49. Issue 227.Google ScholarGoogle Scholar
  16. P. Feautrier. 1988. Array Expansion. In International Conference on Supercomputing.Google ScholarGoogle Scholar
  17. Paul Feautrier. 1992a. Some eficient solutions to the afine scheduling problem. I. One-dimensional time. International Journal of Parallel Programming 21, 5 (Oct. 1992 ), 313-347.Google ScholarGoogle Scholar
  18. Paul Feautrier. 1992b. Some eficient solutions to the afine scheduling problem. Part II. Multidimensional time. International Journal of Parallel Programming 21, 6 (Dec. 1992 ), 389-420.Google ScholarGoogle Scholar
  19. Robert M. Fung and Kuo-Chu Chang. 1989. Weighing and Integrating Evidence for Stochastic Simulation on Bayesian Networks. In Conference on Uncertainty in Artificial Intelligence.Google ScholarGoogle Scholar
  20. Gautam and S. Rajopadhye. 2006. Simplifying Reductions. In Symposium on Principles of Programming Languages.Google ScholarGoogle Scholar
  21. Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics 40, 5 ( 2015 ), 530-543.Google ScholarGoogle ScholarCross RefCross Ref
  22. Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of images. Transactions on Pattern Analysis and Machine Intelligence 6 ( Nov. 1984 ), 721-741. Issue 6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Philip Ginsbach and Michael F. P. O'Boyle. 2017. Discovery and Exploitation of General Reductions: A Constraint Based Approach. In International Symposium on Code Generation and Optimization.Google ScholarGoogle Scholar
  24. Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A language for generative models. In Conference on Uncertainty in Artificial Intelligence.Google ScholarGoogle Scholar
  25. Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org. Accessed: 2020-10-30.Google ScholarGoogle Scholar
  26. Scott Grauer-Gray and John Cavazos. 2011. Optimizing and Auto-tuning Belief Propagation on the GPU. In Workshop on Languages and Compilers for Parallel Computing.Google ScholarGoogle Scholar
  27. T. Grifiths and M. Steyvers. 2004. Finding Scientific Topics. Proceedings of the National Academy of Sciences 101, suppl. 1 (April 2004 ), 5228-5235.Google ScholarGoogle Scholar
  28. Gautam Gupta, Kim Daegon, and Sanjay Rajopadhye. 2007. Scheduling in the Z-Polyhedral Model. International Parallel and Distributed Processing Symposium.Google ScholarGoogle Scholar
  29. Gautam Gupta, Sanjay Rajopadhye, and Patrice Quinton. 2002. Scheduling Reductions on Realistic Machines. In Symposium on Parallel Algorithms and Architectures.Google ScholarGoogle Scholar
  30. W. K. Hastings. 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 1 (April 1970 ), 97-109.Google ScholarGoogle ScholarCross RefCross Ref
  31. Ian Holmes, Keith Harris, and Christopher Quince. 2012. Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. PLOS ONE ( 2012 ).Google ScholarGoogle Scholar
  32. Guillaume Iooss, Christophe Alias, and Sanjay Rajopadhye. 2014. On Program Equivalence with Reductions. In International Static Analysis Symposium.Google ScholarGoogle Scholar
  33. Jian Sun, Nan-Ning Zheng, and Heung-Yeung Shum. 2003. Stereo matching using belief propagation. Transactions on Pattern Analysis and Machine Intelligence 25, 7 ( July 2003 ), 787-800.Google ScholarGoogle Scholar
  34. Ryoichi Kikuchi. 1951. A Theory of Cooperative Phenomena. Physical Review 81 (March 1951 ), 988-1003. Issue 6.Google ScholarGoogle Scholar
  35. Oleg Kiselyov. 2016. Probabilistic Programming Language and its Incremental Evaluation. In Asian Symposium on Programming Languages and Systems.Google ScholarGoogle ScholarCross RefCross Ref
  36. Jun S. Liu. 1994. The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem. J. Amer. Statist. Assoc. 89, 427 (Sept. 1994 ), 958-966.Google ScholarGoogle ScholarCross RefCross Ref
  37. Yanhong A. Liu, Scott D. Stoller, Ning Li, and Tom Rothamel. 2005. Optimizing Aggregate Array Computations in Loops. ACM Transactions on Programming Languages and Systems 27, 1 (Jan. 2005 ), 91-125.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Vikash Mansingkha, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  39. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. 1953. Equation of State Calculations by Fast Computing Machines. Journal of Chemical Physics 21, 6 ( 1953 ), 1087-1092.Google ScholarGoogle ScholarCross RefCross Ref
  40. Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, Massachusets.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru (System Description). In International Symposium on Functional and Logic Programming.Google ScholarGoogle ScholarCross RefCross Ref
  42. George L. Nemhauser and Laurence A. Wolsey. 1988. Integer and Combinatorial Optimization.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. David Newman. 2008. Bag of Words Dataset. In UCI Machine Learning Respository.Google ScholarGoogle Scholar
  44. Aditya V. Nori, Sherjil Ozair, Sriram K. Rajamani, and Deepak Vijaykeerthy. 2015. Eficient Synthesis of Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  45. David Padua (Ed.). 2011. Omega Calculator. Springer US, Boston, MA, 1355-1355. https://doi.org/10.1007/978-0-387-09766-4_2303 Google ScholarGoogle ScholarCross RefCross Ref
  46. Martyn Plummer. 2015. JAGS Version 4.0.0 user manual. Addison-Wesley, Reading, Massachusetts.Google ScholarGoogle Scholar
  47. Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and John Cavazos. 2008. Iterative optimization in the polyhedral model: Part II, multidimensional time.Google ScholarGoogle Scholar
  48. Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and Nicolas Vasilache. 2007. Iterative optimization in the polyhedral model: Part I, one-dimensional time. In International Symposium on Code Generation and Optimization.Google ScholarGoogle Scholar
  49. Louis-Noël Pouchet, Uday Bondhugula, Cédric Bastoul, Albert Cohen, J. Ramanujam, P. Sadayappan, and Nicolas Vasilache. 2011. Loop Transformations: Convexity, Pruning and Optimization. In Symposium on Principles of Programming Languages.Google ScholarGoogle Scholar
  50. L. Rauchwerger and D. A. Padua. 1999. The LRPD test: speculative run-time parallelization of loops with privatization and reduction parallelization. Transactions on Parallel and Distributed Systems 10, 2 (Feb. 1999 ), 160-180.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. C. Reddy, M. Kruse, and A. Cohen. 2016. Reduction drawing: Language constructs and polyhedral compilation for reductions on GPUs.Google ScholarGoogle Scholar
  52. Xavier Redon and Paul Feautrier. 1994. Scheduling Reductions. In International Conference on Supercomputing.Google ScholarGoogle Scholar
  53. Philip Resnik and Eric Hardisty. 2010. Gibbs Sampling for the Uninitiated. Technical Report.Google ScholarGoogle Scholar
  54. Daniel Ritchie, Andreas Stuhlmüller, and Noah Goodman. 2016. C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching. In International Conference on Artificial Intelligence and Statistics.Google ScholarGoogle Scholar
  55. Thomas Rubiano. 2017. Implicit Computational Complexity and Compilers. Ph.D. Dissertation.Google ScholarGoogle Scholar
  56. Yannick Saouter and Patrice Quinton. 1993. Computability of Recurrence Equations. Theoretical Computer Science 116, 2 (Aug. 1993 ), 317-337.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Alexander Schrijver. 1986. Theory of Linear and Integer Programming. John Wiley & Sons, Inc., New York, NY, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. G. A. Susto, A. Schirru, S. Pampuri, S. McLoone, and A. Beghi. 2015. Machine Learning for Predictive Maintenance: A Multiple Classifier Approach. Transactions on Industrial Informatics 11, 3 ( June 2015 ), 812-820.Google ScholarGoogle Scholar
  59. Patricia M Therneau, Terry M. ; Grambsch. 2013. Modeling Survival Data: Extending the Cox Model. Springer, New York.Google ScholarGoogle ScholarCross RefCross Ref
  60. Dustin Tran, Matthew D Hofman, Rif A Saurous, Eugene Brevdo, Kevin Murphy, and David M Blei. 2017. Deep probabilistic programming. In International Conference on Learning Representations.Google ScholarGoogle Scholar
  61. Peter Turnbaugh, Micah Hamady, Tanya Yatsunenko, Brandi Cantarel, Alexis Duncan, Ruth Ley, Mitchell Sogin, Joe Jones, Bruce A Roe, Jason Afourtit, Michael Egholm, Bernard Henrissat, Andrew C Heath, Rob Knight, and Jefrey I Gordon. 2008. A core gut microbiome in obese and lean twins. Nature 457 (12 2008 ), 480-4.Google ScholarGoogle Scholar
  62. Sven Verdoolaege. 2010. isl: An Integer Set Library for the Polyhedral Model. In International Congress on Mathematical Software.Google ScholarGoogle ScholarCross RefCross Ref
  63. Sven Verdoolaege. 2016. Presburger Formulas and Polyhedral Compilation. https://doi.org/10.13140 /RG.2.1.1174.6323 Google ScholarGoogle ScholarCross RefCross Ref
  64. Sven Verdoolaege, Hristo Nikolov, and Todor Stefanov. 2013. On Demand Parametric Array Dataflow Analysis. In International Workshop on Polyhedral Compilation Techniques.Google ScholarGoogle Scholar
  65. Sven Verdoolaege, Rachid Seghir, Kristof Beyls, Vincent Loechner, and Maurice Bruynooghe. 2007. Counting Integer Points in Parametric Polytopes Using Barvinok's Rational Functions. (May 2007 ).Google ScholarGoogle Scholar
  66. Rajan Walia, Praveen Narayanan, Jacques Carette, Sam Tobin-Hochstadt, and Chung-chieh Shan. 2019. From High-level Inference Algorithms to Eficient Code. In International Conference on Functional Programming.Google ScholarGoogle Scholar
  67. Nicola White, Fiona Reid, Adam Harris, Priscilla Harries, and Patrick Stone. 2016. A Systematic Review of Predictions of Survival in Palliative Care: How Accurate Are Clinicians and Who Are the Experts? PLOS ONE ( 08 2016 ).Google ScholarGoogle Scholar
  68. Yi Wu, Lei Li, Stuart Russell, and Rastislav Bodik. 2016. In International Joint Conferences on Artificial Intelligence.Google ScholarGoogle Scholar
  69. Lingfeng Yang, Patrick Hanrahan, and Noah Goodman. 2014. Generating Eficient MCMC Kernels from Probabilistic Programs. In International Conference on Artificial Intelligence and Statistics.Google ScholarGoogle Scholar
  70. Tomofumi Yuki, Gautam Gupta, DaeGon Kim, Tanveer Pathan, and Sanjay Rajopadhye. 2013. AlphaZ: A System for Design Space Exploration in the Polyhedral Model. In Workshop on Languages and Compilers for Parallel Computing.Google ScholarGoogle Scholar
  71. Jieyuan Zhang and Jingling Xue. 2019. Incremental Precision-Preserving Symbolic Inference for Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar

Index Terms

  1. Simplifying dependent reductions in the polyhedral model

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          • Published in

            cover image Proceedings of the ACM on Programming Languages
            Proceedings of the ACM on Programming Languages  Volume 5, Issue POPL
            January 2021
            1789 pages
            EISSN:2475-1421
            DOI:10.1145/3445980
            Issue’s Table of Contents

            Copyright © 2021 Owner/Author

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 4 January 2021
            Published in pacmpl Volume 5, Issue POPL

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!