Abstract
A Reduction – an accumulation over a set of values, using an associative and commutative operator – is a common computation in many numerical computations, including scientific computations, machine learning, computer vision, and financial analytics. Contemporary polyhedral-based compilation techniques make it possible to optimize reductions, such as prefix sums, in which each component of the reduction’s output potentially shares computation with another component in the reduction. Therefore an optimizing compiler can identify the computation shared between multiple components and generate code that computes the shared computation only once.
These techniques, however, do not support reductions that – when phrased in the language of the polyhedral model – span multiple dependent statements. In such cases, existing approaches can generate incorrect code that violates the data dependences of the original, unoptimized program.
In this work, we identify and formalize the optimization of dependent reductions as an integer bilinear program. We present a heuristic optimization algorithm that uses an affine sequential schedule of the program to determine how to simplfy reductions yet still preserve the program’s dependences.
We demonstrate that the algorithm provides optimal complexity for a set of benchmark programs from the literature on probabilistic inference algorithms, whose performance critically relies on simplifying these reductions. The complexities for 10 of the 11 programs improve siginifcantly by factors at least of the sizes of the input data, which are in the range of 104 to 106 for typical real application inputs. We also confirm the significance of the improvement by showing speedups in wall-clock time that range from 1.1x to over 106x.
Supplemental Material
Available for Download
Appendices to the paper.
- Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. 2010. Multi-dimensional Rankings, Program Termination, and Complexity Bounds of Flowchart Programs. In Static Analysis Symposium. https://hal.inria.fr/inria-00523298Google Scholar
Cross Ref
- Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. In arXiv e-prints.Google Scholar
- Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro Dalcin, Dag Sverre Seljebotn, and Kurt Smith. 2011. Cython: The Best of Both Worlds. Computing in Science and Engg. 13, 2 (March 2011 ), 31-39.Google Scholar
Digital Library
- Mohamed-Walid Benabderrahmane, Louis-Noël Pouchet, Albert Cohen, and Cédric Bastoul. 2010. The Polyhedral Model is More Widely Applicable Than You Think. In European Conference on Theory and Practice of Software, International Conference on Compiler Construction.Google Scholar
Digital Library
- Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research 20, 28 ( 2019 ).Google Scholar
- Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning ( Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg.Google Scholar
- David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3 (Jan. 2003 ), 993-1022.Google Scholar
Digital Library
- Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer. In Conference on Programming Language Design and Implementation.Google Scholar
- Jean-François Collard, Denis Barthou, and Paul Feautrier. 1995. Fuzzy Array Dataflow Analysis. In Symposium on Principles and Practice of Parallel Programming.Google Scholar
- D Collett. 1993. Modelling Survival Data in Medical Research. Chapman & Hall, New York.Google Scholar
- D. R. Cox. 1972. Regression Models and Life-Tables. Journal of the Royal Statistical Society: Series B (Methodological) 34, 2 ( 1972 ), 187-202.Google Scholar
Cross Ref
- Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019. Gen: a General-purpose Probabilistic Programming System with Programmable Inference. In Conference on Programming Language Design and Implementation.Google Scholar
Digital Library
- Greg Morisett Daniel Huang, Jean-Baptiste Tristan. 2017. Compiling Markov Chain Monte Carlo Algorithms for Probabilistic Modeling. In Conference on Programming Language Design and Implementation.Google Scholar
- Johannes Doerfert, Kevin Streit, Sebastian Hack, and Zino Benaissa. 2015. Polly's Polyhedral Scheduling in the Presence of Reductions. In International Workshop on Polyhedral Compilation Techniques.Google Scholar
- E. Ehrhardt. 1967. Sur un problème de Géométrie Diophantienne Linéaire. II. Journal für die Reine und Angewandte Mathematik 1967 ( 1967 ), 25-49. Issue 227.Google Scholar
- P. Feautrier. 1988. Array Expansion. In International Conference on Supercomputing.Google Scholar
- Paul Feautrier. 1992a. Some eficient solutions to the afine scheduling problem. I. One-dimensional time. International Journal of Parallel Programming 21, 5 (Oct. 1992 ), 313-347.Google Scholar
- Paul Feautrier. 1992b. Some eficient solutions to the afine scheduling problem. Part II. Multidimensional time. International Journal of Parallel Programming 21, 6 (Dec. 1992 ), 389-420.Google Scholar
- Robert M. Fung and Kuo-Chu Chang. 1989. Weighing and Integrating Evidence for Stochastic Simulation on Bayesian Networks. In Conference on Uncertainty in Artificial Intelligence.Google Scholar
- Gautam and S. Rajopadhye. 2006. Simplifying Reductions. In Symposium on Principles of Programming Languages.Google Scholar
- Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics 40, 5 ( 2015 ), 530-543.Google Scholar
Cross Ref
- Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of images. Transactions on Pattern Analysis and Machine Intelligence 6 ( Nov. 1984 ), 721-741. Issue 6.Google Scholar
Digital Library
- Philip Ginsbach and Michael F. P. O'Boyle. 2017. Discovery and Exploitation of General Reductions: A Constraint Based Approach. In International Symposium on Code Generation and Optimization.Google Scholar
- Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A language for generative models. In Conference on Uncertainty in Artificial Intelligence.Google Scholar
- Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org. Accessed: 2020-10-30.Google Scholar
- Scott Grauer-Gray and John Cavazos. 2011. Optimizing and Auto-tuning Belief Propagation on the GPU. In Workshop on Languages and Compilers for Parallel Computing.Google Scholar
- T. Grifiths and M. Steyvers. 2004. Finding Scientific Topics. Proceedings of the National Academy of Sciences 101, suppl. 1 (April 2004 ), 5228-5235.Google Scholar
- Gautam Gupta, Kim Daegon, and Sanjay Rajopadhye. 2007. Scheduling in the Z-Polyhedral Model. International Parallel and Distributed Processing Symposium.Google Scholar
- Gautam Gupta, Sanjay Rajopadhye, and Patrice Quinton. 2002. Scheduling Reductions on Realistic Machines. In Symposium on Parallel Algorithms and Architectures.Google Scholar
- W. K. Hastings. 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 1 (April 1970 ), 97-109.Google Scholar
Cross Ref
- Ian Holmes, Keith Harris, and Christopher Quince. 2012. Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. PLOS ONE ( 2012 ).Google Scholar
- Guillaume Iooss, Christophe Alias, and Sanjay Rajopadhye. 2014. On Program Equivalence with Reductions. In International Static Analysis Symposium.Google Scholar
- Jian Sun, Nan-Ning Zheng, and Heung-Yeung Shum. 2003. Stereo matching using belief propagation. Transactions on Pattern Analysis and Machine Intelligence 25, 7 ( July 2003 ), 787-800.Google Scholar
- Ryoichi Kikuchi. 1951. A Theory of Cooperative Phenomena. Physical Review 81 (March 1951 ), 988-1003. Issue 6.Google Scholar
- Oleg Kiselyov. 2016. Probabilistic Programming Language and its Incremental Evaluation. In Asian Symposium on Programming Languages and Systems.Google Scholar
Cross Ref
- Jun S. Liu. 1994. The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem. J. Amer. Statist. Assoc. 89, 427 (Sept. 1994 ), 958-966.Google Scholar
Cross Ref
- Yanhong A. Liu, Scott D. Stoller, Ning Li, and Tom Rothamel. 2005. Optimizing Aggregate Array Computations in Loops. ACM Transactions on Programming Languages and Systems 27, 1 (Jan. 2005 ), 91-125.Google Scholar
Digital Library
- Vikash Mansingkha, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Conference on Programming Language Design and Implementation.Google Scholar
- N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. 1953. Equation of State Calculations by Fast Computing Machines. Journal of Chemical Physics 21, 6 ( 1953 ), 1087-1092.Google Scholar
Cross Ref
- Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, Massachusets.Google Scholar
Digital Library
- Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru (System Description). In International Symposium on Functional and Logic Programming.Google Scholar
Cross Ref
- George L. Nemhauser and Laurence A. Wolsey. 1988. Integer and Combinatorial Optimization.Google Scholar
Digital Library
- David Newman. 2008. Bag of Words Dataset. In UCI Machine Learning Respository.Google Scholar
- Aditya V. Nori, Sherjil Ozair, Sriram K. Rajamani, and Deepak Vijaykeerthy. 2015. Eficient Synthesis of Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google Scholar
- David Padua (Ed.). 2011. Omega Calculator. Springer US, Boston, MA, 1355-1355. https://doi.org/10.1007/978-0-387-09766-4_2303 Google Scholar
Cross Ref
- Martyn Plummer. 2015. JAGS Version 4.0.0 user manual. Addison-Wesley, Reading, Massachusetts.Google Scholar
- Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and John Cavazos. 2008. Iterative optimization in the polyhedral model: Part II, multidimensional time.Google Scholar
- Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and Nicolas Vasilache. 2007. Iterative optimization in the polyhedral model: Part I, one-dimensional time. In International Symposium on Code Generation and Optimization.Google Scholar
- Louis-Noël Pouchet, Uday Bondhugula, Cédric Bastoul, Albert Cohen, J. Ramanujam, P. Sadayappan, and Nicolas Vasilache. 2011. Loop Transformations: Convexity, Pruning and Optimization. In Symposium on Principles of Programming Languages.Google Scholar
- L. Rauchwerger and D. A. Padua. 1999. The LRPD test: speculative run-time parallelization of loops with privatization and reduction parallelization. Transactions on Parallel and Distributed Systems 10, 2 (Feb. 1999 ), 160-180.Google Scholar
Digital Library
- C. Reddy, M. Kruse, and A. Cohen. 2016. Reduction drawing: Language constructs and polyhedral compilation for reductions on GPUs.Google Scholar
- Xavier Redon and Paul Feautrier. 1994. Scheduling Reductions. In International Conference on Supercomputing.Google Scholar
- Philip Resnik and Eric Hardisty. 2010. Gibbs Sampling for the Uninitiated. Technical Report.Google Scholar
- Daniel Ritchie, Andreas Stuhlmüller, and Noah Goodman. 2016. C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching. In International Conference on Artificial Intelligence and Statistics.Google Scholar
- Thomas Rubiano. 2017. Implicit Computational Complexity and Compilers. Ph.D. Dissertation.Google Scholar
- Yannick Saouter and Patrice Quinton. 1993. Computability of Recurrence Equations. Theoretical Computer Science 116, 2 (Aug. 1993 ), 317-337.Google Scholar
Digital Library
- Alexander Schrijver. 1986. Theory of Linear and Integer Programming. John Wiley & Sons, Inc., New York, NY, USA.Google Scholar
Digital Library
- G. A. Susto, A. Schirru, S. Pampuri, S. McLoone, and A. Beghi. 2015. Machine Learning for Predictive Maintenance: A Multiple Classifier Approach. Transactions on Industrial Informatics 11, 3 ( June 2015 ), 812-820.Google Scholar
- Patricia M Therneau, Terry M. ; Grambsch. 2013. Modeling Survival Data: Extending the Cox Model. Springer, New York.Google Scholar
Cross Ref
- Dustin Tran, Matthew D Hofman, Rif A Saurous, Eugene Brevdo, Kevin Murphy, and David M Blei. 2017. Deep probabilistic programming. In International Conference on Learning Representations.Google Scholar
- Peter Turnbaugh, Micah Hamady, Tanya Yatsunenko, Brandi Cantarel, Alexis Duncan, Ruth Ley, Mitchell Sogin, Joe Jones, Bruce A Roe, Jason Afourtit, Michael Egholm, Bernard Henrissat, Andrew C Heath, Rob Knight, and Jefrey I Gordon. 2008. A core gut microbiome in obese and lean twins. Nature 457 (12 2008 ), 480-4.Google Scholar
- Sven Verdoolaege. 2010. isl: An Integer Set Library for the Polyhedral Model. In International Congress on Mathematical Software.Google Scholar
Cross Ref
- Sven Verdoolaege. 2016. Presburger Formulas and Polyhedral Compilation. https://doi.org/10.13140 /RG.2.1.1174.6323 Google Scholar
Cross Ref
- Sven Verdoolaege, Hristo Nikolov, and Todor Stefanov. 2013. On Demand Parametric Array Dataflow Analysis. In International Workshop on Polyhedral Compilation Techniques.Google Scholar
- Sven Verdoolaege, Rachid Seghir, Kristof Beyls, Vincent Loechner, and Maurice Bruynooghe. 2007. Counting Integer Points in Parametric Polytopes Using Barvinok's Rational Functions. (May 2007 ).Google Scholar
- Rajan Walia, Praveen Narayanan, Jacques Carette, Sam Tobin-Hochstadt, and Chung-chieh Shan. 2019. From High-level Inference Algorithms to Eficient Code. In International Conference on Functional Programming.Google Scholar
- Nicola White, Fiona Reid, Adam Harris, Priscilla Harries, and Patrick Stone. 2016. A Systematic Review of Predictions of Survival in Palliative Care: How Accurate Are Clinicians and Who Are the Experts? PLOS ONE ( 08 2016 ).Google Scholar
- Yi Wu, Lei Li, Stuart Russell, and Rastislav Bodik. 2016. In International Joint Conferences on Artificial Intelligence.Google Scholar
- Lingfeng Yang, Patrick Hanrahan, and Noah Goodman. 2014. Generating Eficient MCMC Kernels from Probabilistic Programs. In International Conference on Artificial Intelligence and Statistics.Google Scholar
- Tomofumi Yuki, Gautam Gupta, DaeGon Kim, Tanveer Pathan, and Sanjay Rajopadhye. 2013. AlphaZ: A System for Design Space Exploration in the Polyhedral Model. In Workshop on Languages and Compilers for Parallel Computing.Google Scholar
- Jieyuan Zhang and Jingling Xue. 2019. Incremental Precision-Preserving Symbolic Inference for Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google Scholar
Index Terms
Simplifying dependent reductions in the polyhedral model
Recommendations
Simplifying reductions
Proceedings of the 2006 POPL ConferenceWe present optimization techniques for high level equational programs that are generalizations of affine control loops (ACLs). Significant parts of the SpecFP and PerfectClub benchmarks are ACLs. They often contain reductions: associative and ...
Simplifying reductions
POPL '06: Conference record of the 33rd ACM SIGPLAN-SIGACT symposium on Principles of programming languagesWe present optimization techniques for high level equational programs that are generalizations of affine control loops (ACLs). Significant parts of the SpecFP and PerfectClub benchmarks are ACLs. They often contain reductions: associative and ...
Global-view abstractions for user-defined reductions and scans
PPoPP '06: Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programmingSince APL, reductions and scans have been recognized as powerful programming concepts. Abstracting an accumulation loop (reduction) and an update loop (scan), the concepts have efficient parallel implementations based on the parallel prefix algorithm. ...






Comments