Abstract
Modern probabilistic programming languages aim to formalize and automate key aspects of probabilistic modeling and inference. Many languages provide constructs for programmable inference that enable developers to improve inference speed and accuracy by tailoring an algorithm for use with a particular model or dataset. Unfortunately, it is easy to use these constructs to write unsound programs that appear to run correctly but produce incorrect results. To address this problem, we present a denotational semantics for programmable inference in higher-order probabilistic programming languages, along with a type system that ensures that well-typed inference programs are sound by construction. A central insight is that the type of a probabilistic expression can track the space of its possible execution traces, not just the type of value that it returns, as these traces are often the objects that inference algorithms manipulate. We use our semantics and type system to establish soundness properties of custom inference programs that use constructs for variational, sequential Monte Carlo, importance sampling, and Markov chain Monte Carlo inference.
Supplemental Material
- Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. 2003. An Introduction to MCMC for Machine Learning. Machine Learning 50, 1 (Jan 2003), 5–43.Google Scholar
Cross Ref
- Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. (May 2018). arXiv: 1805.01863Google Scholar
- Sooraj Bhat, Ashish Agarwal, Richard Vuduc, and Alexander Gray. 2012. A Type Theory for Probability Density Functions. In Proceedings of the 39th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2012). ACM, New York, NY, USA, 545–556.Google Scholar
Digital Library
- Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res. 20, Article 28 (Feb. 2019), 6 pages.Google Scholar
- Keith A. Bonawitz. 2008. Composable Probabilistic Inference with Blaise. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A Probabilistic Programming Language. J. Stat. Softw. 76, 1 (2017), 1–32.Google Scholar
Cross Ref
- Simon Castellan and Hugo Paquet. 2019. Probabilistic Programming Inference Via Intensional Semantics. In European Symposium on Programming (ESOP 2019) (Lecture Notes in Computer Science), Vol. 11423. Springer, Berlin, 322–349.Google Scholar
- Joseph T. Chang and David Pollard. 1997. Conditioning as Disintegration. Stat. Neer. 51, 3 (Nov. 1997), 287–317.Google Scholar
Cross Ref
- Sourav Chatterjee and Persi Diaconis. 2018. The Sample Size Required in Importance Sampling. The Annals of Applied Probability 28, 2 (2018), 1099–1135.Google Scholar
Cross Ref
- Corinna Cortes, Yishay Mansour, and Mehryar Mohri. 2010. Learning Bounds for Importance Weighting. In Advances in Neural Information Processing Systems 23 (NIPS 2010). Curran Associates, Inc., Red Hook, NY, USA, 442–450.Google Scholar
- Marco Cusumano-Towner and Vikash K. Mansinghka. 2018. A Design Proposal for Gen: Probabilistic Programming with Fast Custom Inference via Code Generation. In Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL 2018). ACM, New York, NY, USA, 52–57.Google Scholar
Digital Library
- Marco F. Cusumano-Towner and Vikash K. Mansinghka. 2017. AIDE: An Algorithm for Measuring the Accuracy of Probabilistic Inference Algorithms. In Advances in Neural Information Processing Systems 30 (NIPS 2017). Curran Associates, Inc., Red Hook, NY, USA, 3000–3010.Google Scholar
- Marco F. Cusumano-Towner, Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. 2019. Gen: A General-purpose Probabilistic Programming System with Programmable Inference. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). ACM, New York, NY, USA, 221–236.Google Scholar
Digital Library
- Persi Diaconis. 2009. The Markov Chain Monte Carlo Revolution. Bull. Am. Math. Soc 46, 2 (2009), 179–205.Google Scholar
Cross Ref
- Adam Foster, Martin Jankowiak, Eli Bingham, Yee Whye Teh, Tom Rainforth, and Noah Goodman. 2019. Variational Optimal Experiment Design: Efficient Automation of Adaptive Experiments. (March 2019). arXiv: 1903.05480 NeurIPS 2019 Bayesian Deep Learning Workshop.Google Scholar
- Cameron E. Freer, Vikash K. Mansinghka, and Daniel M. Roy. 2010. When are Probabilistic Programs Probably Computationally Tractable?. In Workshop on Monte Carlo Methods for Modern Applications (NIPS 2010). Curran Associates, Inc., Red Hook, NY, USA. http://danroy.org/papers/FreerManRoy- NIPSMC- 2010.pdfGoogle Scholar
- Soichiro Fujii, Shin-ya Katsumata, and Paul-André Melliès. 2016. Towards a Formal Theory of Graded Monads. In Foundations of Software Science and Computation Structures (FOSSACS 2016) (Lecture Notes in Computer Science), Bart Jacobs and Christof Löding (Eds.), Vol. 9634. Springer, Berlin, 513–530.Google Scholar
- Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: A Language for Flexible Probabilistic Inference. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS 2018) (Proceedings of Machine Learning Research), Vol. 84. PMLR, 1682–1690.Google Scholar
- Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis (3 ed.). Taylor & Francis.Google Scholar
- Noah Goodman, Vikash Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A Language for Generative Models. In Proceedings of the 24th Annual Conference on Uncertainty in Artificial Intelligence (UAI 2008). AUAI Press, 220–229.Google Scholar
- Peter J. Green. 1995. Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination. Biometrika 82, 4 (Dec. 1995), 711–732.Google Scholar
Cross Ref
- Georges Harik and Noam Shazeer. 2010. Variational Program Inference. (June 2010). arXiv: 1006.0991Google Scholar
- Chris Heunen, Ohad Kammar, Sam Staton, Sean Moss, Matthijs Vákár, Adam Ścibior, and Hongseok Yang. 2018. The Semantic Structure of Quasi-Borel Spaces. https://pps2018.sice.indiana.edu/files/2018/01/pps18- qbs- semantic- structure.pdf Workshop on Probabilistic Programming Semantics (PPS 2018).Google Scholar
- Chris Heunen, Ohad Kammar, Sam Staton, and Hongseok Yang. 2017. A Convenient Category for Higher-order Probability Theory. In Proceedings of the 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2017). IEEE, Piscataway, NJ, USA, Article 77, 12 pages.Google Scholar
Digital Library
- Daniel Huang, Jean-Baptiste Tristan, and Greg Morrisett. 2017. Compiling Markov Chain Monte Carlo Algorithms For Probabilistic Modeling. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2017). ACM, New York, NY, USA, 111–125.Google Scholar
Digital Library
- Shin-ya Katsumata. 2014. Parametric Effect Monads and Semantics of Effect Systems. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2014). ACM, New York, NY, USA, 633–645.Google Scholar
- Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. 2019. Towards Verified Stochastic Variational Inference for Probabilistic Programs. (July 2019). arXiv: 1907.08827Google Scholar
- Jun S. Liu and Rong Chen. 1998. Sequential Monte Carlo Methods for Dynamic Systems. J. Am. Stat. Assoc. 93, 443 (1998), 1032–1044.Google Scholar
Cross Ref
- Vikash K. Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2018). ACM, New York, NY, USA, 603–616.Google Scholar
Digital Library
- Vikash K. Mansinghka, Daniel Selsam, and Yura Perov. 2014. Venture: A Higher-Order Probabilistic Programming Platform with Programmable Inference. (April 2014). arXiv: 1404.0099Google Scholar
- Lawrence M. Murray. 2013. Bayesian State-Space Modelling on High-performance Hardware Using Libbi. (June 2013). arXiv: 1306.3277Google Scholar
- Praveen Narayanan. 2019. Verifiable And Reusable Conditioning. Ph.D. Dissertation. Indiana University.Google Scholar
- Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference By Program Transformation In Hakaru (System Description). In Proceedings of the 13th International Symposium on Functional and Logic Programming (FLOPS 2016) (Lecture Notes in Computer Science), Vol. 9613. Springer, New York, NY, USA, 62–79.Google Scholar
Cross Ref
- Daniel Ritchie, Paul Horsfall, and Noah D. Goodman. 2016. Deep Amortized Inference For Probabilistic Programs. (Oct. 2016). arXiv: 1610.05735Google Scholar
- David A. Roberts, Marcus Gallagher, and Thomas Taimre. 2019. Reversible Jump Probabilistic Programming. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019), Vol. 89. PMLR, 634–643.Google Scholar
- Stuart Russell and Peter Norvig. 2016. Artificial Intelligence: A Modern Approach (4 ed.). Pearson Education Limited.Google Scholar
- Feras A. Saad, Marco Cusumano-Towner, Ulrich Schaechtle, Martin C. Rinard, and Vikash K. Mansinghka. 2019. Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling. Proc. ACM Program. Lang. 3, POPL, Article 37 (2019), 29 pages.Google Scholar
- Tetsuya Sato, Alejandro Aguirre, Gilles Barthe, Marco Gaboardi, Deepak Garg, and Justin Hsu. 2019. Formal Verification of Higher-order Probabilistic Programs: Reasoning About Approximation, Convergence, Bayesian Inference, and Optimization. Proc. ACM Program. Lang. 3, POPL, Article 38 (Jan. 2019), 30 pages.Google Scholar
Digital Library
- Adam Ścibior, Zoubin Ghahramani, and Andrew D. Gordon. 2015. Practical Probabilistic Programming with Monads. In Proceedings of the 2015 ACM SIGPLAN Symposium on Haskell (Haskell 2015). ACM, New York, NY, USA, 165–176.Google Scholar
Digital Library
- Adam Ścibior, Ohad Kammar, and Zoubin Ghahramani. 2018. Functional Programming for Modular Bayesian Inference. Proc. ACM Program. Lang. 2, ICFP, Article 83 (July 2018), 29 pages.Google Scholar
- Adam Ścibior, Ohad Kammar, Matthijs Vákár, Sam Staton, Hongseok Yang, Yufei Cai, Klaus Ostermann, Sean K Moss, Chris Heunen, and Zoubin Ghahramani. 2017. Denotational Validation of Higher-Order Bayesian Inference. Proc. ACM Program. Lang. 2, POPL (Jan. 2017), 60.Google Scholar
- Adam Ścibior and Michael Thomas. 2019. Strongly Typed Tracing of Probabilistic Programs. https://www.cs.ubc.ca/ ~ascibior/assets/pdf/lafi19a.pdfGoogle Scholar
- Sam Staton, Hongseok Yang, Frank Wood, Chris Heunen, and Ohad Kammar. 2016. Semantics for Probabilistic Programming: Higher-order Functions, Continuous Distributions, and Soft Constraints. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2016). ACM, New York, NY, USA, 525–534.Google Scholar
Digital Library
- Sebastian Thrun, Wolfram Burgard, and Dieter Fox. 2005. Probabilistic Robotics. MIT Press.Google Scholar
- Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Programming. In International Conference on Learning Representations (ICLR 2017).Google Scholar
- David Wingate, Andreas Stuhlmüller, and Noah Goodman. 2011. Lightweight Implementations of Probabilistic Programming Languages via Transformational Compilation. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS 2011) (Proceedings of Machine Learning Research), Vol. 15. PMLR, 770–778.Google Scholar
- David Wingate and Theophane Weber. 2013. Automated Variational Inference in Probabilistic Programming. (Jan. 2013). arXiv: 1301.1299Google Scholar
- Frank Wood, Jan Willem Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS 2014) (Proceedings of Machine Learning Research), Vol. 33. PMLR, 1024–1032.Google Scholar
- Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. 2019. Advances in Variational Inference. IEEE Trans. Pattern Anal. Mach. Intell. 41, 8 (Aug. 2019), 2008–2026.Google Scholar
Cross Ref
- Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, and Frank Wood. 2019. LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019) (Proceedings of Machine Learning Research), Vol. 89. PMLR, 148–157.Google Scholar
- Robert Zinkov and Chung-chieh Shan. 2016. Composing Inference Algorithms as Program Transformations. (March 2016). arXiv: 1603.01882Google Scholar
Index Terms
Trace types and denotational semantics for sound programmable inference in probabilistic languages
Recommendations
Sound probabilistic inference via guide types
PLDI 2021: Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and ImplementationProbabilistic programming languages aim to describe and automate Bayesian modeling and inference. Modern languages support programmable inference, which allows users to customize inference algorithms by incorporating guide programs to improve inference ...
Probabilistic programming with programmable inference
PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and ImplementationWe introduce inference metaprogramming for probabilistic programming languages, including new language constructs, a formalism, and the rst demonstration of e ectiveness in practice. Instead of relying on rigid black-box inference algorithms hard-coded ...
Probabilistic abductive logic programming using Dirichlet priors
Probabilistic programming is an area of research that aims to develop general inference algorithms for probabilistic models expressed as probabilistic programs whose execution corresponds to inferring the parameters of those models. In this paper, we ...






Comments