skip to main content
research-article
Open Access

Trace types and denotational semantics for sound programmable inference in probabilistic languages

Published:20 December 2019Publication History
Skip Abstract Section

Abstract

Modern probabilistic programming languages aim to formalize and automate key aspects of probabilistic modeling and inference. Many languages provide constructs for programmable inference that enable developers to improve inference speed and accuracy by tailoring an algorithm for use with a particular model or dataset. Unfortunately, it is easy to use these constructs to write unsound programs that appear to run correctly but produce incorrect results. To address this problem, we present a denotational semantics for programmable inference in higher-order probabilistic programming languages, along with a type system that ensures that well-typed inference programs are sound by construction. A central insight is that the type of a probabilistic expression can track the space of its possible execution traces, not just the type of value that it returns, as these traces are often the objects that inference algorithms manipulate. We use our semantics and type system to establish soundness properties of custom inference programs that use constructs for variational, sequential Monte Carlo, importance sampling, and Markov chain Monte Carlo inference.

Skip Supplemental Material Section

Supplemental Material

a19-lew.webm

References

  1. Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. 2003. An Introduction to MCMC for Machine Learning. Machine Learning 50, 1 (Jan 2003), 5–43.Google ScholarGoogle ScholarCross RefCross Ref
  2. Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. (May 2018). arXiv: 1805.01863Google ScholarGoogle Scholar
  3. Sooraj Bhat, Ashish Agarwal, Richard Vuduc, and Alexander Gray. 2012. A Type Theory for Probability Density Functions. In Proceedings of the 39th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2012). ACM, New York, NY, USA, 545–556.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res. 20, Article 28 (Feb. 2019), 6 pages.Google ScholarGoogle Scholar
  5. Keith A. Bonawitz. 2008. Composable Probabilistic Inference with Blaise. Ph.D. Dissertation. Massachusetts Institute of Technology.Google ScholarGoogle Scholar
  6. Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A Probabilistic Programming Language. J. Stat. Softw. 76, 1 (2017), 1–32.Google ScholarGoogle ScholarCross RefCross Ref
  7. Simon Castellan and Hugo Paquet. 2019. Probabilistic Programming Inference Via Intensional Semantics. In European Symposium on Programming (ESOP 2019) (Lecture Notes in Computer Science), Vol. 11423. Springer, Berlin, 322–349.Google ScholarGoogle Scholar
  8. Joseph T. Chang and David Pollard. 1997. Conditioning as Disintegration. Stat. Neer. 51, 3 (Nov. 1997), 287–317.Google ScholarGoogle ScholarCross RefCross Ref
  9. Sourav Chatterjee and Persi Diaconis. 2018. The Sample Size Required in Importance Sampling. The Annals of Applied Probability 28, 2 (2018), 1099–1135.Google ScholarGoogle ScholarCross RefCross Ref
  10. Corinna Cortes, Yishay Mansour, and Mehryar Mohri. 2010. Learning Bounds for Importance Weighting. In Advances in Neural Information Processing Systems 23 (NIPS 2010). Curran Associates, Inc., Red Hook, NY, USA, 442–450.Google ScholarGoogle Scholar
  11. Marco Cusumano-Towner and Vikash K. Mansinghka. 2018. A Design Proposal for Gen: Probabilistic Programming with Fast Custom Inference via Code Generation. In Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL 2018). ACM, New York, NY, USA, 52–57.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Marco F. Cusumano-Towner and Vikash K. Mansinghka. 2017. AIDE: An Algorithm for Measuring the Accuracy of Probabilistic Inference Algorithms. In Advances in Neural Information Processing Systems 30 (NIPS 2017). Curran Associates, Inc., Red Hook, NY, USA, 3000–3010.Google ScholarGoogle Scholar
  13. Marco F. Cusumano-Towner, Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. 2019. Gen: A General-purpose Probabilistic Programming System with Programmable Inference. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). ACM, New York, NY, USA, 221–236.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Persi Diaconis. 2009. The Markov Chain Monte Carlo Revolution. Bull. Am. Math. Soc 46, 2 (2009), 179–205.Google ScholarGoogle ScholarCross RefCross Ref
  15. Adam Foster, Martin Jankowiak, Eli Bingham, Yee Whye Teh, Tom Rainforth, and Noah Goodman. 2019. Variational Optimal Experiment Design: Efficient Automation of Adaptive Experiments. (March 2019). arXiv: 1903.05480 NeurIPS 2019 Bayesian Deep Learning Workshop.Google ScholarGoogle Scholar
  16. Cameron E. Freer, Vikash K. Mansinghka, and Daniel M. Roy. 2010. When are Probabilistic Programs Probably Computationally Tractable?. In Workshop on Monte Carlo Methods for Modern Applications (NIPS 2010). Curran Associates, Inc., Red Hook, NY, USA. http://danroy.org/papers/FreerManRoy- NIPSMC- 2010.pdfGoogle ScholarGoogle Scholar
  17. Soichiro Fujii, Shin-ya Katsumata, and Paul-André Melliès. 2016. Towards a Formal Theory of Graded Monads. In Foundations of Software Science and Computation Structures (FOSSACS 2016) (Lecture Notes in Computer Science), Bart Jacobs and Christof Löding (Eds.), Vol. 9634. Springer, Berlin, 513–530.Google ScholarGoogle Scholar
  18. Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: A Language for Flexible Probabilistic Inference. In Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS 2018) (Proceedings of Machine Learning Research), Vol. 84. PMLR, 1682–1690.Google ScholarGoogle Scholar
  19. Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis (3 ed.). Taylor & Francis.Google ScholarGoogle Scholar
  20. Noah Goodman, Vikash Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A Language for Generative Models. In Proceedings of the 24th Annual Conference on Uncertainty in Artificial Intelligence (UAI 2008). AUAI Press, 220–229.Google ScholarGoogle Scholar
  21. Peter J. Green. 1995. Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination. Biometrika 82, 4 (Dec. 1995), 711–732.Google ScholarGoogle ScholarCross RefCross Ref
  22. Georges Harik and Noam Shazeer. 2010. Variational Program Inference. (June 2010). arXiv: 1006.0991Google ScholarGoogle Scholar
  23. Chris Heunen, Ohad Kammar, Sam Staton, Sean Moss, Matthijs Vákár, Adam Ścibior, and Hongseok Yang. 2018. The Semantic Structure of Quasi-Borel Spaces. https://pps2018.sice.indiana.edu/files/2018/01/pps18- qbs- semantic- structure.pdf Workshop on Probabilistic Programming Semantics (PPS 2018).Google ScholarGoogle Scholar
  24. Chris Heunen, Ohad Kammar, Sam Staton, and Hongseok Yang. 2017. A Convenient Category for Higher-order Probability Theory. In Proceedings of the 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2017). IEEE, Piscataway, NJ, USA, Article 77, 12 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Daniel Huang, Jean-Baptiste Tristan, and Greg Morrisett. 2017. Compiling Markov Chain Monte Carlo Algorithms For Probabilistic Modeling. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2017). ACM, New York, NY, USA, 111–125.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Shin-ya Katsumata. 2014. Parametric Effect Monads and Semantics of Effect Systems. In Proceedings of the 41st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2014). ACM, New York, NY, USA, 633–645.Google ScholarGoogle Scholar
  27. Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. 2019. Towards Verified Stochastic Variational Inference for Probabilistic Programs. (July 2019). arXiv: 1907.08827Google ScholarGoogle Scholar
  28. Jun S. Liu and Rong Chen. 1998. Sequential Monte Carlo Methods for Dynamic Systems. J. Am. Stat. Assoc. 93, 443 (1998), 1032–1044.Google ScholarGoogle ScholarCross RefCross Ref
  29. Vikash K. Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2018). ACM, New York, NY, USA, 603–616.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Vikash K. Mansinghka, Daniel Selsam, and Yura Perov. 2014. Venture: A Higher-Order Probabilistic Programming Platform with Programmable Inference. (April 2014). arXiv: 1404.0099Google ScholarGoogle Scholar
  31. Lawrence M. Murray. 2013. Bayesian State-Space Modelling on High-performance Hardware Using Libbi. (June 2013). arXiv: 1306.3277Google ScholarGoogle Scholar
  32. Praveen Narayanan. 2019. Verifiable And Reusable Conditioning. Ph.D. Dissertation. Indiana University.Google ScholarGoogle Scholar
  33. Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference By Program Transformation In Hakaru (System Description). In Proceedings of the 13th International Symposium on Functional and Logic Programming (FLOPS 2016) (Lecture Notes in Computer Science), Vol. 9613. Springer, New York, NY, USA, 62–79.Google ScholarGoogle ScholarCross RefCross Ref
  34. Daniel Ritchie, Paul Horsfall, and Noah D. Goodman. 2016. Deep Amortized Inference For Probabilistic Programs. (Oct. 2016). arXiv: 1610.05735Google ScholarGoogle Scholar
  35. David A. Roberts, Marcus Gallagher, and Thomas Taimre. 2019. Reversible Jump Probabilistic Programming. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019), Vol. 89. PMLR, 634–643.Google ScholarGoogle Scholar
  36. Stuart Russell and Peter Norvig. 2016. Artificial Intelligence: A Modern Approach (4 ed.). Pearson Education Limited.Google ScholarGoogle Scholar
  37. Feras A. Saad, Marco Cusumano-Towner, Ulrich Schaechtle, Martin C. Rinard, and Vikash K. Mansinghka. 2019. Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling. Proc. ACM Program. Lang. 3, POPL, Article 37 (2019), 29 pages.Google ScholarGoogle Scholar
  38. Tetsuya Sato, Alejandro Aguirre, Gilles Barthe, Marco Gaboardi, Deepak Garg, and Justin Hsu. 2019. Formal Verification of Higher-order Probabilistic Programs: Reasoning About Approximation, Convergence, Bayesian Inference, and Optimization. Proc. ACM Program. Lang. 3, POPL, Article 38 (Jan. 2019), 30 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Adam Ścibior, Zoubin Ghahramani, and Andrew D. Gordon. 2015. Practical Probabilistic Programming with Monads. In Proceedings of the 2015 ACM SIGPLAN Symposium on Haskell (Haskell 2015). ACM, New York, NY, USA, 165–176.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Adam Ścibior, Ohad Kammar, and Zoubin Ghahramani. 2018. Functional Programming for Modular Bayesian Inference. Proc. ACM Program. Lang. 2, ICFP, Article 83 (July 2018), 29 pages.Google ScholarGoogle Scholar
  41. Adam Ścibior, Ohad Kammar, Matthijs Vákár, Sam Staton, Hongseok Yang, Yufei Cai, Klaus Ostermann, Sean K Moss, Chris Heunen, and Zoubin Ghahramani. 2017. Denotational Validation of Higher-Order Bayesian Inference. Proc. ACM Program. Lang. 2, POPL (Jan. 2017), 60.Google ScholarGoogle Scholar
  42. Adam Ścibior and Michael Thomas. 2019. Strongly Typed Tracing of Probabilistic Programs. https://www.cs.ubc.ca/ ~ascibior/assets/pdf/lafi19a.pdfGoogle ScholarGoogle Scholar
  43. Sam Staton, Hongseok Yang, Frank Wood, Chris Heunen, and Ohad Kammar. 2016. Semantics for Probabilistic Programming: Higher-order Functions, Continuous Distributions, and Soft Constraints. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS 2016). ACM, New York, NY, USA, 525–534.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Sebastian Thrun, Wolfram Burgard, and Dieter Fox. 2005. Probabilistic Robotics. MIT Press.Google ScholarGoogle Scholar
  45. Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Programming. In International Conference on Learning Representations (ICLR 2017).Google ScholarGoogle Scholar
  46. David Wingate, Andreas Stuhlmüller, and Noah Goodman. 2011. Lightweight Implementations of Probabilistic Programming Languages via Transformational Compilation. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS 2011) (Proceedings of Machine Learning Research), Vol. 15. PMLR, 770–778.Google ScholarGoogle Scholar
  47. David Wingate and Theophane Weber. 2013. Automated Variational Inference in Probabilistic Programming. (Jan. 2013). arXiv: 1301.1299Google ScholarGoogle Scholar
  48. Frank Wood, Jan Willem Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS 2014) (Proceedings of Machine Learning Research), Vol. 33. PMLR, 1024–1032.Google ScholarGoogle Scholar
  49. Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. 2019. Advances in Variational Inference. IEEE Trans. Pattern Anal. Mach. Intell. 41, 8 (Aug. 2019), 2008–2026.Google ScholarGoogle ScholarCross RefCross Ref
  50. Yuan Zhou, Bradley J. Gram-Hansen, Tobias Kohn, Tom Rainforth, Hongseok Yang, and Frank Wood. 2019. LF-PPL: A Low-Level First Order Probabilistic Programming Language for Non-Differentiable Models. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019) (Proceedings of Machine Learning Research), Vol. 89. PMLR, 148–157.Google ScholarGoogle Scholar
  51. Robert Zinkov and Chung-chieh Shan. 2016. Composing Inference Algorithms as Program Transformations. (March 2016). arXiv: 1603.01882Google ScholarGoogle Scholar

Index Terms

  1. Trace types and denotational semantics for sound programmable inference in probabilistic languages

              Recommendations

              Comments

              Login options

              Check if you have access through your login credentials or your institution to get full access on this article.

              Sign in

              Full Access

              PDF Format

              View or Download as a PDF file.

              PDF

              eReader

              View online with eReader.

              eReader
              About Cookies On This Site

              We use cookies to ensure that we give you the best experience on our website.

              Learn more

              Got it!