Abstract
Probabilistic programming languages aid developers performing Bayesian inference. These languages provide programming constructs and tools for probabilistic modeling and automated inference. Prior work introduced a probabilistic programming language, ProbZelus, to extend probabilistic programming functionality to unbounded streams of data. This work demonstrated that the delayed sampling inference algorithm could be extended to work in a streaming context. ProbZelus showed that while delayed sampling could be effectively deployed on some programs, depending on the probabilistic model under consideration, delayed sampling is not guaranteed to use a bounded amount of memory over the course of the execution of the program.
In this paper, we the present conditions on a probabilistic program’s execution under which delayed sampling will execute in bounded memory. The two conditions are dataflow properties of the core operations of delayed sampling: the m-consumed property and the unseparated paths property. A program executes in bounded memory under delayed sampling if, and only if, it satisfies the m-consumed and unseparated paths properties. We propose a static analysis that abstracts over these properties to soundly ensure that any program that passes the analysis satisfies these properties, and thus executes in bounded memory under delayed sampling.
Supplemental Material
Available for Download
Appendices to the paper.
- Eric Atkinson, Guillaume Baudart, Louis Mandel, Charles Yuan, and Michael Carbin. 2021. Statically Bounded-Memory Delayed Sampling for Probabilistic Streams. In arXiv e-prints. arxiv:2109.12473Google Scholar
- Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. In arXiv e-prints.Google Scholar
- Guillaume Baudart, Louis Mandel, Eric Atkinson, Benjamin Sherman, Marc Pouzet, and Michael Carbin. 2020. Reactive Probabilistic Programming. In Conference on Programming Language Design and Implementation.Google Scholar
- Leonard E. Baum and Ted Petrie. 1966. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37, 6 (1966).Google Scholar
Cross Ref
- Atilim Güneş Baydin, Lei Shao, Wahid Bhimji, Lukas Heinrich, Lawrence Meadows, Jialin Liu, Andreas Munk, Saeid Naderiparizi, Bradley Gram-Hansen, Gilles Louppe, Mingfei Ma, Xiaohui Zhao, Philip Torr, Victor Lee, Kyle Cranmer, Prabhat, and Frank Wood. 2019. Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’19).Google Scholar
Digital Library
- Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 20, 28 (2019).Google Scholar
- Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, and Michael I. Jordan. 2013. Streaming Variational Bayes. In International Conference on Neural Information Processing Systems.Google Scholar
- Jean-Louis Colaço, Bruno Pagano, and Marc Pouzet. 2017. SCADE 6: A formal language for embedded critical software development (invited paper). In TASE. IEEE Computer Society, 1–11.Google Scholar
- Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019. Gen: a General-purpose Probabilistic Programming System with Programmable Inference. In Conference on Programming Language Design and Implementation.Google Scholar
Digital Library
- Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. 2006. Sequential Monte Carlo samplers. J. Royal Statistical Society: Series B (Statistical Methodology), 68, 3 (2006), 411–436.Google Scholar
Cross Ref
- Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In UAI.Google Scholar
Digital Library
- Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In Conference on Uncertainty in Artificial Intelligence.Google Scholar
Digital Library
- Daniel Fink. 1997. A Compendium of Conjugate Priors.Google Scholar
- Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: Composable inference for probabilistic programming. In International Conference on Artificial Intelligence and Statistics.Google Scholar
- Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics, 40, 5 (2015), 530–543.Google Scholar
Cross Ref
- Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A language for generative models. In Conference on Uncertainty in Artificial Intelligence.Google Scholar
- Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org Accessed: 2020-10-30.Google Scholar
- Andrew D. Gordon, Thore Graepel, Nicolas Rolland, Claudio Russo, Johannes Borgstrom, and John Guiver. 2014. Tabular: a schema-driven probabilistic programming language. In Symposium on Principles of Programming Languages.Google Scholar
Digital Library
- Vineet Gupta, Radha Jagadeesan, and Vijay A. Saraswat. 1997. Probabilistic Concurrent Constraint Programming. In CONCUR (Lecture Notes in Computer Science, Vol. 1243). Springer, 243–257.Google Scholar
- Daniel Huang, Jean-Baptiste Tristan, and Greg Morisett. 2017. Compiling Markov Chain Monte Carlo Algorithms for Probabilistic Modeling. In Conference on Programming Language Design and Implementation.Google Scholar
- R. E. Kalman. 1960. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82, 1 (1960).Google Scholar
Cross Ref
- Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models - Principles and Techniques. MIT Press.Google Scholar
- Jun S. Liu and Rong Chen. 1998. Sequential Monte Carlo Methods for Dynamic Systems. J. Amer. Statist. Assoc., 93, 443 (1998), 1032–1044.Google Scholar
Cross Ref
- Daniel Lundén. 2017. Delayed sampling in the probabilistic programming language Anglican. Master’s thesis. KTH Royal Institute of Technology.Google Scholar
- Vikash Mansingkha, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Conference on Programming Language Design and Implementation.Google Scholar
- George H. Mealy. 1955. A method for synthesizing sequential circuits. The Bell System Technical Journal, 34, 5 (1955), 1045–1079.Google Scholar
Cross Ref
- Brian Milch, Bhaskara Marthi, Stuart Russell, David Sontag, Daniel L. Ong, and Andrey Kolobov. 2007. BLOG: Probabilistic models with unknown objects. Statistical relational learning.Google Scholar
- Thomas P. Minka. 2001. Expectation Propagation for Approximate Bayesian Inference. In Conference in Uncertainty in Artificial Intelligence.Google Scholar
- Lawrence M. Murray, Daniel Lundén, Jan Kudlicka, David Broman, and Thomas B. Schön. 2018. Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. In International Conference on Artificial Intelligence and Statistics.Google Scholar
- Lawrence M. Murray and Thomas B. Schön. 2018. Automated learning with a probabilistic programming language: Birch. Annual Reviews in Control, 46 (2018).Google Scholar
- Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming.Google Scholar
Cross Ref
- Van Chan Ngo, Quentin Carbonneaux, and Jan Hoffmann. 2018. Bounded Expectations: Resource Analysis for Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google Scholar
- Aditya V. Nori, Sherjil Ozair, Sriram K. Rajamani, and Deepak Vijaykeerthy. 2015. Efficient Synthesis of Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google Scholar
- Avi Pfeffer. 2009. Figaro: An object-oriented probabilistic programming language. 137, 96.Google Scholar
- Eduardo D Sontag. 2013. Mathematical control theory: deterministic finite dimensional systems. 6, Springer Science & Business Media.Google Scholar
- Sam Staton. 2017. Commutative Semantics for Probabilistic Programming. In European Symposium on Programming.Google Scholar
- Dustin Tran, Matthew D Hoffman, Rif A Saurous, Eugene Brevdo, Kevin Murphy, and David M Blei. 2017. Deep probabilistic programming. In International Conference on Learning Representations.Google Scholar
Index Terms
Statically bounded-memory delayed sampling for probabilistic streams
Recommendations
Reactive probabilistic programming
PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and ImplementationSynchronous modeling is at the heart of programming languages like Lustre, Esterel, or Scade used routinely for implementing safety critical control software, e.g., fly-by-wire and engine control in planes. However, to date these languages have had ...
Semi-symbolic inference for efficient streaming probabilistic programming
A streaming probabilistic program receives a stream of observations and produces a stream of distributions that are conditioned on these observations. Efficient inference is often possible in a streaming context using Rao-Blackwellized particle ...
Probabilistic abductive logic programming using Dirichlet priors
Probabilistic programming is an area of research that aims to develop general inference algorithms for probabilistic models expressed as probabilistic programs whose execution corresponds to inferring the parameters of those models. In this paper, we ...






Comments