Abstract
A streaming probabilistic program receives a stream of observations and produces a stream of distributions that are conditioned on these observations. Efficient inference is often possible in a streaming context using Rao-Blackwellized particle filters (RBPFs), which exactly solve inference problems when possible and fall back on sampling approximations when necessary. While RBPFs can be implemented by hand to provide efficient inference, the goal of streaming probabilistic programming is to automatically generate such efficient inference implementations given input probabilistic programs.
In this work, we propose semi-symbolic inference, a technique for executing probabilistic programs using a runtime inference system that automatically implements Rao-Blackwellized particle filtering. To perform exact and approximate inference together, the semi-symbolic inference system manipulates symbolic distributions to perform exact inference when possible and falls back on approximate sampling when necessary. This approach enables the system to implement the same RBPF a developer would write by hand. To ensure this, we identify closed families of distributions – such as linear-Gaussian and finite discrete models – on which the inference system guarantees exact inference. We have implemented the runtime inference system in the ProbZelus streaming probabilistic programming language. Despite an average 1.6× slowdown compared to the state of the art on existing benchmarks, our evaluation shows that speedups of 3×-87× are obtainable on a new set of challenging benchmarks we have designed to exploit closed families.
- Eric Atkinson, Charles Yuan, Guillaume Baudart, Louis Mandel, and Michael Carbin. 2022. Semi-Symbolic Inference for Efficient Streaming Probabilistic Programming. https://doi.org/10.5281/zenodo.7082520
Google Scholar
Digital Library
- Eric Atkinson, Charles Yuan, Guillaume Baudart, Louis Mandel, and Michael Carbin. 2022. Semi-Symbolic Inference for Efficient Streaming Probabilistic Programming. arxiv:2209.07490
Google Scholar
- Guillaume Baudart, Louis Mandel, Eric Atkinson, Benjamin Sherman, Marc Pouzet, and Michael Carbin. 2020. Reactive probabilistic programming. In PLDI. ACM, 898–912. https://doi.org/10.1145/3385412.3386009
Google Scholar
Digital Library
- Keni Bernardin and Rainer Stiefelhagen. 2008. Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. EURASIP J. Image and Video Processing, 2008 (2008), https://doi.org/10.1155/2008/246309
Google Scholar
Cross Ref
- Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res., 20 (2019), 28:1–28:6. http://jmlr.org/papers/v20/18-403.html
Google Scholar
- Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In UAI. Morgan Kaufmann, 176–183. https://doi.org/10.1007/978-1-4757-3437-9_24
Google Scholar
Cross Ref
- Daniel Fink. 1997. A Compendium of Conjugate Priors.
Google Scholar
- Timon Gehr, Sasa Misailovic, and Martin T. Vechev. 2016. PSI: Exact Symbolic Inference for Probabilistic Programs. In CAV (1) (Lecture Notes in Computer Science, Vol. 9779). Springer, 62–83. https://doi.org/10.1007/978-3-319-41528-4_4
Google Scholar
Cross Ref
- Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 6 (1984), https://doi.org/10.1109/TPAMI.1984.4767596
Google Scholar
Digital Library
- Noah D. Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org Accessed October 2022
Google Scholar
- N.J. Gordon, D.J. Salmond, and A.F.M. Smith. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE Proceedings-F. https://doi.org/10.1049/ip-f-2.1993.0015
Google Scholar
Cross Ref
- Matthew D Hoffman, Matthew J Johnson, and Dustin Tran. 2018. Autoconj: Recognizing and Exploiting Conjugacy Without a Domain-specific Language. In NeurIPS. https://proceedings.neurips.cc/paper/2018/file/9b89bedda1fc8a2d88c448e361194f02-Paper.pdf
Google Scholar
- Steven Holtzen, Guy Van den Broeck, and Todd Millstein. 2020. Scaling Exact Inference for Discrete Probabilistic Programs. In OOPSLA. https://doi.org/10.1145/3428208
Google Scholar
Digital Library
- T.D. Larsen, K.L. Hansen, N.A. Andersen, and Ole Ravn. 1999. Design of Kalman filters for mobile robots; evaluation of the kinematic and odometric approach. In Proceedings of the IEEE International Conference on Control Applications. https://doi.org/10.1109/CCA.1999.801027
Google Scholar
Cross Ref
- Daniel Lundén. 2017. Delayed sampling in the probabilistic programming language Anglican. Master’s thesis. KTH Royal Institute of Technology. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-210756
Google Scholar
- Vikash K. Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic programming with programmable inference. In PLDI. https://doi.org/10.1145/3192366.3192409
Google Scholar
Digital Library
- T. Minka, J.M. Winn, J.P. Guiver, Y. Zaykov, D. Fabian, and J. Bronskill. 2018. Infer.NET 0.3. http://dotnet.github.io/infer Microsoft Research Cambridge.
Google Scholar
- Thomas P. Minka. 2001. Expectation Propagation for approximate Bayesian inference. In UAI. Morgan Kaufmann, 362–369. isbn:1558608001
Google Scholar
- Lawrence M. Murray, Daniel Lundén, Jan Kudlicka, David Broman, and Thomas B. Schön. 2018. Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. In AISTATS (Proceedings of Machine Learning Research, Vol. 84). PMLR, 1037–1046. https://proceedings.mlr.press/v84/murray18a.html
Google Scholar
- Lawrence M. Murray and Thomas B. Schön. 2018. Automated learning with a probabilistic programming language: Birch. Annual Reviews in Control, 46 (2018), 29–43.
Google Scholar
Cross Ref
- Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru (System Description). In FLOPS (Lecture Notes in Computer Science, Vol. 9613). Springer, 62–79. https://doi.org/10.1007/978-3-319-29604-3_5
Google Scholar
Cross Ref
- Fritz Obermeyer, Eli Bingham, Martin Jankowiak, Du Phan, and Jonathan Chen. 2019. Functional Tensors for Probabilistic Programming. In Program Transformations for ML Workshop at NeurIPS.
Google Scholar
- Fritz Obermeyer, Elias Bingham, Martin Jankowiak, Neeraj Pradhan, Justin Chiu, Alexander Rush, and Noah Goodman. 2019. Tensor Variable Elimination for Plated Factor Graphs. In ICML. https://proceedings.mlr.press/v97/obermeyer19a.html
Google Scholar
- Judea Pearl. 1982. Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach. In AAAI.
Google Scholar
- Feras Saad, Martin Rinard, and Vikash Mansinghka. 2021. SPPL: Probabilistic Programming with Fast Exact Symbolic Inference. In PLDI. https://doi.org/10.1145/3453483.3454078
Google Scholar
Digital Library
- P. D. Solomon, Jinling Wang, and Chris Rizos. 2012. Latency Determination and Compensation in Real-Time Gnss/ins Integrated Navigation Systems. ISPRS, 3822 (2012), 303–307.
Google Scholar
- Eduardo D Sontag. 2013. Mathematical control theory: deterministic finite dimensional systems. 6, Springer Science & Business Media. https://doi.org/10.1007/978-1-4612-0577-7
Google Scholar
Cross Ref
- David Tolpin, Jan-Willem van de Meent, Hongseok Yang, and Frank D. Wood. 2016. Design and Implementation of Probabilistic Programming Language Anglican. In IFL. ACM, 6:1–6:12. https://doi.org/10.1145/3064899.3064910
Google Scholar
Digital Library
- Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Programming. In ICLR (Poster). OpenReview.net. https://openreview.net/forum?id=Hy6b4Pqee
Google Scholar
- John Winn and Christopher M. Bishop. 2005. Variational Message Passing. Journal of Machine Learning Research, 6, 4 (2005), https://www.jmlr.org/papers/v6/winn05a.html
Google Scholar
- N.L. Zhang and D. Poole. 1994. A Simple Approach to Bayesian Network Computations. In Canadian Conference on Artificial Intelligence.
Google Scholar
Index Terms
Semi-symbolic inference for efficient streaming probabilistic programming
Recommendations
Probabilistic abductive logic programming using Dirichlet priors
Probabilistic programming is an area of research that aims to develop general inference algorithms for probabilistic models expressed as probabilistic programs whose execution corresponds to inferring the parameters of those models. In this paper, we ...
Reactive probabilistic programming
PLDI 2020: Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and ImplementationSynchronous modeling is at the heart of programming languages like Lustre, Esterel, or Scade used routinely for implementing safety critical control software, e.g., fly-by-wire and engine control in planes. However, to date these languages have had ...
JAX based parallel inference for reactive probabilistic programming
LCTES 2022: Proceedings of the 23rd ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, and Tools for Embedded SystemsProbZelus is a synchronous probabilistic language for the design of reactive probabilistic models in interaction with an environment. Reactive inference methods continuously learn distributions over the unobserved parameters of the model from ...






Comments