skip to main content

Semi-symbolic inference for efficient streaming probabilistic programming

Published:31 October 2022Publication History
Skip Abstract Section

Abstract

A streaming probabilistic program receives a stream of observations and produces a stream of distributions that are conditioned on these observations. Efficient inference is often possible in a streaming context using Rao-Blackwellized particle filters (RBPFs), which exactly solve inference problems when possible and fall back on sampling approximations when necessary. While RBPFs can be implemented by hand to provide efficient inference, the goal of streaming probabilistic programming is to automatically generate such efficient inference implementations given input probabilistic programs.

In this work, we propose semi-symbolic inference, a technique for executing probabilistic programs using a runtime inference system that automatically implements Rao-Blackwellized particle filtering. To perform exact and approximate inference together, the semi-symbolic inference system manipulates symbolic distributions to perform exact inference when possible and falls back on approximate sampling when necessary. This approach enables the system to implement the same RBPF a developer would write by hand. To ensure this, we identify closed families of distributions – such as linear-Gaussian and finite discrete models – on which the inference system guarantees exact inference. We have implemented the runtime inference system in the ProbZelus streaming probabilistic programming language. Despite an average 1.6× slowdown compared to the state of the art on existing benchmarks, our evaluation shows that speedups of 3×-87× are obtainable on a new set of challenging benchmarks we have designed to exploit closed families.

References

  1. Eric Atkinson, Charles Yuan, Guillaume Baudart, Louis Mandel, and Michael Carbin. 2022. Semi-Symbolic Inference for Efficient Streaming Probabilistic Programming. https://doi.org/10.5281/zenodo.7082520 Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Eric Atkinson, Charles Yuan, Guillaume Baudart, Louis Mandel, and Michael Carbin. 2022. Semi-Symbolic Inference for Efficient Streaming Probabilistic Programming. arxiv:2209.07490 Google ScholarGoogle Scholar
  3. Guillaume Baudart, Louis Mandel, Eric Atkinson, Benjamin Sherman, Marc Pouzet, and Michael Carbin. 2020. Reactive probabilistic programming. In PLDI. ACM, 898–912. https://doi.org/10.1145/3385412.3386009 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Keni Bernardin and Rainer Stiefelhagen. 2008. Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. EURASIP J. Image and Video Processing, 2008 (2008), https://doi.org/10.1155/2008/246309 Google ScholarGoogle ScholarCross RefCross Ref
  5. Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res., 20 (2019), 28:1–28:6. http://jmlr.org/papers/v20/18-403.html Google ScholarGoogle Scholar
  6. Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In UAI. Morgan Kaufmann, 176–183. https://doi.org/10.1007/978-1-4757-3437-9_24 Google ScholarGoogle ScholarCross RefCross Ref
  7. Daniel Fink. 1997. A Compendium of Conjugate Priors. Google ScholarGoogle Scholar
  8. Timon Gehr, Sasa Misailovic, and Martin T. Vechev. 2016. PSI: Exact Symbolic Inference for Probabilistic Programs. In CAV (1) (Lecture Notes in Computer Science, Vol. 9779). Springer, 62–83. https://doi.org/10.1007/978-3-319-41528-4_4 Google ScholarGoogle ScholarCross RefCross Ref
  9. Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 6 (1984), https://doi.org/10.1109/TPAMI.1984.4767596 Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Noah D. Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org Accessed October 2022 Google ScholarGoogle Scholar
  11. N.J. Gordon, D.J. Salmond, and A.F.M. Smith. 1993. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In IEE Proceedings-F. https://doi.org/10.1049/ip-f-2.1993.0015 Google ScholarGoogle ScholarCross RefCross Ref
  12. Matthew D Hoffman, Matthew J Johnson, and Dustin Tran. 2018. Autoconj: Recognizing and Exploiting Conjugacy Without a Domain-specific Language. In NeurIPS. https://proceedings.neurips.cc/paper/2018/file/9b89bedda1fc8a2d88c448e361194f02-Paper.pdf Google ScholarGoogle Scholar
  13. Steven Holtzen, Guy Van den Broeck, and Todd Millstein. 2020. Scaling Exact Inference for Discrete Probabilistic Programs. In OOPSLA. https://doi.org/10.1145/3428208 Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. T.D. Larsen, K.L. Hansen, N.A. Andersen, and Ole Ravn. 1999. Design of Kalman filters for mobile robots; evaluation of the kinematic and odometric approach. In Proceedings of the IEEE International Conference on Control Applications. https://doi.org/10.1109/CCA.1999.801027 Google ScholarGoogle ScholarCross RefCross Ref
  15. Daniel Lundén. 2017. Delayed sampling in the probabilistic programming language Anglican. Master’s thesis. KTH Royal Institute of Technology. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-210756 Google ScholarGoogle Scholar
  16. Vikash K. Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic programming with programmable inference. In PLDI. https://doi.org/10.1145/3192366.3192409 Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. T. Minka, J.M. Winn, J.P. Guiver, Y. Zaykov, D. Fabian, and J. Bronskill. 2018. Infer.NET 0.3. http://dotnet.github.io/infer Microsoft Research Cambridge. Google ScholarGoogle Scholar
  18. Thomas P. Minka. 2001. Expectation Propagation for approximate Bayesian inference. In UAI. Morgan Kaufmann, 362–369. isbn:1558608001 Google ScholarGoogle Scholar
  19. Lawrence M. Murray, Daniel Lundén, Jan Kudlicka, David Broman, and Thomas B. Schön. 2018. Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. In AISTATS (Proceedings of Machine Learning Research, Vol. 84). PMLR, 1037–1046. https://proceedings.mlr.press/v84/murray18a.html Google ScholarGoogle Scholar
  20. Lawrence M. Murray and Thomas B. Schön. 2018. Automated learning with a probabilistic programming language: Birch. Annual Reviews in Control, 46 (2018), 29–43. Google ScholarGoogle ScholarCross RefCross Ref
  21. Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru (System Description). In FLOPS (Lecture Notes in Computer Science, Vol. 9613). Springer, 62–79. https://doi.org/10.1007/978-3-319-29604-3_5 Google ScholarGoogle ScholarCross RefCross Ref
  22. Fritz Obermeyer, Eli Bingham, Martin Jankowiak, Du Phan, and Jonathan Chen. 2019. Functional Tensors for Probabilistic Programming. In Program Transformations for ML Workshop at NeurIPS. Google ScholarGoogle Scholar
  23. Fritz Obermeyer, Elias Bingham, Martin Jankowiak, Neeraj Pradhan, Justin Chiu, Alexander Rush, and Noah Goodman. 2019. Tensor Variable Elimination for Plated Factor Graphs. In ICML. https://proceedings.mlr.press/v97/obermeyer19a.html Google ScholarGoogle Scholar
  24. Judea Pearl. 1982. Reverend Bayes on Inference Engines: A Distributed Hierarchical Approach. In AAAI. Google ScholarGoogle Scholar
  25. Feras Saad, Martin Rinard, and Vikash Mansinghka. 2021. SPPL: Probabilistic Programming with Fast Exact Symbolic Inference. In PLDI. https://doi.org/10.1145/3453483.3454078 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. P. D. Solomon, Jinling Wang, and Chris Rizos. 2012. Latency Determination and Compensation in Real-Time Gnss/ins Integrated Navigation Systems. ISPRS, 3822 (2012), 303–307. Google ScholarGoogle Scholar
  27. Eduardo D Sontag. 2013. Mathematical control theory: deterministic finite dimensional systems. 6, Springer Science & Business Media. https://doi.org/10.1007/978-1-4612-0577-7 Google ScholarGoogle ScholarCross RefCross Ref
  28. David Tolpin, Jan-Willem van de Meent, Hongseok Yang, and Frank D. Wood. 2016. Design and Implementation of Probabilistic Programming Language Anglican. In IFL. ACM, 6:1–6:12. https://doi.org/10.1145/3064899.3064910 Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Programming. In ICLR (Poster). OpenReview.net. https://openreview.net/forum?id=Hy6b4Pqee Google ScholarGoogle Scholar
  30. John Winn and Christopher M. Bishop. 2005. Variational Message Passing. Journal of Machine Learning Research, 6, 4 (2005), https://www.jmlr.org/papers/v6/winn05a.html Google ScholarGoogle Scholar
  31. N.L. Zhang and D. Poole. 1994. A Simple Approach to Bayesian Network Computations. In Canadian Conference on Artificial Intelligence. Google ScholarGoogle Scholar

Index Terms

  1. Semi-symbolic inference for efficient streaming probabilistic programming

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!