skip to main content

Statically bounded-memory delayed sampling for probabilistic streams

Published:15 October 2021Publication History
Skip Abstract Section

Abstract

Probabilistic programming languages aid developers performing Bayesian inference. These languages provide programming constructs and tools for probabilistic modeling and automated inference. Prior work introduced a probabilistic programming language, ProbZelus, to extend probabilistic programming functionality to unbounded streams of data. This work demonstrated that the delayed sampling inference algorithm could be extended to work in a streaming context. ProbZelus showed that while delayed sampling could be effectively deployed on some programs, depending on the probabilistic model under consideration, delayed sampling is not guaranteed to use a bounded amount of memory over the course of the execution of the program.

In this paper, we the present conditions on a probabilistic program’s execution under which delayed sampling will execute in bounded memory. The two conditions are dataflow properties of the core operations of delayed sampling: the m-consumed property and the unseparated paths property. A program executes in bounded memory under delayed sampling if, and only if, it satisfies the m-consumed and unseparated paths properties. We propose a static analysis that abstracts over these properties to soundly ensure that any program that passes the analysis satisfies these properties, and thus executes in bounded memory under delayed sampling.

Skip Supplemental Material Section

Supplemental Material

Auxiliary Presentation Video

Video of presentation.

References

  1. Eric Atkinson, Guillaume Baudart, Louis Mandel, Charles Yuan, and Michael Carbin. 2021. Statically Bounded-Memory Delayed Sampling for Probabilistic Streams. In arXiv e-prints. arxiv:2109.12473Google ScholarGoogle Scholar
  2. Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. In arXiv e-prints.Google ScholarGoogle Scholar
  3. Guillaume Baudart, Louis Mandel, Eric Atkinson, Benjamin Sherman, Marc Pouzet, and Michael Carbin. 2020. Reactive Probabilistic Programming. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  4. Leonard E. Baum and Ted Petrie. 1966. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37, 6 (1966).Google ScholarGoogle ScholarCross RefCross Ref
  5. Atilim Güneş Baydin, Lei Shao, Wahid Bhimji, Lukas Heinrich, Lawrence Meadows, Jialin Liu, Andreas Munk, Saeid Naderiparizi, Bradley Gram-Hansen, Gilles Louppe, Mingfei Ma, Xiaohui Zhao, Philip Torr, Victor Lee, Kyle Cranmer, Prabhat, and Frank Wood. 2019. Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’19).Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 20, 28 (2019).Google ScholarGoogle Scholar
  7. Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, and Michael I. Jordan. 2013. Streaming Variational Bayes. In International Conference on Neural Information Processing Systems.Google ScholarGoogle Scholar
  8. Jean-Louis Colaço, Bruno Pagano, and Marc Pouzet. 2017. SCADE 6: A formal language for embedded critical software development (invited paper). In TASE. IEEE Computer Society, 1–11.Google ScholarGoogle Scholar
  9. Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019. Gen: a General-purpose Probabilistic Programming System with Programmable Inference. In Conference on Programming Language Design and Implementation.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. 2006. Sequential Monte Carlo samplers. J. Royal Statistical Society: Series B (Statistical Methodology), 68, 3 (2006), 411–436.Google ScholarGoogle ScholarCross RefCross Ref
  11. Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In UAI.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Arnaud Doucet, Nando de Freitas, Kevin P. Murphy, and Stuart J. Russell. 2000. Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks. In Conference on Uncertainty in Artificial Intelligence.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Daniel Fink. 1997. A Compendium of Conjugate Priors.Google ScholarGoogle Scholar
  14. Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: Composable inference for probabilistic programming. In International Conference on Artificial Intelligence and Statistics.Google ScholarGoogle Scholar
  15. Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics, 40, 5 (2015), 530–543.Google ScholarGoogle ScholarCross RefCross Ref
  16. Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A language for generative models. In Conference on Uncertainty in Artificial Intelligence.Google ScholarGoogle Scholar
  17. Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org Accessed: 2020-10-30.Google ScholarGoogle Scholar
  18. Andrew D. Gordon, Thore Graepel, Nicolas Rolland, Claudio Russo, Johannes Borgstrom, and John Guiver. 2014. Tabular: a schema-driven probabilistic programming language. In Symposium on Principles of Programming Languages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Vineet Gupta, Radha Jagadeesan, and Vijay A. Saraswat. 1997. Probabilistic Concurrent Constraint Programming. In CONCUR (Lecture Notes in Computer Science, Vol. 1243). Springer, 243–257.Google ScholarGoogle Scholar
  20. Daniel Huang, Jean-Baptiste Tristan, and Greg Morisett. 2017. Compiling Markov Chain Monte Carlo Algorithms for Probabilistic Modeling. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  21. R. E. Kalman. 1960. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82, 1 (1960).Google ScholarGoogle ScholarCross RefCross Ref
  22. Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models - Principles and Techniques. MIT Press.Google ScholarGoogle Scholar
  23. Jun S. Liu and Rong Chen. 1998. Sequential Monte Carlo Methods for Dynamic Systems. J. Amer. Statist. Assoc., 93, 443 (1998), 1032–1044.Google ScholarGoogle ScholarCross RefCross Ref
  24. Daniel Lundén. 2017. Delayed sampling in the probabilistic programming language Anglican. Master’s thesis. KTH Royal Institute of Technology.Google ScholarGoogle Scholar
  25. Vikash Mansingkha, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  26. George H. Mealy. 1955. A method for synthesizing sequential circuits. The Bell System Technical Journal, 34, 5 (1955), 1045–1079.Google ScholarGoogle ScholarCross RefCross Ref
  27. Brian Milch, Bhaskara Marthi, Stuart Russell, David Sontag, Daniel L. Ong, and Andrey Kolobov. 2007. BLOG: Probabilistic models with unknown objects. Statistical relational learning.Google ScholarGoogle Scholar
  28. Thomas P. Minka. 2001. Expectation Propagation for Approximate Bayesian Inference. In Conference in Uncertainty in Artificial Intelligence.Google ScholarGoogle Scholar
  29. Lawrence M. Murray, Daniel Lundén, Jan Kudlicka, David Broman, and Thomas B. Schön. 2018. Delayed Sampling and Automatic Rao-Blackwellization of Probabilistic Programs. In International Conference on Artificial Intelligence and Statistics.Google ScholarGoogle Scholar
  30. Lawrence M. Murray and Thomas B. Schön. 2018. Automated learning with a probabilistic programming language: Birch. Annual Reviews in Control, 46 (2018).Google ScholarGoogle Scholar
  31. Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic inference by program transformation in Hakaru (system description). In International Symposium on Functional and Logic Programming.Google ScholarGoogle ScholarCross RefCross Ref
  32. Van Chan Ngo, Quentin Carbonneaux, and Jan Hoffmann. 2018. Bounded Expectations: Resource Analysis for Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  33. Aditya V. Nori, Sherjil Ozair, Sriram K. Rajamani, and Deepak Vijaykeerthy. 2015. Efficient Synthesis of Probabilistic Programs. In Conference on Programming Language Design and Implementation.Google ScholarGoogle Scholar
  34. Avi Pfeffer. 2009. Figaro: An object-oriented probabilistic programming language. 137, 96.Google ScholarGoogle Scholar
  35. Eduardo D Sontag. 2013. Mathematical control theory: deterministic finite dimensional systems. 6, Springer Science & Business Media.Google ScholarGoogle Scholar
  36. Sam Staton. 2017. Commutative Semantics for Probabilistic Programming. In European Symposium on Programming.Google ScholarGoogle Scholar
  37. Dustin Tran, Matthew D Hoffman, Rif A Saurous, Eugene Brevdo, Kevin Murphy, and David M Blei. 2017. Deep probabilistic programming. In International Conference on Learning Representations.Google ScholarGoogle Scholar

Index Terms

  1. Statically bounded-memory delayed sampling for probabilistic streams

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Article Metrics

          • Downloads (Last 12 months)56
          • Downloads (Last 6 weeks)3

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!