Abstract
Probabilistic programming languages (PPLs) allow programmers to construct statistical models and then simulate data or perform inference over them. Many PPLs restrict models to a particular instance of simulation or inference, limiting their reusability. In other PPLs, models are not readily composable. Using Haskell as the host language, we present an embedded domain specific language based on algebraic effects, where probabilistic models are modular, first-class, and reusable for both simulation and inference. We also demonstrate how simulation and inference can be expressed naturally as composable program transformations using algebraic effect handlers.
- I Ameen, Dumitru Baleanu, and Hegagi Mohamed Ali. 2020. An efficient algorithm for solving the fractional optimal control of SIRV epidemic model with a combination of vaccination and treatment. Chaos, Solitons & Fractals, 137 (2020), 109892. https://doi.org/10.1016/j.chaos.2020.109892
Google Scholar
Cross Ref
- Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res., 20 (2019), 28:1–28:6. https://doi.org/10.48550/arXiv.1810.09538
Google Scholar
- Bob Carpenter, Andrew Gelman, Matthew Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan : A Probabilistic Programming Language. Journal of Statistical Software, 76 (2017), https://doi.org/10.18637/jss.v076.i01
Google Scholar
Cross Ref
- Jike Chong, Ekaterina Gonina, and Kurt Keutzer. 2010. Monte Carlo Methods: A Computational Pattern for Our Pattern Language. In Proceedings of the 2010 Workshop on Parallel Programming Patterns (ParaPLoP ’10). Association for Computing Machinery, New York, NY, USA. Article 15, 10 pages. isbn:9781450301275 https://doi.org/10.1145/1953611.1953626
Google Scholar
Digital Library
- Nicolas Chopin. 2002. A sequential particle filter method for static models. Biometrika, 89, 3 (2002), 539–552. issn:0006-3444 https://doi.org/10.1093/biomet/89.3.539
Google Scholar
Cross Ref
- Marco F. Cusumano-Towner, Feras A. Saad, Alexander K. Lew, and Vikash K. Mansinghka. 2019. Gen: A General-Purpose Probabilistic Programming System with Programmable Inference. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2019). Association for Computing Machinery, New York, NY, USA. 221–236. isbn:9781450367127 https://doi.org/10.1145/3314221.3314642
Google Scholar
Digital Library
- Zhenghao Ding, Jun Li, and Hong Hao. 2019. Structural damage identification using improved Jaya algorithm based on sparse regularization and Bayesian inference. Mechanical Systems and Signal Processing, 132 (2019), 211–231. https://doi.org/10.1016/j.ymssp.2019.06.029
Google Scholar
Cross Ref
- Arnaud Doucet, Nando de Freitas, and Neil Gordon. 2001. An Introduction to Sequential Monte Carlo Methods. Springer New York, New York, NY. 3–14. isbn:978-1-4757-3437-9 https://doi.org/10.1007/978-1-4757-3437-9_1
Google Scholar
Cross Ref
- Conal Elliott, Sigbjørn Finne, and Oege De Moor. 2003. Compiling Embedded Languages. J. Funct. Program., 13, 3 (2003), 455–481. issn:0956-7968 https://doi.org/10.1017/S0956796802004574
Google Scholar
Digital Library
- Martin Erwig and Steve Kollmansberger. 2006. Functional Pearls: Probabilistic functional programming in Haskell. J. Funct. Program., 16, 1 (2006), 21–34. https://doi.org/10.1017/S0956796805005721
Google Scholar
Digital Library
- Phil Freeman. 2017. PureScript by Example. https://book.purescript.org/
Google Scholar
- Tadayoshi Fushiki. 2010. Bayesian bootstrap prediction. Journal of statistical planning and inference, 140, 1 (2010), 65–74. https://doi.org/10.1016/j.jspi.2009.06.007
Google Scholar
Cross Ref
- Hong Ge, Kai Xu, and Zoubin Ghahramani. 2018. Turing: a language for flexible probabilistic inference. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain. 1682–1690. http://proceedings.mlr.press/v84/ge18b.html
Google Scholar
- Andrew Gelman and Jennifer Hill. 2006. Data analysis using regression and multilevel/hierarchical models. Cambridge university press. https://doi.org/10.1017/CBO9780511790942
Google Scholar
Cross Ref
- Jeremy Gibbons and Ralf Hinze. 2011. Just do it: simple monadic equational reasoning. ACM SIGPLAN Notices, 46, 9 (2011), 2–14. https://doi.org/doi/10.1145/2034574.2034777
Google Scholar
Digital Library
- Jeremy Gibbons and Nicolas Wu. 2014. Folding Domain-Specific Languages: Deep and Shallow Embeddings (Functional Pearl). In Proceedings of the 19th ACM SIGPLAN International Conference on Functional Programming (ICFP ’14). Association for Computing Machinery, New York, NY, USA. 339–347. isbn:9781450328739 https://doi.org/10.1145/2628136.2628138
Google Scholar
Digital Library
- Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org Accessed: 2021-5-24
Google Scholar
- Daniel Hillerström. 2022. Foundations for programming and implementing effect handlers. Ph. D. Dissertation. The University of Edinburgh, Scotland. https://doi.org/10.7488/era/2122
Google Scholar
Cross Ref
- Daniel Hillerström and Sam Lindley. 2018. Shallow effect handlers. In Asian Symposium on Programming Languages and Systems. 415–435. https://doi.org/10.1007/978-3-030-02768-1_22
Google Scholar
Cross Ref
- Matthew D Hoffman and Andrew Gelman. 2014. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.. J. Mach. Learn. Res., 15, 1 (2014), 1593–1623. https://doi.org/10.48550/arXiv.1111.4246
Google Scholar
- Stratos Idreos, Olga Papaemmanouil, and Surajit Chaudhuri. 2015. Overview of data exploration techniques. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 277–281. https://doi.org/10.1145/2723372.2731084
Google Scholar
Digital Library
- Oleg Kiselyov. 2010. Typed Tagless Final Interpreters. In Proceedings of the 2010 International Spring School Conference on Generic and Indexed Programming (SSGIP’10). Springer-Verlag, Berlin, Heidelberg. 130–174. isbn:9783642322013 https://doi.org/10.1007/978-3-642-32202-0_3
Google Scholar
Digital Library
- Oleg Kiselyov. 2016. Probabilistic programming language and its incremental evaluation. In Programming Languages and Systems - 14th Asian Symposium, APLAS 2016, Proceedings, Atsushi Igarashi (Ed.). Springer Verlag, 357–376. isbn:9783319479576 https://doi.org/10.1007/978-3-319-47958-3_19
Google Scholar
Cross Ref
- Oleg Kiselyov. 2016. Problems of the lightweight implementation of probabilistic programming. In Proceedings of Workshop on Probabilistic Programming Semantics.
Google Scholar
- Oleg Kiselyov and Hiromi Ishii. 2015. Freer Monads, More Extensible Effects. In Proceedings of the 2015 ACM SIGPLAN Symposium on Haskell (Haskell ’15). Association for Computing Machinery, New York, NY, USA. 94–105. isbn:9781450338080 https://doi.org/10.1145/2804302.2804319
Google Scholar
Digital Library
- Oleg Kiselyov, Amr Sabry, and Cameron Swords. 2013. Extensible Effects: An Alternative to Monad Transformers. In Proceedings of the 2013 ACM SIGPLAN Symposium on Haskell (Haskell ’13). Association for Computing Machinery, New York, NY, USA. 59–70. isbn:9781450323833 https://doi.org/10.1145/2503778.2503791
Google Scholar
Digital Library
- Oleg Kiselyov and Chung-Chieh Shan. 2009. Embedded Probabilistic Programming. In Proceedings of the IFIP TC 2 Working Conference on Domain-Specific Languages (DSL ’09). Springer-Verlag, Berlin, Heidelberg. 360–384. isbn:9783642030338 https://doi.org/10.1007/978-3-642-03034-5_17
Google Scholar
Digital Library
- Brendan Kline and Elie Tamer. 2016. Bayesian inference in a class of partially identified models. Quantitative Economics, 7, 2 (2016), 329–366. https://doi.org/10.3982/QE399
Google Scholar
Cross Ref
- Edward Kmett, Barak Pearlmutter, and Jeffrey Mark Siskind. 2010-2021. ad: Automatic Differentiation. Haskell package at. https://hackage.haskell.org/package/ad
Google Scholar
- Sourabh Kulkarni, Kinjal Divesh Shah, Nimar S. Arora, Xiaoyan Wang, Yucen Lily Li, Nazanin Khosravani Tehrani, Michael Tingley, David Noursi, Narjes Torabi, Sepehr Akhavan-Masouleh, Eric Lippert, and Erik Meijer. 2020. PPL Bench: Evaluation Framework For Probabilistic Programming Languages. CoRR, abs/2010.08886 (2020), https://doi.org/10.48550/arXiv.2010.08886
Google Scholar
- Daan Leijen. 2005. Extensible records with scoped labels. In Proceedings of the 2005 Symposium on Trends in Functional Programming. 297–312. https://www.microsoft.com/en-us/research/publication/extensible-records-with-scoped-labels/
Google Scholar
- Xavier Leroy, Damien Doligez, Alain Frisch, Jacques Garrigue, Didier Rémy, and Jérôme Vouillon. 2020. The OCaml system release 4.11: Documentation and user’s manual. https://v2.ocaml.org/releases/4.11/htmlman/index.html
Google Scholar
- Benjamin Letham, Lydia M. Letham, and Cynthia Rudin. 2016. Bayesian Inference of Arrival Rate and Substitution Behavior from Sales Transaction Data with Stockouts. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). Association for Computing Machinery, New York, NY, USA. 1695–1704. isbn:9781450342322 https://doi.org/10.1145/2939672.2939810
Google Scholar
Digital Library
- Alexander K Lew, Marco F Cusumano-Towner, Benjamin Sherman, Michael Carbin, and Vikash K Mansinghka. 2019. Trace types and denotational semantics for sound programmable inference in probabilistic languages. Proceedings of the ACM on Programming Languages, 4, POPL (2019), 1–32. https://doi.org/10.1145/3371087
Google Scholar
Digital Library
- Feng Liang and Ziyuan Li. 2021. Statistical Analysis on COVID-19 Based on SIR Model. In 2020 4th International Conference on Computational Biology and Bioinformatics (ICCBB 2020). Association for Computing Machinery, New York, NY, USA. 14–19. isbn:9781450388443 https://doi.org/10.1145/3449258.3449261
Google Scholar
Digital Library
- Sheng Liang, Paul Hudak, and Mark Jones. 1995. Monad Transformers and Modular Interpreters. In Proceedings of the 22nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’95). Association for Computing Machinery, New York, NY, USA. 333–343. isbn:0897916921 https://doi.org/10.1145/199448.199528
Google Scholar
Digital Library
- David J. Lunn, Andrew Thomas, Nicky Best, and David Spiegelhalter. 2000. WinBUGS – A Bayesian Modelling Framework: Concepts, Structure, and Extensibility. Statistics and Computing, 10, 4 (2000), 325–337. issn:0960-3174 https://doi.org/10.1023/A:1008929526011
Google Scholar
Digital Library
- Sandy Maguire. 2019. Polysemy. https://github.com/polysemy-research/polysemy
Google Scholar
- Todd K Moon. 1996. The expectation-maximization algorithm. IEEE Signal processing magazine, 13, 6 (1996), 47–60. https://doi.org/10.1109/79.543975
Google Scholar
Cross Ref
- Dave Moore and Maria I Gorinova. 2018. Effect handling for composable program transformations in edward2. arXiv preprint arXiv:1811.06150, https://doi.org/10.48550/arXiv.1811.06150
Google Scholar
- Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru. In International Symposium on Functional and Logic Programming. 62–79. isbn:978-3-319-29603-6 https://doi.org/10.1007/978-3-319-29604-3_5
Google Scholar
Cross Ref
- Minh Nguyen, Roly Perera, Meng Wang, and Nicolas Wu. 2022. Reproduction Package for Article: Modular Probabilistic Models via Algebraic Effects. https://doi.org/10.5281/zenodo.6651953
Google Scholar
Digital Library
- Gordon Plotkin and John Power. 2003. Algebraic operations and generic effects. Applied categorical structures, 11, 1 (2003), 69–94. https://doi.org/10.1023/A:1023064908962
Google Scholar
Cross Ref
- Gordon Plotkin and Matija Pretnar. 2013. Handling Algebraic Effects. Logical Methods in Computer Science, 9, 4 (2013), issn:1860-5974 https://doi.org/10.2168/lmcs-9(4:23)2013
Google Scholar
Cross Ref
- Nicholas G Polson, James G Scott, and Jesse Windle. 2013. Bayesian inference for logistic models using Pólya–Gamma latent variables. Journal of the American statistical Association, 108, 504 (2013), 1339–1349. https://doi.org/10.1080/01621459.2013.829001
Google Scholar
Cross Ref
- L. Rabiner and B. Juang. 1986. An introduction to hidden Markov models. IEEE ASSP Magazine, 3, 1 (1986), 4–16. https://doi.org/10.1109/MASSP.1986.1165342
Google Scholar
Cross Ref
- Christian P Robert and DM Titterington. 1998. Reparameterization strategies for hidden Markov models and Bayesian approaches to maximum likelihood estimation. Statistics and Computing, 8, 2 (1998), 145–158. https://doi.org/10.1023/A:1008938201645
Google Scholar
Digital Library
- John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2 (2016), e55. https://doi.org/10.7717/peerj-cs.55
Google Scholar
Cross Ref
- Tom Schrijvers, Maciej Piróg, Nicolas Wu, and Mauro Jaskelioff. 2019. Monad Transformers and Modular Algebraic Effects: What Binds Them Together. In Proceedings of the 12th ACM SIGPLAN International Symposium on Haskell (Haskell 2019). Association for Computing Machinery, New York, NY, USA. 98–113. isbn:9781450368131 https://doi.org/10.1145/3331545.3342595
Google Scholar
Digital Library
- Adam Ścibior, Zoubin Ghahramani, and Andrew D. Gordon. 2015. Practical Probabilistic Programming with Monads. In Proceedings of the 2015 ACM SIGPLAN Symposium on Haskell (Haskell ’15). Association for Computing Machinery, New York, NY, USA. 165–176. isbn:9781450338080 https://doi.org/10.1145/2804302.2804317
Google Scholar
Digital Library
- Adam Ścibior and Ohad Kammar. 2015. Effects in Bayesian inference. In Workshop on Higher-Order Programming with Effects (HOPE). https://www.cs.ubc.ca/~ascibior/assets/pdf/hope.pdf
Google Scholar
- Adam Ścibior, Ohad Kammar, and Zoubin Ghahramani. 2018. Functional Programming for Modular Bayesian Inference. Proc. ACM Program. Lang., 2, ICFP (2018), Article 83, 29 pages. https://doi.org/10.1145/3236778
Google Scholar
Digital Library
- Hongjing Shi, Zhisheng Duan, and Guanrong Chen. 2008. An SIS model with infective medium on complex networks. Physica A: Statistical Mechanics and its Applications, 387, 8-9 (2008), 2133–2144. https://doi.org/10.1016/j.physa.2007.11.048
Google Scholar
Cross Ref
- Jesse Sigal. 2021. Automatic differentiation via effects and handlers: An implementation in Frank. arXiv preprint arXiv:2101.08095, https://doi.org/10.48550/arXiv.2101.08095
Google Scholar
- Wouter Swierstra. 2008. Data types à La Carte. J. Funct. Program., 18, 4 (2008), 423–436. issn:0956-7968 https://doi.org/10.1017/S0956796808006758
Google Scholar
Digital Library
- David Tolpin, Jan-Willem van de Meent, Hongseok Yang, and Frank Wood. 2016. Design and Implementation of Probabilistic Programming Language Anglican. In Proceedings of the 28th Symposium on the Implementation and Application of Functional Programming Languages (IFL 2016). Association for Computing Machinery, New York, NY, USA. Article 6, 12 pages. isbn:9781450347679 https://doi.org/10.1145/3064899.3064910
Google Scholar
Digital Library
- Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Programming. https://doi.org/10.48550/arXiv.1701.03757 arxiv:1701.03757.
Google Scholar
- Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. 2018. An introduction to probabilistic programming. arXiv preprint arXiv:1809.10756, https://doi.org/10.48550/arXiv.1809.10756
Google Scholar
- Janis Voigtländer. 2008. Asymptotic improvement of computations over free monads. In International Conference on Mathematics of Program Construction. 388–403. https://doi.org/10.1007/978-3-540-70594-9_20
Google Scholar
Digital Library
- David Wingate, Andreas Stuhlmueller, and Noah Goodman. 2011. Lightweight Implementations of Probabilistic Programming Languages Via Transformational Compilation. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.) (Proceedings of Machine Learning Research, Vol. 15). PMLR, Fort Lauderdale, FL, USA. 770–778. https://proceedings.mlr.press/v15/wingate11a.html
Google Scholar
- N Wu and T Schrijvers. 2015. Fusion for Free Efficient Algebraic Effect Handlers. Springer-Verlag Berlin, 302–322. https://doi.org/10.1007/978-3-319-19797-5_15
Google Scholar
Cross Ref
- Nicolas Wu, Tom Schrijvers, and Ralf Hinze. 2014. Effect handlers in scope. In Proceedings of the 2014 ACM SIGPLAN Symposium on Haskell. 1–12. https://doi.org/10.1145/2633357.2633358
Google Scholar
Digital Library
- Daniel Yekutieli. 2012. Adjusted Bayesian inference for selected parameters. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74, 3 (2012), 515–541. https://doi.org/10.1111/j.1467-9868.2011.01016.x
Google Scholar
Cross Ref
Index Terms
Modular probabilistic models via algebraic effects
Recommendations
Functional programming for modular Bayesian inference
We present an architectural design of a library for Bayesian modelling and inference in modern functional programming languages. The novel aspect of our approach are modular implementations of existing state-of-the-art inference algorithms. Our design ...
A design proposal for Gen: probabilistic programming with fast custom inference via code generation
MAPL 2018: Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming LanguagesProbabilistic programming languages have the potential to make probabilistic modeling and inference easier to use in practice, but only if inference is sufficiently fast and accurate for real applications. Thus far, this has only been possible for ...
Probabilistic abductive logic programming using Dirichlet priors
Probabilistic programming is an area of research that aims to develop general inference algorithms for probabilistic models expressed as probabilistic programs whose execution corresponds to inferring the parameters of those models. In this paper, we ...






Comments