Abstract
This article presents a type-based analysis for deriving upper bounds on the expected execution cost of probabilistic programs. The analysis is naturally compositional, parametric in the cost model, and supports higher-order functions and inductive data types. The derived bounds are multivariate polynomials that are functions of data structures. Bound inference is enabled by local type rules that reduce type inference to linear constraint solving. The type system is based on the potential method of amortized analysis and extends automatic amortized resource analysis (AARA) for deterministic programs. A main innovation is that bounds can contain symbolic probabilities, which may appear in data structures and function arguments. Another contribution is a novel soundness proof that establishes the correctness of the derived bounds with respect to a distribution-based operational cost semantics that also includes nontrivial diverging behavior. For cost models like time, derived bounds imply termination with probability one. To highlight the novel ideas, the presentation focuses on linear potential and a core language. However, the analysis is implemented as an extension of Resource Aware ML and supports polynomial bounds and user defined data structures. The effectiveness of the technique is evaluated by analyzing the sample complexity of discrete distributions and with a novel average-case estimation for deterministic programs that combines expected cost analysis with statistical methods.
Supplemental Material
- E. Albert, P. Arenas, S. Genaim, M. Gómez-Zamalloa, G. Puebla, D. Ramírez, G. Román, and D. Zanardini. 2009. Termination and Cost Analysis with COSTA and its User Interfaces. Electr. Notes Theor. Comp. Sci. 258 ( December 2009 ). Issue 1.Google Scholar
- E. Albert, J. C. Fernández, and G. Román-Díez. 2015. Non-cumulative Resource Analysis. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'15).Google Scholar
- R. Atkey. 2010. Amortised Resource Analysis with Separation Logic. In European Symp. on Programming (ESOP'10).Google Scholar
- M. Avanzini, U. Dal Lago, and A. Ghyselen. 2019. Type-Based Complexity Analysis of Probabilistic Functional Programs. In Logic in Computer Science (LICS'19).Google Scholar
- M. Avanzini, U. Dal Lago, and G. Moser. 2015. Analysing the Complexity of Functional Programs: Higher-Order Meets First-Order. In Int. Conf. on Functional Programming (ICFP'15).Google Scholar
- M. Avanzini and G. Moser. 2013. A Combination Framework for Complexity. In Int. Conf. on Rewriting Techniques and Applications (RTA'13).Google Scholar
- Ziv Bar-Yossef and Maxim Gurevich. 2008. Random sampling from a search engine's index. Journal of the ACM (JACM) 55, 5 ( 2008 ), 1-74.Google Scholar
Digital Library
- G. Barthe, B. Grégoire, and S. Zanella Béguelin. 2009. Formal Certification of Code-based Cryptographic Proofs. In Princ. of Prog. Lang. (POPL'09).Google Scholar
- G. Barthe, B. Köpf, F. Olmedo, and S. Zanella Béguelin. 2012. Probabilistic Relational Reasoning for Diferential Privacy. In Princ. of Prog. Lang. (POPL'12).Google Scholar
- Kevin Batz, B. L. Kaminski, J.-P. Katoen, and C. Matheja. 2018. How long, O Bayesian network, will I sample thee?. In European Symp. on Programming (ESOP'18).Google Scholar
- S. Bhat, A. Agarwal, R. Vuduc, and A. Gray. 2012. A Type Theory for Probability Density Functions. In Princ. of Prog. Lang. (POPL'12).Google Scholar
- S. Bhat, J. Borgström, A. D. Gordon, and C. Russo. 2013. Deriving probability density functions from probabilistic functional programs. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'13).Google Scholar
- P. Billingsley. 2012. Probability and Measure. John Wiley & Sons, Inc.Google Scholar
- R. Blanc, T. A. Henzinger, T. Hottelier, and L. Kovács. 2010. ABC: Algebraic Bound Computation for Loops. In Logic for Prog., AI., and Reasoning (LPAR'10).Google Scholar
Digital Library
- J. Borgström, U. Dal Lago, A. D. Gordon, and M. Szymczak. 2016. A Lambda-Calculus Foundation for Universal Probabilistic Programming. In Int. Conf. on Functional Programming (ICFP'16).Google Scholar
- M. Brockschmidt, F. Emmes, S. Falke, C. Fuhs, and J. Giesl. 2014. Alternating Runtime and Size Complexity Analysis of Integer Programs. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'14).Google Scholar
- J. Burnim, S. Juvekar, and K. Sen. 2009. WISE: Automated Test Generation for Worst-case Complexity. In Int. Conf. on Softw. Eng. (ICSE'09).Google Scholar
- Q. Carbonneaux, J. Hofmann, T. Reps, and Z. Shao. 2017. Automated Resource Analysis with Coq Proof Objects. In Computer Aided Verif. (CAV'17).Google Scholar
- Q. Carbonneaux, J. Hofmann, and Z. Shao. 2015. Compositional Certified Resource Bounds. In Prog. Lang. Design and Impl. (PLDI'15).Google Scholar
- B. Carpenter, A. Gelman, M. D. Hofman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. 2017. Stan: A Probabilistic Programming Language. J. Statistical Softw. 76 ( 2017 ). Issue 1.Google Scholar
- A. Charguéraud and F. Pottier. 2015. Machine-Checked Verification of the Correctness and Amortized Complexity of an Eficient Union-Find Implementation. In Interactive Theorem Proving (ITP'15).Google Scholar
- K. Chatterjee, H. Fu, and A. K. Goharshady. 2016a. Termination Analysis of Probabilistic Programs Through Positivstellensatz's. In Computer Aided Verif. (CAV'16).Google Scholar
- K. Chatterjee, H. Fu, P. Novotný, and R. Hasheminezhad. 2016b. Algorithmic Analysis of Qualitative and Quantitative Termination Problems for Afine Probabilistic Programs. In Princ. of Prog. Lang. (POPL'16).Google Scholar
- B. Chen, Y. Liu, and W. Le. 2016. Generating Performance Distributions via Probabilistic Symbolic Execution. In Int. Conf. on Softw. Eng. (ICSE'16).Google Scholar
- E. Çiçek, G. Barthe, M. Gaboardi, D. Garg, and J. Hofmann. 2017. Relational Cost Analysis. In Princ. of Prog. Lang. (POPL'17).Google Scholar
- E. Çiçek, D. Garg, and U. A. Acar. 2015. Refinement Types for Incremental Computational Complexity. In European Symp. on Programming (ESOP'15).Google Scholar
- Edward A Codling, Michael J Plank, and Simon Benhamou. 2008. Random walk models in biology. Journal of the Royal Society Interface 5, 25 ( 2008 ), 813-834.Google Scholar
Cross Ref
- K. Crary and S. Weirich. 2000. Resource Bound Certification. In Princ. of Prog. Lang. (POPL'00).Google Scholar
- J. Da Silva and J. G. Stefan. 2006. A Probabilistic Pointer Analysis for Speculative Optimizations. In Architectural Support for Prog. Lang. and Op. Syst. (ASPLOS'06).Google Scholar
- U. Dal Lago and M. Gaboardi. 2011. Linear Dependent Types and Relative Completeness. In Logic in Computer Science (LICS'11).Google Scholar
- U. Dal Lago and A. Ghyselen. 2018. On Linear Dependent Types and Probabilistic Termination. In International Workshop on Developments in Implicit Computational Complexity.Google Scholar
- U. Dal Lago and C. Grellois. 2019. Probabilistic Termination by Monadic Afine Sized Typing. Trans. on Prog. Lang. and Syst. 41 ( June 2019 ). Issue 2.Google Scholar
- U. Dal Lago and B. Petit. 2013. The Geometry of Types. In Princ. of Prog. Lang. (POPL'13).Google Scholar
- N. A. Danielsson. 2008. Lightweight Semiformal Time Complexity Analysis for Purely Functional Data Structures. In Princ. of Prog. Lang. (POPL'08).Google Scholar
- N. Danner, D. R. Licata, and R. Ramyaa. 2015. Denotational Cost Semantics for Functional Languages with Inductive Types. In Int. Conf. on Functional Programming (ICFP'15).Google Scholar
- D. Djuric. 2019. Billions of Random Numbers in a Blink of an Eye. Available on https://dragan.rocks/articles/19/Billionrandom-numbers-blink-eye-Clojure.Google Scholar
- A. Filieri, C. S. Păsăreanu, and W. Visser. 2013. Reliability Analysis in Symbolic Pathfinder. In Int. Conf. on Softw. Eng. (ICSE'13).Google Scholar
- A. Filieri, C. S. Păsăreanu, W. Visser, and J. Geldenhuys. 2014. Statistical Symbolic Execution with Informed Sampling. In Found. of Softw. Eng. (FSE'14).Google Scholar
- A. Flores-Montoya and R. Hähnle. 2014. Resource Analysis of Complex Programs with Cost Equations. In Asian Symp. on Prog. Lang. and Systems (APLAS'14).Google Scholar
- F. Frohn, M. Naaf, J. Hensel, M. Brockschmidt, and J. Giesl. 2016. Lower Runtime Bounds for Integer Programs. In Int. Joint Conf. on Automated Reasoning (IJCAR'16).Google Scholar
- M. Gaboardi, A. Haeberlen, J. Hsu, A. Narayan, and B. C. Pierce. 2013. Linear Dependent Types for Diferential Privacy. In Princ. of Prog. Lang. (POPL'13).Google Scholar
- N. D. Goodman and A. Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. Available on http://dippl.org.Google Scholar
- Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming. In Proceedings of the on Future of Software Engineering. 167-181.Google Scholar
Digital Library
- S. Gulwani. 2009. SPEED: Symbolic Complexity Bound Analysis. In Computer Aided Verif. (CAV'09).Google Scholar
- M. Hark, B. L. Kaminski, J. Giesl, and J.-P. Katoen. 2020. Aiming Low Is Harder: Induction for Lower Bounds in Probabilistic Program Verification. In Princ. of Prog. Lang. (POPL'20).Google Scholar
- R. Harper. 2016. Practical Foundations for Programming Languages. Cambridge University Press.Google Scholar
Digital Library
- J. Hofmann, K. Aehlig, and M. Hofmann. 2011. Multivariate Amortized Resource Analysis. In Princ. of Prog. Lang. (POPL'11).Google Scholar
- J. Hofmann, A. Das, and S.-C. Weng. 2017. Towards Automatic Resource Bound Analysis for OCaml. In Princ. of Prog. Lang. (POPL'17).Google Scholar
- J. Hofmann and M. Hofmann. 2010a. Amortized Resource Analysis with Polymorphic Recursion and Partial Big-Step Operational Semantics. In Asian Symp. on Prog. Lang. and Systems (APLAS'10).Google Scholar
- J. Hofmann and M. Hofmann. 2010b. Amortized Resource Analysis with Polynomial Potential. In European Symp. on Programming (ESOP'10).Google Scholar
- M. Hofmann and S. Jost. 2003. Static Prediction of Heap Space Usage for First-Order Functional Programs. In Princ. of Prog. Lang. (POPL'03).Google Scholar
- M. Hofmann and G. Moser. 2015. Multivariate Amortised Resource Analysis for Term Rewrite Systems. In Int. Conf. on Typed Lambda Calculi and Applications (TLCA'15).Google Scholar
- M. Hofmann and G. Moser. 2018. Analysis of Logarithmic Amortised Complexity. Technical Report. Computing Research Repository.Google Scholar
- S. Jost, K. Hammond, H.-W. Loidl, and M. Hofmann. 2010. Static Determination of Quantitative Resource Usage for Higher-Order Programs. In Princ. of Prog. Lang. (POPL'10).Google Scholar
- S. Jost, H.-W. Loidl, K. Hammond, N. Scaife, and M. Hofmann. 2009. Carbon Credits for Resource-Bounded Computations using Amortised Analysis. In Symp. on Form. Meth. (FM'09).Google Scholar
- D. M. Kahn and J. Hofmann. 2020. Exponential Automatic Amortized Resource Analysis. In International Conference on Foundations of Software Science and Computation Structures (FoSSaCS'20).Google Scholar
- B. L. Kaminski, J.-P. Katoen, C. Matheja, and F. Olmedo. 2016. Weakest Precondition Reasoning for Expected Run-Times of Probabilistic Programs. In European Symp. on Programming (ESOP'16).Google Scholar
- G. A. Kavvos, E. Morehouse, D. R. Licata, and N. Danner. 2020. Recurrence Extraction for Functional Programs through Call-by-Push-Value. In Princ. of Prog. Lang. (POPL'20).Google Scholar
- Z. Kincaid, J. Breck, A. F. Boroujeni, and T. Reps. 2017. Compositional Recurrence Analysis Revisited. In Prog. Lang. Design and Impl. (PLDI'17).Google Scholar
- T. Knoth, D. Wang, N. Polikarpova, and J. Hofmann. 2019. Resource-Guided Program Synthesis. In Prog. Lang. Design and Impl. (PLDI'19).Google Scholar
- Donald Knuth and Andrew Yao. 1976. Algorithms and Complexity: New Directions and Recent Results, chapter The complexity of nonuniform random number generation.Google Scholar
- D. Kozen. 1981. Semantics of Probabilistic Programs. J. Comput. Syst. Sci. 22 ( June 1981 ). Issue 3.Google Scholar
Cross Ref
- S. Kura, N. Urabe, and I. Hasuo. 2019. Tail Probability for Randomized Program Runtimes via Martingales for Higher Moments. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'19).Google Scholar
- A. K. Lew, M. F. Cusumano-Towner, B. Sherman, M. Carbin, and V. K. Mansinghka. 2020. Trace Types and Denotational Semantics for Sound Programmable Inference in Probabilistic Languages. In Princ. of Prog. Lang. (POPL'20).Google Scholar
- Ch L MacLeod, Ž Ivezić, CS Kochanek, S Kozłowski, B Kelly, E Bullock, A Kimball, B Sesar, D Westman, K Brooks, et al. 2010. Modeling the time variability of SDSS stripe 82 quasars as a damped random walk. The Astrophysical Journal 721, 2 ( 2010 ), 1014.Google Scholar
Cross Ref
- V. K. Mansinghka, U. Schaechtle, S. Handa, A. Radul, Y. Chen, and M. C. Rinard. 2018. Probabilistic Programming with Programmable Inference. In Prog. Lang. Design and Impl. (PLDI'18).Google Scholar
- A. K. McIver and C. C. Morgan. 2005. Abstraction, Refinement and Proof for Probabilistic Systems. Springer Science+Business Media, Inc.Google Scholar
- Richard A Meese and Kenneth Rogof. 1983. Empirical exchange rate models of the seventies: Do they fit out of sample? Journal of international economics 14, 1-2 ( 1983 ), 3-24.Google Scholar
Cross Ref
- V. C. Ngo, Q. Carbonneaux, and J. Hofmann. 2018. Bounded Expectations: Resource Analysis for Probabilistic Programs. In Prog. Lang. Design and Impl. (PLDI'18).Google Scholar
- V. C. Ngo, Mario Dehesa-Azuara, M. Fredrikson, and J. Hofmann. 2017. Verifying and Synthesizing Constant-Resource Implementations with Types. In Symp. on Sec. and Privacy (SP'17).Google Scholar
- T. Nipkow. 2015. Amortized Complexity Verified. In Interactive Theorem Proving (ITP'15).Google Scholar
- Y. Noller, R. Kersten, and C. S. Păsăreanu. 2018. Badger: Complexity Analysis with Fuzzing and Symbolic Execution. In Int. Symp. on Softw. Testing and Analysis (ISSTA'18).Google Scholar
- L. Noschinski, F. Emmes, and J. Giesl. 2013. Analyzing Innermost Runtime Complexity of Term Rewriting by Dependency Pairs. J. Automated Reasoning 51 ( June 2013 ). Issue 1.Google Scholar
- F. Olmedo, B. L. Kaminski, J.-P. Katoen, and C. Matheja. 2016. Reasoning about Recursive Probabilistic Programs. In Logic in Computer Science (LICS'16).Google Scholar
- G. D. Plotkin. 1977. LCF Considered as a Programming Language. Theor. Comput. Sci. 5 ( 1977 ), 223-255.Google Scholar
- I. Radicek, G. Barthe, M. Gaboardi, D. Garg, and F. Zuleger. 2018. Monadic Refinements for Relational Cost Analysis. In Princ. of Prog. Lang. (POPL'18).Google Scholar
- G. Ramalingam. 1996. Data Flow Frequency Analysis. In Prog. Lang. Design and Impl. (PLDI'96).Google Scholar
- J. Reed and B. C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Diferential Privacy. In Int. Conf. on Functional Programming (ICFP'10).Google Scholar
- F. A. Saad, C. E. Freer, M. C. Rinard, and V. K. Mansinghka. 2020. Optimal Approximate Sampling from Discrete Probability Distributions. In Princ. of Prog. Lang. (POPL'20).Google Scholar
- M. Sinn, F. Zuleger, and H. Veith. 2014. A Simple and Scalable Approach to Bound Analysis and Amortized Complexity Analysis. In Computer Aided Verif. (CAV'14).Google Scholar
- R. E. Tarjan. 1985. Amortized Computational Complexity. SIAM J. Algebraic Discrete Methods 6 ( August 1985 ). Issue 2.Google Scholar
- Joseph Tassarotti and Robert Harper. 2019. A Separation Logic for Concurrent Randomized Programs. Proc. ACM Program. Lang. 3, POPL, Article 64 ( Jan. 2019 ), 30 pages. https://doi.org/10.1145/3290377 Google Scholar
Digital Library
- P. B. Vasconcelos. 2008. Space Cost Analysis Using Sized Types. Ph.D. Dissertation. School of Computer Science, University of St Andrews.Google Scholar
- Andre W Visser. 1997. Using random walk models to simulate the vertical distribution of particles in a turbulent water column. Marine Ecology Progress Series 158 ( 1997 ), 275-281.Google Scholar
- D. Walker. 2002. Substructural Type Systems. In Advanced Topics in Types and Programming Languages. MIT Press.Google Scholar
- D. Wang and J. Hofmann. 2019. Type-Guided Worst-Case Input Generation. In Princ. of Prog. Lang. (POPL'19).Google Scholar
- Di Wang, Jan Hofmann, and Thomas Reps. 2020a. Tail Bound Analysis for Probabilistic Programs via Central Moments. arXiv: 2001. 10150 [cs.PL]Google Scholar
- Di Wang, David M Kahn, and Jan Hofmann. 2020b. Raising Expectations: Automating Expected Cost Analysis with Types. arXiv: 2006. 14010 [cs.PL]Google Scholar
- P. Wang, H. Fu, A. K. Goharshady, K. Chatterjee, X. Qin, and W. Shi. 2019. Cost Analysis of Nondeterministic Probabilistic Programs. In Prog. Lang. Design and Impl. (PLDI'19).Google Scholar
- P. Wang, D. Wang, and A. Chlipala. 2017. TiML: A Functional Language for Practical Complexity Analysis with Invariants. In Object-Oriented Prog., Syst., Lang., and Applications (OOPSLA'17).Google Scholar
- D. Williams. 1991. Probability with Martingales. Cambridge University Press.Google Scholar
- D. Wingate and T. Weber. 2013. Automated Variational Inference in Probabilistic Programming. Technical Report. Computing Research Repository.Google Scholar
- H. Xi. 2002. Dependent Types for Program Termination Verification. J. Higher-Order and Symbolic Comp. 15 ( 2002 ). Issue 1.Google Scholar
- F. Zuleger, M. Sinn, S. Gulwani, and H. Veith. 2011. Bound Analysis of Imperative Programs with the Size-change Abstraction. In Static Analysis Symp. (SAS'11).Google Scholar
Digital Library
Index Terms
Raising expectations: automating expected cost analysis with types
Recommendations
Type-Indexed Data Types
MPC '02: Proceedings of the 6th International Conference on Mathematics of Program ConstructionA polytypic function is a function that can be instantiated on many data types to obtain data type specific functionality. Examples of polytypic functions are the functions that can be derived in Haskell, such as show, read, and '=='. More advanced ...
Polymorphic type inference and abstract data types
Many statically typed programming languages provide an abstract data type construct, such as the module in Modula-2. However, in most of these languages, implementations of abstract data types are not first-class values. Thus, they cannot be assigned to ...






Comments