skip to main content

Raising expectations: automating expected cost analysis with types

Published:03 August 2020Publication History
Related Artifact: Resource Aware ML software https://doi.org/10.1145/3410233
Skip Abstract Section

Abstract

This article presents a type-based analysis for deriving upper bounds on the expected execution cost of probabilistic programs. The analysis is naturally compositional, parametric in the cost model, and supports higher-order functions and inductive data types. The derived bounds are multivariate polynomials that are functions of data structures. Bound inference is enabled by local type rules that reduce type inference to linear constraint solving. The type system is based on the potential method of amortized analysis and extends automatic amortized resource analysis (AARA) for deterministic programs. A main innovation is that bounds can contain symbolic probabilities, which may appear in data structures and function arguments. Another contribution is a novel soundness proof that establishes the correctness of the derived bounds with respect to a distribution-based operational cost semantics that also includes nontrivial diverging behavior. For cost models like time, derived bounds imply termination with probability one. To highlight the novel ideas, the presentation focuses on linear potential and a core language. However, the analysis is implemented as an extension of Resource Aware ML and supports polynomial bounds and user defined data structures. The effectiveness of the technique is evaluated by analyzing the sample complexity of discrete distributions and with a novel average-case estimation for deterministic programs that combines expected cost analysis with statistical methods.

Skip Supplemental Material Section

Supplemental Material

Presentation at ICFP '20

References

  1. E. Albert, P. Arenas, S. Genaim, M. Gómez-Zamalloa, G. Puebla, D. Ramírez, G. Román, and D. Zanardini. 2009. Termination and Cost Analysis with COSTA and its User Interfaces. Electr. Notes Theor. Comp. Sci. 258 ( December 2009 ). Issue 1.Google ScholarGoogle Scholar
  2. E. Albert, J. C. Fernández, and G. Román-Díez. 2015. Non-cumulative Resource Analysis. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'15).Google ScholarGoogle Scholar
  3. R. Atkey. 2010. Amortised Resource Analysis with Separation Logic. In European Symp. on Programming (ESOP'10).Google ScholarGoogle Scholar
  4. M. Avanzini, U. Dal Lago, and A. Ghyselen. 2019. Type-Based Complexity Analysis of Probabilistic Functional Programs. In Logic in Computer Science (LICS'19).Google ScholarGoogle Scholar
  5. M. Avanzini, U. Dal Lago, and G. Moser. 2015. Analysing the Complexity of Functional Programs: Higher-Order Meets First-Order. In Int. Conf. on Functional Programming (ICFP'15).Google ScholarGoogle Scholar
  6. M. Avanzini and G. Moser. 2013. A Combination Framework for Complexity. In Int. Conf. on Rewriting Techniques and Applications (RTA'13).Google ScholarGoogle Scholar
  7. Ziv Bar-Yossef and Maxim Gurevich. 2008. Random sampling from a search engine's index. Journal of the ACM (JACM) 55, 5 ( 2008 ), 1-74.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. G. Barthe, B. Grégoire, and S. Zanella Béguelin. 2009. Formal Certification of Code-based Cryptographic Proofs. In Princ. of Prog. Lang. (POPL'09).Google ScholarGoogle Scholar
  9. G. Barthe, B. Köpf, F. Olmedo, and S. Zanella Béguelin. 2012. Probabilistic Relational Reasoning for Diferential Privacy. In Princ. of Prog. Lang. (POPL'12).Google ScholarGoogle Scholar
  10. Kevin Batz, B. L. Kaminski, J.-P. Katoen, and C. Matheja. 2018. How long, O Bayesian network, will I sample thee?. In European Symp. on Programming (ESOP'18).Google ScholarGoogle Scholar
  11. S. Bhat, A. Agarwal, R. Vuduc, and A. Gray. 2012. A Type Theory for Probability Density Functions. In Princ. of Prog. Lang. (POPL'12).Google ScholarGoogle Scholar
  12. S. Bhat, J. Borgström, A. D. Gordon, and C. Russo. 2013. Deriving probability density functions from probabilistic functional programs. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'13).Google ScholarGoogle Scholar
  13. P. Billingsley. 2012. Probability and Measure. John Wiley & Sons, Inc.Google ScholarGoogle Scholar
  14. R. Blanc, T. A. Henzinger, T. Hottelier, and L. Kovács. 2010. ABC: Algebraic Bound Computation for Loops. In Logic for Prog., AI., and Reasoning (LPAR'10).Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. Borgström, U. Dal Lago, A. D. Gordon, and M. Szymczak. 2016. A Lambda-Calculus Foundation for Universal Probabilistic Programming. In Int. Conf. on Functional Programming (ICFP'16).Google ScholarGoogle Scholar
  16. M. Brockschmidt, F. Emmes, S. Falke, C. Fuhs, and J. Giesl. 2014. Alternating Runtime and Size Complexity Analysis of Integer Programs. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'14).Google ScholarGoogle Scholar
  17. J. Burnim, S. Juvekar, and K. Sen. 2009. WISE: Automated Test Generation for Worst-case Complexity. In Int. Conf. on Softw. Eng. (ICSE'09).Google ScholarGoogle Scholar
  18. Q. Carbonneaux, J. Hofmann, T. Reps, and Z. Shao. 2017. Automated Resource Analysis with Coq Proof Objects. In Computer Aided Verif. (CAV'17).Google ScholarGoogle Scholar
  19. Q. Carbonneaux, J. Hofmann, and Z. Shao. 2015. Compositional Certified Resource Bounds. In Prog. Lang. Design and Impl. (PLDI'15).Google ScholarGoogle Scholar
  20. B. Carpenter, A. Gelman, M. D. Hofman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. 2017. Stan: A Probabilistic Programming Language. J. Statistical Softw. 76 ( 2017 ). Issue 1.Google ScholarGoogle Scholar
  21. A. Charguéraud and F. Pottier. 2015. Machine-Checked Verification of the Correctness and Amortized Complexity of an Eficient Union-Find Implementation. In Interactive Theorem Proving (ITP'15).Google ScholarGoogle Scholar
  22. K. Chatterjee, H. Fu, and A. K. Goharshady. 2016a. Termination Analysis of Probabilistic Programs Through Positivstellensatz's. In Computer Aided Verif. (CAV'16).Google ScholarGoogle Scholar
  23. K. Chatterjee, H. Fu, P. Novotný, and R. Hasheminezhad. 2016b. Algorithmic Analysis of Qualitative and Quantitative Termination Problems for Afine Probabilistic Programs. In Princ. of Prog. Lang. (POPL'16).Google ScholarGoogle Scholar
  24. B. Chen, Y. Liu, and W. Le. 2016. Generating Performance Distributions via Probabilistic Symbolic Execution. In Int. Conf. on Softw. Eng. (ICSE'16).Google ScholarGoogle Scholar
  25. E. Çiçek, G. Barthe, M. Gaboardi, D. Garg, and J. Hofmann. 2017. Relational Cost Analysis. In Princ. of Prog. Lang. (POPL'17).Google ScholarGoogle Scholar
  26. E. Çiçek, D. Garg, and U. A. Acar. 2015. Refinement Types for Incremental Computational Complexity. In European Symp. on Programming (ESOP'15).Google ScholarGoogle Scholar
  27. Edward A Codling, Michael J Plank, and Simon Benhamou. 2008. Random walk models in biology. Journal of the Royal Society Interface 5, 25 ( 2008 ), 813-834.Google ScholarGoogle ScholarCross RefCross Ref
  28. K. Crary and S. Weirich. 2000. Resource Bound Certification. In Princ. of Prog. Lang. (POPL'00).Google ScholarGoogle Scholar
  29. J. Da Silva and J. G. Stefan. 2006. A Probabilistic Pointer Analysis for Speculative Optimizations. In Architectural Support for Prog. Lang. and Op. Syst. (ASPLOS'06).Google ScholarGoogle Scholar
  30. U. Dal Lago and M. Gaboardi. 2011. Linear Dependent Types and Relative Completeness. In Logic in Computer Science (LICS'11).Google ScholarGoogle Scholar
  31. U. Dal Lago and A. Ghyselen. 2018. On Linear Dependent Types and Probabilistic Termination. In International Workshop on Developments in Implicit Computational Complexity.Google ScholarGoogle Scholar
  32. U. Dal Lago and C. Grellois. 2019. Probabilistic Termination by Monadic Afine Sized Typing. Trans. on Prog. Lang. and Syst. 41 ( June 2019 ). Issue 2.Google ScholarGoogle Scholar
  33. U. Dal Lago and B. Petit. 2013. The Geometry of Types. In Princ. of Prog. Lang. (POPL'13).Google ScholarGoogle Scholar
  34. N. A. Danielsson. 2008. Lightweight Semiformal Time Complexity Analysis for Purely Functional Data Structures. In Princ. of Prog. Lang. (POPL'08).Google ScholarGoogle Scholar
  35. N. Danner, D. R. Licata, and R. Ramyaa. 2015. Denotational Cost Semantics for Functional Languages with Inductive Types. In Int. Conf. on Functional Programming (ICFP'15).Google ScholarGoogle Scholar
  36. D. Djuric. 2019. Billions of Random Numbers in a Blink of an Eye. Available on https://dragan.rocks/articles/19/Billionrandom-numbers-blink-eye-Clojure.Google ScholarGoogle Scholar
  37. A. Filieri, C. S. Păsăreanu, and W. Visser. 2013. Reliability Analysis in Symbolic Pathfinder. In Int. Conf. on Softw. Eng. (ICSE'13).Google ScholarGoogle Scholar
  38. A. Filieri, C. S. Păsăreanu, W. Visser, and J. Geldenhuys. 2014. Statistical Symbolic Execution with Informed Sampling. In Found. of Softw. Eng. (FSE'14).Google ScholarGoogle Scholar
  39. A. Flores-Montoya and R. Hähnle. 2014. Resource Analysis of Complex Programs with Cost Equations. In Asian Symp. on Prog. Lang. and Systems (APLAS'14).Google ScholarGoogle Scholar
  40. F. Frohn, M. Naaf, J. Hensel, M. Brockschmidt, and J. Giesl. 2016. Lower Runtime Bounds for Integer Programs. In Int. Joint Conf. on Automated Reasoning (IJCAR'16).Google ScholarGoogle Scholar
  41. M. Gaboardi, A. Haeberlen, J. Hsu, A. Narayan, and B. C. Pierce. 2013. Linear Dependent Types for Diferential Privacy. In Princ. of Prog. Lang. (POPL'13).Google ScholarGoogle Scholar
  42. N. D. Goodman and A. Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. Available on http://dippl.org.Google ScholarGoogle Scholar
  43. Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming. In Proceedings of the on Future of Software Engineering. 167-181.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. S. Gulwani. 2009. SPEED: Symbolic Complexity Bound Analysis. In Computer Aided Verif. (CAV'09).Google ScholarGoogle Scholar
  45. M. Hark, B. L. Kaminski, J. Giesl, and J.-P. Katoen. 2020. Aiming Low Is Harder: Induction for Lower Bounds in Probabilistic Program Verification. In Princ. of Prog. Lang. (POPL'20).Google ScholarGoogle Scholar
  46. R. Harper. 2016. Practical Foundations for Programming Languages. Cambridge University Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. J. Hofmann, K. Aehlig, and M. Hofmann. 2011. Multivariate Amortized Resource Analysis. In Princ. of Prog. Lang. (POPL'11).Google ScholarGoogle Scholar
  48. J. Hofmann, A. Das, and S.-C. Weng. 2017. Towards Automatic Resource Bound Analysis for OCaml. In Princ. of Prog. Lang. (POPL'17).Google ScholarGoogle Scholar
  49. J. Hofmann and M. Hofmann. 2010a. Amortized Resource Analysis with Polymorphic Recursion and Partial Big-Step Operational Semantics. In Asian Symp. on Prog. Lang. and Systems (APLAS'10).Google ScholarGoogle Scholar
  50. J. Hofmann and M. Hofmann. 2010b. Amortized Resource Analysis with Polynomial Potential. In European Symp. on Programming (ESOP'10).Google ScholarGoogle Scholar
  51. M. Hofmann and S. Jost. 2003. Static Prediction of Heap Space Usage for First-Order Functional Programs. In Princ. of Prog. Lang. (POPL'03).Google ScholarGoogle Scholar
  52. M. Hofmann and G. Moser. 2015. Multivariate Amortised Resource Analysis for Term Rewrite Systems. In Int. Conf. on Typed Lambda Calculi and Applications (TLCA'15).Google ScholarGoogle Scholar
  53. M. Hofmann and G. Moser. 2018. Analysis of Logarithmic Amortised Complexity. Technical Report. Computing Research Repository.Google ScholarGoogle Scholar
  54. S. Jost, K. Hammond, H.-W. Loidl, and M. Hofmann. 2010. Static Determination of Quantitative Resource Usage for Higher-Order Programs. In Princ. of Prog. Lang. (POPL'10).Google ScholarGoogle Scholar
  55. S. Jost, H.-W. Loidl, K. Hammond, N. Scaife, and M. Hofmann. 2009. Carbon Credits for Resource-Bounded Computations using Amortised Analysis. In Symp. on Form. Meth. (FM'09).Google ScholarGoogle Scholar
  56. D. M. Kahn and J. Hofmann. 2020. Exponential Automatic Amortized Resource Analysis. In International Conference on Foundations of Software Science and Computation Structures (FoSSaCS'20).Google ScholarGoogle Scholar
  57. B. L. Kaminski, J.-P. Katoen, C. Matheja, and F. Olmedo. 2016. Weakest Precondition Reasoning for Expected Run-Times of Probabilistic Programs. In European Symp. on Programming (ESOP'16).Google ScholarGoogle Scholar
  58. G. A. Kavvos, E. Morehouse, D. R. Licata, and N. Danner. 2020. Recurrence Extraction for Functional Programs through Call-by-Push-Value. In Princ. of Prog. Lang. (POPL'20).Google ScholarGoogle Scholar
  59. Z. Kincaid, J. Breck, A. F. Boroujeni, and T. Reps. 2017. Compositional Recurrence Analysis Revisited. In Prog. Lang. Design and Impl. (PLDI'17).Google ScholarGoogle Scholar
  60. T. Knoth, D. Wang, N. Polikarpova, and J. Hofmann. 2019. Resource-Guided Program Synthesis. In Prog. Lang. Design and Impl. (PLDI'19).Google ScholarGoogle Scholar
  61. Donald Knuth and Andrew Yao. 1976. Algorithms and Complexity: New Directions and Recent Results, chapter The complexity of nonuniform random number generation.Google ScholarGoogle Scholar
  62. D. Kozen. 1981. Semantics of Probabilistic Programs. J. Comput. Syst. Sci. 22 ( June 1981 ). Issue 3.Google ScholarGoogle ScholarCross RefCross Ref
  63. S. Kura, N. Urabe, and I. Hasuo. 2019. Tail Probability for Randomized Program Runtimes via Martingales for Higher Moments. In Tools and Algs. for the Construct. and Anal. of Syst. (TACAS'19).Google ScholarGoogle Scholar
  64. A. K. Lew, M. F. Cusumano-Towner, B. Sherman, M. Carbin, and V. K. Mansinghka. 2020. Trace Types and Denotational Semantics for Sound Programmable Inference in Probabilistic Languages. In Princ. of Prog. Lang. (POPL'20).Google ScholarGoogle Scholar
  65. Ch L MacLeod, Ž Ivezić, CS Kochanek, S Kozłowski, B Kelly, E Bullock, A Kimball, B Sesar, D Westman, K Brooks, et al. 2010. Modeling the time variability of SDSS stripe 82 quasars as a damped random walk. The Astrophysical Journal 721, 2 ( 2010 ), 1014.Google ScholarGoogle ScholarCross RefCross Ref
  66. V. K. Mansinghka, U. Schaechtle, S. Handa, A. Radul, Y. Chen, and M. C. Rinard. 2018. Probabilistic Programming with Programmable Inference. In Prog. Lang. Design and Impl. (PLDI'18).Google ScholarGoogle Scholar
  67. A. K. McIver and C. C. Morgan. 2005. Abstraction, Refinement and Proof for Probabilistic Systems. Springer Science+Business Media, Inc.Google ScholarGoogle Scholar
  68. Richard A Meese and Kenneth Rogof. 1983. Empirical exchange rate models of the seventies: Do they fit out of sample? Journal of international economics 14, 1-2 ( 1983 ), 3-24.Google ScholarGoogle ScholarCross RefCross Ref
  69. V. C. Ngo, Q. Carbonneaux, and J. Hofmann. 2018. Bounded Expectations: Resource Analysis for Probabilistic Programs. In Prog. Lang. Design and Impl. (PLDI'18).Google ScholarGoogle Scholar
  70. V. C. Ngo, Mario Dehesa-Azuara, M. Fredrikson, and J. Hofmann. 2017. Verifying and Synthesizing Constant-Resource Implementations with Types. In Symp. on Sec. and Privacy (SP'17).Google ScholarGoogle Scholar
  71. T. Nipkow. 2015. Amortized Complexity Verified. In Interactive Theorem Proving (ITP'15).Google ScholarGoogle Scholar
  72. Y. Noller, R. Kersten, and C. S. Păsăreanu. 2018. Badger: Complexity Analysis with Fuzzing and Symbolic Execution. In Int. Symp. on Softw. Testing and Analysis (ISSTA'18).Google ScholarGoogle Scholar
  73. L. Noschinski, F. Emmes, and J. Giesl. 2013. Analyzing Innermost Runtime Complexity of Term Rewriting by Dependency Pairs. J. Automated Reasoning 51 ( June 2013 ). Issue 1.Google ScholarGoogle Scholar
  74. F. Olmedo, B. L. Kaminski, J.-P. Katoen, and C. Matheja. 2016. Reasoning about Recursive Probabilistic Programs. In Logic in Computer Science (LICS'16).Google ScholarGoogle Scholar
  75. G. D. Plotkin. 1977. LCF Considered as a Programming Language. Theor. Comput. Sci. 5 ( 1977 ), 223-255.Google ScholarGoogle Scholar
  76. I. Radicek, G. Barthe, M. Gaboardi, D. Garg, and F. Zuleger. 2018. Monadic Refinements for Relational Cost Analysis. In Princ. of Prog. Lang. (POPL'18).Google ScholarGoogle Scholar
  77. G. Ramalingam. 1996. Data Flow Frequency Analysis. In Prog. Lang. Design and Impl. (PLDI'96).Google ScholarGoogle Scholar
  78. J. Reed and B. C. Pierce. 2010. Distance Makes the Types Grow Stronger: A Calculus for Diferential Privacy. In Int. Conf. on Functional Programming (ICFP'10).Google ScholarGoogle Scholar
  79. F. A. Saad, C. E. Freer, M. C. Rinard, and V. K. Mansinghka. 2020. Optimal Approximate Sampling from Discrete Probability Distributions. In Princ. of Prog. Lang. (POPL'20).Google ScholarGoogle Scholar
  80. M. Sinn, F. Zuleger, and H. Veith. 2014. A Simple and Scalable Approach to Bound Analysis and Amortized Complexity Analysis. In Computer Aided Verif. (CAV'14).Google ScholarGoogle Scholar
  81. R. E. Tarjan. 1985. Amortized Computational Complexity. SIAM J. Algebraic Discrete Methods 6 ( August 1985 ). Issue 2.Google ScholarGoogle Scholar
  82. Joseph Tassarotti and Robert Harper. 2019. A Separation Logic for Concurrent Randomized Programs. Proc. ACM Program. Lang. 3, POPL, Article 64 ( Jan. 2019 ), 30 pages. https://doi.org/10.1145/3290377 Google ScholarGoogle ScholarDigital LibraryDigital Library
  83. P. B. Vasconcelos. 2008. Space Cost Analysis Using Sized Types. Ph.D. Dissertation. School of Computer Science, University of St Andrews.Google ScholarGoogle Scholar
  84. Andre W Visser. 1997. Using random walk models to simulate the vertical distribution of particles in a turbulent water column. Marine Ecology Progress Series 158 ( 1997 ), 275-281.Google ScholarGoogle Scholar
  85. D. Walker. 2002. Substructural Type Systems. In Advanced Topics in Types and Programming Languages. MIT Press.Google ScholarGoogle Scholar
  86. D. Wang and J. Hofmann. 2019. Type-Guided Worst-Case Input Generation. In Princ. of Prog. Lang. (POPL'19).Google ScholarGoogle Scholar
  87. Di Wang, Jan Hofmann, and Thomas Reps. 2020a. Tail Bound Analysis for Probabilistic Programs via Central Moments. arXiv: 2001. 10150 [cs.PL]Google ScholarGoogle Scholar
  88. Di Wang, David M Kahn, and Jan Hofmann. 2020b. Raising Expectations: Automating Expected Cost Analysis with Types. arXiv: 2006. 14010 [cs.PL]Google ScholarGoogle Scholar
  89. P. Wang, H. Fu, A. K. Goharshady, K. Chatterjee, X. Qin, and W. Shi. 2019. Cost Analysis of Nondeterministic Probabilistic Programs. In Prog. Lang. Design and Impl. (PLDI'19).Google ScholarGoogle Scholar
  90. P. Wang, D. Wang, and A. Chlipala. 2017. TiML: A Functional Language for Practical Complexity Analysis with Invariants. In Object-Oriented Prog., Syst., Lang., and Applications (OOPSLA'17).Google ScholarGoogle Scholar
  91. D. Williams. 1991. Probability with Martingales. Cambridge University Press.Google ScholarGoogle Scholar
  92. D. Wingate and T. Weber. 2013. Automated Variational Inference in Probabilistic Programming. Technical Report. Computing Research Repository.Google ScholarGoogle Scholar
  93. H. Xi. 2002. Dependent Types for Program Termination Verification. J. Higher-Order and Symbolic Comp. 15 ( 2002 ). Issue 1.Google ScholarGoogle Scholar
  94. F. Zuleger, M. Sinn, S. Gulwani, and H. Veith. 2011. Bound Analysis of Imperative Programs with the Size-change Abstraction. In Static Analysis Symp. (SAS'11).Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Raising expectations: automating expected cost analysis with types

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!