skip to main content

Finding code that explodes under symbolic evaluation

Published:24 October 2018Publication History
Skip Abstract Section

Abstract

Solver-aided tools rely on symbolic evaluation to reduce programming tasks, such as verification and synthesis, to satisfiability queries. Many reusable symbolic evaluation engines are now available as part of solver-aided languages and frameworks, which have made it possible for a broad population of programmers to create and apply solver-aided tools to new domains. But to achieve results for real-world problems, programmers still need to write code that makes effective use of the underlying engine, and understand where their code needs careful design to elicit the best performance. This task is made difficult by the all-paths execution model of symbolic evaluators, which defies both human intuition and standard profiling techniques.

This paper presents symbolic profiling, a new approach to identifying and diagnosing performance bottlenecks in programs under symbolic evaluation. To help with diagnosis, we develop a catalog of common performance anti-patterns in solver-aided code. To locate these bottlenecks, we develop SymPro, a new profiling technique for symbolic evaluation. SymPro identifies bottlenecks by analyzing two implicit resources at the core of every symbolic evaluation engine: the symbolic heap and symbolic evaluation graph. These resources form a novel performance model of symbolic evaluation that is general (encompassing all forms of symbolic evaluation), explainable (providing programmers with a conceptual framework for understanding symbolic evaluation), and actionable (enabling precise localization of bottlenecks). Performant solver-aided code carefully manages the shape of these implicit structures; SymPro makes their evolution explicit to the programmer.

To evaluate SymPro, we implement profilers for the Rosette solver-aided language and the Jalangi program analysis framework. Applying SymPro to 15 published solver-aided tools, we discover 8 previously undiagnosed performance issues. Repairing these issues improves performance by orders of magnitude, and our patches were accepted by the tools' developers. We also conduct a small user study with Rosette programmers, finding that SymPro helps them both understand what the symbolic evaluator is doing and identify performance issues they could not otherwise locate.

Skip Supplemental Material Section

Supplemental Material

a149-bornholt.webm

References

  1. Amazon Web Services. 2018. Quivela. (2018). https://github.com/awslabs/quivelaGoogle ScholarGoogle Scholar
  2. Glenn Ammons, Jong-Deok Choi, Manish Gupta, and Nikhil Swamy. 2004. Finding and Removing Performance Bottlenecks in Large Systems. In Proceedings of the 18th European Conference on Object-Oriented Programming (ECOOP). Oslo, Norway, 170–194.Google ScholarGoogle ScholarCross RefCross Ref
  3. Domagoj Babić and Alan J. Hu. 2008. Calysto: scalable and precise extended static checking. In Proceedings of the 30th International Conference on Software Engineering (ICSE). Leipzig, Germany, 211–220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Eli Barzilay. 2017. Profile: Statistical Profiler. http://docs.racket-lang.org/profile/. (2017).Google ScholarGoogle Scholar
  5. Armin Biere, Alessandro Cimatti, Edmund M. Clarke, and Yunshan Zhu. 1999. Symbolic Model Checking Without BDDs. In Proceedings of the 5th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Amsterdam, The Netherlands, 193–207. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Nicolas Boichat. 2015. Issue 502898: ext4: Filesystem corruption on panic. (June 2015). https://code.google.com/p/chromium/ issues/detail?id=502898 .Google ScholarGoogle Scholar
  7. James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang. 2016. Specifying and checking file system crash-consistency models. In Proceedings of the 21st International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Atlanta, GA, USA, 83–98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. James Bornholt and Emina Torlak. 2017. Synthesizing Memory Models from Framework Sketches and Litmus Tests. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Barcelona, Spain, 467–481. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Alan Borning. 2016. Wallingford: Toward a Constraint Reactive Programming Language. In Proceedings of the Constrained and Reactive Objects Workshop (CROW). Málaga, Spain. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. William J. Bowman, Swaha Miller, Vincent St-Amour, and R. Kent Dybvig. 2015. Profile-guided Meta-programming. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Portland, OR, USA, 229–239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Stefan Bucur, Johannes Kinder, and George Candea. 2014. Prototyping symbolic execution engines for interpreted languages. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Salt Lake City, UT, USA, 239–254. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Eric Butler, Emina Torlak, and Zoran Popović. 2017. Synthesizing Interpretable Strategies for Solving Puzzle Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games (FDG). Hyannis, MA, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Cristian Cadar. 2015. Targeted program transformations for symbolic execution. In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Bergamo, Italy, 906–909. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Proceedings of the 8th Symposium on Operating Systems Design and Implementation (OSDI). San Diego, CA, 209–224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Cristian Cadar and Koushik Sen. 2013. Symbolic Execution for Software Testing: Three Decades Later. Commun. ACM 56, 2 (2013), 82–90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Kartik Chandra and Rastislav Bodik. 2018. Bonsai: Synthesis-Based Reasoning for Type Systems. Proc. ACM Program. Lang. 2, POPL (Jan. 2018), 62:1–62:34. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017a. Cosette. (2017). http://github.com/uwdb/CosetteGoogle ScholarGoogle Scholar
  18. Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017b. Cosette: An Automated Prover for SQL. In Proceedings of the 8th Biennial Conference on Innovative Data Systems (CIDR). Chaminade, CA, USA.Google ScholarGoogle Scholar
  19. Edmund Clarke, Daniel Kroening, and Flavio Lerda. 2004. A Tool for Checking ANSI-C Programs. In Proceedings of the 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Barcelona, Spain, 168–176.Google ScholarGoogle ScholarCross RefCross Ref
  20. Lori A. Clarke. 1976. A System to Generate Test Data and Symbolically Execute Programs. IEEE Transactions on Software Engineering 2, 3 (1976), 215–222. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Emilio Coppa, Camil Demetrescu, and Irene Finocchi. 2012. Input-sensitive Profiling. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Beijing, China, 89–98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Charlie Curtsinger and Emery D. Berger. 2015. Coz: Finding Code That Counts with Causal Profiling. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP). Monterey, CA, USA, 184–197. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. EPICS. 2017. Experimental Physics and Industrial Control System. (2017). http://www.aps.anl.gov/epics/Google ScholarGoogle Scholar
  24. A. P. Ershov. 1958. On Programming of Arithmetic Operations. Commun. ACM 1, 8 (1958), 3–6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Malay Ganai and Aarti Gupta. 2008. Tunneling and slicing: Towards scalable BMC. In Proceedings of the 45th Design Automation Conference (DAC). Anaheim, CA, USA, 137–142. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 26th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Chicago, IL, USA, 213–223. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Patrice Godefroid, Michael Y. Levin, and David Molnar. 2008. Automated Whitebox Fuzz Testing. In Proceedings of the 15th Network and Distributed System Security Symposium (NDSS). San Diego, CA, USA.Google ScholarGoogle Scholar
  28. Milod Kazerounian, Niki Vazou, Austin Bourgerie, Jeffrey S. Foster, and Emina Torlak. 2018. Refinement Types for Ruby. In Proceedings of the 19th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI). Los Angeles, CA, USA, 269–290.Google ScholarGoogle ScholarCross RefCross Ref
  29. James C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (1976), 385–394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Volodymyr Kuznetsov, Johannes Kinder, Stefan Bucur, and George Candea. 2012. Efficient State Merging in Symbolic Execution. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Beijing, China, 89–98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael D. Ernst, and Jonathan Jacky. 2016. Investigating Safety of a Radiotherapy Machine Using System Models with Pluggable Checkers. In Proceedings of the 28th International Conference on Computer Aided Verification (CAV), Vol. 2. Toronto, ON, Canada, 23–41.Google ScholarGoogle ScholarCross RefCross Ref
  32. Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodik, and Dinakar Dhurjati. 2016. Scaling Up Superoptimization. In Proceedings of the 21st International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Atlanta, GA, USA, 297–310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Racket 2017. The Racket Programming Language. (2017). https://racket- lang.orgGoogle ScholarGoogle Scholar
  34. Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: a selective record-replay and dynamic analysis framework for JavaScript. In Proceedings of the 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Saint Petersburg, Russian Federation, 488–498. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Koushik Sen, George Necula, Liang Gong, and Wontae Choi. 2015. MultiSE: multi-path symbolic execution using value summaries. In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Bergamo, Italy, 842–853. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Vijay Saraswat, and Sanjit Seshia. 2006. Combinatorial Sketching for Finite Programs. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). San Jose, CA, USA, 404–415. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Vincent St-Amour, Sam Tobin-Hochstadt, and Matthias Felleisen. 2012. Optimization Coaching: Optimizers Learn to Communicate with Programmers. In Proceedings of the 27th ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Tuscon, AZ, USA, 163–178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Emina Torlak. 2018. Rosette. (2018). http://github.com/emina/rosetteGoogle ScholarGoogle Scholar
  39. Emina Torlak and Rastislav Bodik. 2013. Growing Solver-Aided Languages with Rosette. In Proceedings of the 2013 ACM Symposium on New Ideas in Programming and Reflections on Software (Onward!). Indianapolis, IN, USA, 135–152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Emina Torlak and Rastislav Bodik. 2014. A Lightweight Symbolic Virtual Machine for Solver-Aided Host Languages. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Edinburgh, United Kingdom, 530–541. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Richard Uhler and Nirav Dave. 2014. Smten with Satisfiability-Based Search. In Proceedings of the 29th ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Portland, OR, USA, 157–176. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Jonas Wagner, Volodymyr Kuznetsov, and George Candea. 2013. -Overify: Optimizing Programs for Fast Verification. In Proceedings of the 14th Workshop on Hot Topics in Operating Systems (HotOS). Santa Ana Pueblo, NM, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Konstantin Weitz, Doug Woos, Emina Torlak, Michael D. Ernst, Arvind Krishnamurthy, and Zachary Tatlock. 2016. Scalable Verification of Border Gateway Protocol Configurations with an SMT Solver. In Proceedings of the 31st ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Amsterdam, The Netherlands, 765–780. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Max Willsey, Luis Ceze, and Karin Strauss. 2018. Puddle: An Operating System for Reliable, High-Level Programming of Digital Microfluidic Devices. In Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Wild and Crazy Ideas Session. Williamsburg, VA, USA.Google ScholarGoogle Scholar
  45. Yichen Xie and Alex Aiken. 2005. Scalable Error Detection Using Boolean Satisfiability. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL). Long Beach, CA, USA, 351–363. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Finding code that explodes under symbolic evaluation

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!