Abstract
Solver-aided tools rely on symbolic evaluation to reduce programming tasks, such as verification and synthesis, to satisfiability queries. Many reusable symbolic evaluation engines are now available as part of solver-aided languages and frameworks, which have made it possible for a broad population of programmers to create and apply solver-aided tools to new domains. But to achieve results for real-world problems, programmers still need to write code that makes effective use of the underlying engine, and understand where their code needs careful design to elicit the best performance. This task is made difficult by the all-paths execution model of symbolic evaluators, which defies both human intuition and standard profiling techniques.
This paper presents symbolic profiling, a new approach to identifying and diagnosing performance bottlenecks in programs under symbolic evaluation. To help with diagnosis, we develop a catalog of common performance anti-patterns in solver-aided code. To locate these bottlenecks, we develop SymPro, a new profiling technique for symbolic evaluation. SymPro identifies bottlenecks by analyzing two implicit resources at the core of every symbolic evaluation engine: the symbolic heap and symbolic evaluation graph. These resources form a novel performance model of symbolic evaluation that is general (encompassing all forms of symbolic evaluation), explainable (providing programmers with a conceptual framework for understanding symbolic evaluation), and actionable (enabling precise localization of bottlenecks). Performant solver-aided code carefully manages the shape of these implicit structures; SymPro makes their evolution explicit to the programmer.
To evaluate SymPro, we implement profilers for the Rosette solver-aided language and the Jalangi program analysis framework. Applying SymPro to 15 published solver-aided tools, we discover 8 previously undiagnosed performance issues. Repairing these issues improves performance by orders of magnitude, and our patches were accepted by the tools' developers. We also conduct a small user study with Rosette programmers, finding that SymPro helps them both understand what the symbolic evaluator is doing and identify performance issues they could not otherwise locate.
Supplemental Material
- Amazon Web Services. 2018. Quivela. (2018). https://github.com/awslabs/quivelaGoogle Scholar
- Glenn Ammons, Jong-Deok Choi, Manish Gupta, and Nikhil Swamy. 2004. Finding and Removing Performance Bottlenecks in Large Systems. In Proceedings of the 18th European Conference on Object-Oriented Programming (ECOOP). Oslo, Norway, 170–194.Google Scholar
Cross Ref
- Domagoj Babić and Alan J. Hu. 2008. Calysto: scalable and precise extended static checking. In Proceedings of the 30th International Conference on Software Engineering (ICSE). Leipzig, Germany, 211–220. Google Scholar
Digital Library
- Eli Barzilay. 2017. Profile: Statistical Profiler. http://docs.racket-lang.org/profile/. (2017).Google Scholar
- Armin Biere, Alessandro Cimatti, Edmund M. Clarke, and Yunshan Zhu. 1999. Symbolic Model Checking Without BDDs. In Proceedings of the 5th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Amsterdam, The Netherlands, 193–207. Google Scholar
Digital Library
- Nicolas Boichat. 2015. Issue 502898: ext4: Filesystem corruption on panic. (June 2015). https://code.google.com/p/chromium/ issues/detail?id=502898 .Google Scholar
- James Bornholt, Antoine Kaufmann, Jialin Li, Arvind Krishnamurthy, Emina Torlak, and Xi Wang. 2016. Specifying and checking file system crash-consistency models. In Proceedings of the 21st International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Atlanta, GA, USA, 83–98. Google Scholar
Digital Library
- James Bornholt and Emina Torlak. 2017. Synthesizing Memory Models from Framework Sketches and Litmus Tests. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Barcelona, Spain, 467–481. Google Scholar
Digital Library
- Alan Borning. 2016. Wallingford: Toward a Constraint Reactive Programming Language. In Proceedings of the Constrained and Reactive Objects Workshop (CROW). Málaga, Spain. Google Scholar
Digital Library
- William J. Bowman, Swaha Miller, Vincent St-Amour, and R. Kent Dybvig. 2015. Profile-guided Meta-programming. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Portland, OR, USA, 229–239. Google Scholar
Digital Library
- Stefan Bucur, Johannes Kinder, and George Candea. 2014. Prototyping symbolic execution engines for interpreted languages. In Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Salt Lake City, UT, USA, 239–254. Google Scholar
Digital Library
- Eric Butler, Emina Torlak, and Zoran Popović. 2017. Synthesizing Interpretable Strategies for Solving Puzzle Games. In Proceedings of the 12th International Conference on the Foundations of Digital Games (FDG). Hyannis, MA, USA. Google Scholar
Digital Library
- Cristian Cadar. 2015. Targeted program transformations for symbolic execution. In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Bergamo, Italy, 906–909. Google Scholar
Digital Library
- Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. Klee: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Proceedings of the 8th Symposium on Operating Systems Design and Implementation (OSDI). San Diego, CA, 209–224. Google Scholar
Digital Library
- Cristian Cadar and Koushik Sen. 2013. Symbolic Execution for Software Testing: Three Decades Later. Commun. ACM 56, 2 (2013), 82–90. Google Scholar
Digital Library
- Kartik Chandra and Rastislav Bodik. 2018. Bonsai: Synthesis-Based Reasoning for Type Systems. Proc. ACM Program. Lang. 2, POPL (Jan. 2018), 62:1–62:34. Google Scholar
Digital Library
- Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017a. Cosette. (2017). http://github.com/uwdb/CosetteGoogle Scholar
- Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017b. Cosette: An Automated Prover for SQL. In Proceedings of the 8th Biennial Conference on Innovative Data Systems (CIDR). Chaminade, CA, USA.Google Scholar
- Edmund Clarke, Daniel Kroening, and Flavio Lerda. 2004. A Tool for Checking ANSI-C Programs. In Proceedings of the 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Barcelona, Spain, 168–176.Google Scholar
Cross Ref
- Lori A. Clarke. 1976. A System to Generate Test Data and Symbolically Execute Programs. IEEE Transactions on Software Engineering 2, 3 (1976), 215–222. Google Scholar
Digital Library
- Emilio Coppa, Camil Demetrescu, and Irene Finocchi. 2012. Input-sensitive Profiling. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Beijing, China, 89–98. Google Scholar
Digital Library
- Charlie Curtsinger and Emery D. Berger. 2015. Coz: Finding Code That Counts with Causal Profiling. In Proceedings of the 25th ACM Symposium on Operating Systems Principles (SOSP). Monterey, CA, USA, 184–197. Google Scholar
Digital Library
- EPICS. 2017. Experimental Physics and Industrial Control System. (2017). http://www.aps.anl.gov/epics/Google Scholar
- A. P. Ershov. 1958. On Programming of Arithmetic Operations. Commun. ACM 1, 8 (1958), 3–6. Google Scholar
Digital Library
- Malay Ganai and Aarti Gupta. 2008. Tunneling and slicing: Towards scalable BMC. In Proceedings of the 45th Design Automation Conference (DAC). Anaheim, CA, USA, 137–142. Google Scholar
Digital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 26th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Chicago, IL, USA, 213–223. Google Scholar
Digital Library
- Patrice Godefroid, Michael Y. Levin, and David Molnar. 2008. Automated Whitebox Fuzz Testing. In Proceedings of the 15th Network and Distributed System Security Symposium (NDSS). San Diego, CA, USA.Google Scholar
- Milod Kazerounian, Niki Vazou, Austin Bourgerie, Jeffrey S. Foster, and Emina Torlak. 2018. Refinement Types for Ruby. In Proceedings of the 19th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI). Los Angeles, CA, USA, 269–290.Google Scholar
Cross Ref
- James C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (1976), 385–394. Google Scholar
Digital Library
- Volodymyr Kuznetsov, Johannes Kinder, Stefan Bucur, and George Candea. 2012. Efficient State Merging in Symbolic Execution. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Beijing, China, 89–98. Google Scholar
Digital Library
- Stuart Pernsteiner, Calvin Loncaric, Emina Torlak, Zachary Tatlock, Xi Wang, Michael D. Ernst, and Jonathan Jacky. 2016. Investigating Safety of a Radiotherapy Machine Using System Models with Pluggable Checkers. In Proceedings of the 28th International Conference on Computer Aided Verification (CAV), Vol. 2. Toronto, ON, Canada, 23–41.Google Scholar
Cross Ref
- Phitchaya Mangpo Phothilimthana, Aditya Thakur, Rastislav Bodik, and Dinakar Dhurjati. 2016. Scaling Up Superoptimization. In Proceedings of the 21st International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). Atlanta, GA, USA, 297–310. Google Scholar
Digital Library
- Racket 2017. The Racket Programming Language. (2017). https://racket- lang.orgGoogle Scholar
- Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. 2013. Jalangi: a selective record-replay and dynamic analysis framework for JavaScript. In Proceedings of the 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Saint Petersburg, Russian Federation, 488–498. Google Scholar
Digital Library
- Koushik Sen, George Necula, Liang Gong, and Wontae Choi. 2015. MultiSE: multi-path symbolic execution using value summaries. In Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE). Bergamo, Italy, 842–853. Google Scholar
Digital Library
- Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Vijay Saraswat, and Sanjit Seshia. 2006. Combinatorial Sketching for Finite Programs. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). San Jose, CA, USA, 404–415. Google Scholar
Digital Library
- Vincent St-Amour, Sam Tobin-Hochstadt, and Matthias Felleisen. 2012. Optimization Coaching: Optimizers Learn to Communicate with Programmers. In Proceedings of the 27th ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Tuscon, AZ, USA, 163–178. Google Scholar
Digital Library
- Emina Torlak. 2018. Rosette. (2018). http://github.com/emina/rosetteGoogle Scholar
- Emina Torlak and Rastislav Bodik. 2013. Growing Solver-Aided Languages with Rosette. In Proceedings of the 2013 ACM Symposium on New Ideas in Programming and Reflections on Software (Onward!). Indianapolis, IN, USA, 135–152. Google Scholar
Digital Library
- Emina Torlak and Rastislav Bodik. 2014. A Lightweight Symbolic Virtual Machine for Solver-Aided Host Languages. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Edinburgh, United Kingdom, 530–541. Google Scholar
Digital Library
- Richard Uhler and Nirav Dave. 2014. Smten with Satisfiability-Based Search. In Proceedings of the 29th ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Portland, OR, USA, 157–176. Google Scholar
Digital Library
- Jonas Wagner, Volodymyr Kuznetsov, and George Candea. 2013. -Overify: Optimizing Programs for Fast Verification. In Proceedings of the 14th Workshop on Hot Topics in Operating Systems (HotOS). Santa Ana Pueblo, NM, USA. Google Scholar
Digital Library
- Konstantin Weitz, Doug Woos, Emina Torlak, Michael D. Ernst, Arvind Krishnamurthy, and Zachary Tatlock. 2016. Scalable Verification of Border Gateway Protocol Configurations with an SMT Solver. In Proceedings of the 31st ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). Amsterdam, The Netherlands, 765–780. Google Scholar
Digital Library
- Max Willsey, Luis Ceze, and Karin Strauss. 2018. Puddle: An Operating System for Reliable, High-Level Programming of Digital Microfluidic Devices. In Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Wild and Crazy Ideas Session. Williamsburg, VA, USA.Google Scholar
- Yichen Xie and Alex Aiken. 2005. Scalable Error Detection Using Boolean Satisfiability. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL). Long Beach, CA, USA, 351–363. Google Scholar
Digital Library
Index Terms
Finding code that explodes under symbolic evaluation
Recommendations
Symbolic execution and program testing
This paper describes the symbolic execution of programs. Instead of supplying the normal inputs to a program (e.g. numbers) one supplies symbols representing arbitrary values. The execution proceeds as in a normal execution except that values may be ...
Fixing Code that Explodes Under Symbolic Evaluation
Verification, Model Checking, and Abstract InterpretationAbstractEffective symbolic evaluation is key to building scalable verification and synthesis tools based on SMT solving. These tools use symbolic evaluators to reduce the semantics of all paths through a finite program to logical constraints, discharged ...
Symbolic types for lenient symbolic execution
We present lambda_sym, a typed λ-calculus for lenient symbolic execution, where some language constructs do not recognize symbolic values. Its type system, however, ensures safe behavior of all symbolic values in a program. Our calculus extends a base ...






Comments