Abstract
Programming languages and software engineering tools routinely encounter components that are difficult to reason on via formal techniques or whose formal semantics are not even available—third-party libraries, inline assembly code, SIMD instructions, system calls, calls to machine learning models, etc. However, often access to these components is available as input-output oracles—interfaces are available to query these components on certain inputs to receive the respective outputs. We refer to such functions as closed-box functions. Regular SMT solvers are unable to handle such closed-box functions.
We propose Sādhak, a solver for SMT theories modulo closed-box functions. Our core idea is to use a synergistic combination of a fuzzer to reason on closed-box functions and an SMT engine to solve the constraints pertaining to the SMT theories. The fuzz and the SMT engines attempt to converge to a model by exchanging a rich set of interface constraints that are relevant and interpretable by them. Our implementation, Sādhak, demonstrates a significant advantage over the only other solver that is capable of handling such closed-box constraints: Sādhak solves 36.45% more benchmarks than the best-performing mode of this state-of-the-art solver and has 5.72x better PAR-2 score; on the benchmarks that are solved by both tools, Sādhak is (on an average) 14.62x faster.
- 2015. SMTCOMP SMTLib2 benchmarks.. https://smtlib.cs.uiowa.edu/benchmarks.shtml
Google Scholar
- 2019. Benchmarks for SyGuS Competition. SyGuS-Comp, https://github.com/SyGuS-Org/benchmarks
Google Scholar
- 2021. ESBMC. https://github.com/esbmc/esbmc/tree/master/regression
Google Scholar
- Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. 2013. Syntax-guided synthesis. In 2013 Formal Methods in Computer-Aided Design. 1–8. https://doi.org/10.1109/FMCAD.2013.6679385
Google Scholar
Cross Ref
- George Argyros, Ioannis Stais, Aggelos Kiayias, and Angelos D. Keromytis. 2016. Back in Black: Towards Formal, Black Box Analysis of Sanitizers and Filters. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016. IEEE Computer Society, 91–109.
Google Scholar
- 2017. Proceedings of SAT Competition 2017 : Solver and Benchmark Descriptions. http://hdl.handle.net/10138/224324
Google Scholar
- Clark Barrett, Aaron Stump, and Cesare Tinelli. 2010. The smt-lib standard: Version 2.0. In Proceedings of the 8th international workshop on satisfiability modulo theories (Edinburgh, England). 13, 14.
Google Scholar
- Clark W. Barrett, Christopher L. Conway, Morgan Deters, Liana Hadarean, Dejan Jovanovic, Tim King, Andrew Reynolds, and Cesare Tinelli. 2011. CVC4. In Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings, Ganesh Gopalakrishnan and Shaz Qadeer (Eds.) (Lecture Notes in Computer Science, Vol. 6806). Springer, 171–177. https://doi.org/10.1007/978-3-642-22110-1_14
Google Scholar
Cross Ref
- Luca Borzacchiello, Emilio Coppa, and Camil Demetrescu. 2021. Fuzzing Symbolic Expressions. In Proceedings of the 43rd International Conference on Software Engineering (ICSE ’21). https://doi.org/10.1109/ICSE43902.2021.00071
Google Scholar
Digital Library
- Cristian Cadar, Daniel Dunbar, and Dawson Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI’08). USENIX Association, USA.
Google Scholar
Digital Library
- Ankush Das, Shuvendu K. Lahiri, Akash Lal, and Yi Li. 2015. Angelic Verification: Precise Verification Modulo Unknowns. In Computer Aided Verification - 27th International Conference, CAV 2015, San Francisco, CA, USA, July 18-24, 2015, Proceedings, Part I, Daniel Kroening and Corina S. Pasareanu (Eds.) (Lecture Notes in Computer Science, Vol. 9206). Springer, 324–342. https://doi.org/10.1007/978-3-319-21690-4_19
Google Scholar
Cross Ref
- Ankush Das and Akash Lal. 2017. Precise Null Pointer Analysis Through Global Value Numbering. In Automated Technology for Verification and Analysis - 15th International Symposium, ATVA 2017, Pune, India, October 3-6, 2017, Proceedings (Lecture Notes in Computer Science, Vol. 10482). Springer, 25–41. https://doi.org/10.1007/978-3-319-68167-2_2
Google Scholar
Cross Ref
- Peter Dinges and Gul Agha. 2014. Solving complex path conditions through heuristic search on induced polytopes. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. https://doi.org/10.1145/2635868.2635889
Google Scholar
Digital Library
- Andrea Fioraldi, Dominik Maier, Heiko Eiß feldt, and Marc Heuse. 2020. AFL++: Combining Incremental Steps of Fuzzing Research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20). USENIX Association.
Google Scholar
- Harald Ganzinger, George Hagen, Robert Nieuwenhuis, Albert Oliveras, and Cesare Tinelli. 2004. DPLL (T): Fast decision procedures. In International Conference on Computer Aided Verification. 175–188. https://doi.org/10.1007/978-3-540-27813-9_14
Google Scholar
Cross Ref
- Anshul Garg and Subhajit Roy. 2015. Synthesizing Heap Manipulations via Integer Linear Programming. In Static Analysis - 22nd International Symposium, SAS 2015, Saint-Malo, France, September 9-11, 2015, Proceedings, Sandrine Blazy and Thomas P. Jensen (Eds.) (Lecture Notes in Computer Science, Vol. 9291). Springer, 109–127. https://doi.org/10.1007/978-3-662-48288-9_7
Google Scholar
Cross Ref
- Patrice Godefroid. 2011. Higher-Order Test Generation. SIGPLAN Not., 46, 6 (2011), jun, 258–269. issn:0362-1340 https://doi.org/10.1145/1993316.1993529
Google Scholar
Digital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN PLDI Conference (PLDI ’05). ACM, New York, NY, USA. 213–223. isbn:1595930566 https://doi.org/10.1145/1065010.1065036
Google Scholar
Digital Library
- Priyanka Golia, Subhajit Roy, and Kuldeep S. Meel. 2020. Manthan: A Data-Driven Approach for Boolean Function Synthesis. In Computer Aided Verification: 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21–24, 2020, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg. 611–633. isbn:978-3-030-53290-1 https://doi.org/10.1007/978-3-030-53291-8_31
Google Scholar
Digital Library
- Priyanka Golia, Subhajit Roy, and špace 0mm Kuldeep S. Meel. 2021. Program Synthesis as Dependency Quantified Formula Modulo Theory. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, Zhi-Hua Zhou (Ed.). ijcai.org, 1894–1900. https://doi.org/10.24963/ijcai.2021/261
Google Scholar
Cross Ref
- Priyanka Golia, Friedrich Slivovsky, Subhajit Roy, and Kuldeep S. Meel. 2021. Engineering an Efficient Boolean Functional Synthesis Engine. In IEEE/ACM International Conference On Computer Aided Design, ICCAD 2021, Munich, Germany, November 1-4, 2021. IEEE, 1–9. https://doi.org/10.1109/ICCAD51958.2021.9643583
Google Scholar
Digital Library
- Google Inc.. 2020. HonggFuzz: Security oriented software fuzzer. Supports evolutionary, feedback-driven fuzzing based on code coverage (SW and HW based). https://honggfuzz.dev/
Google Scholar
- Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated Program Repair. Commun. ACM, 62, 12 (2019), nov, 56–65. issn:0001-0782 https://doi.org/10.1145/3318162
Google Scholar
Digital Library
- Sumit Gulwani, Susmit Jha, Ashish Tiwari, and Ramarathnam Venkatesan. 2011. Synthesis of Loop-Free Programs. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’11). Association for Computing Machinery, New York, NY, USA. 62–73. isbn:9781450306638 https://doi.org/10.1145/1993498.1993506
Google Scholar
Digital Library
- Saurabh Joshi, Shuvendu K. Lahiri, and Akash Lal. 2012. Underspecified harnesses and interleaved bugs. In Proceedings of the 39th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2012, Philadelphia, Pennsylvania, USA, January 22-28, 2012, John Field and Michael Hicks (Eds.). ACM, 19–30. https://doi.org/10.1145/2103656.2103662
Google Scholar
Digital Library
- Pankaj Kumar Kalita, Miriyala Jeevan Kumar, and Subhajit Roy. 2022. Synthesis of Semantic Actions in Attribute Grammars. In Formal Methods in Computer Aided Design (FMCAD ’22). https://doi.org/10.34727/2021/isbn.978-3-85448-053-2_37
Google Scholar
- Daniel Kroening and Michael Tautschnig. 2014. CBMC–C bounded model checker. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 389–391. https://doi.org/10.1007/978-3-642-54862-8_26
Google Scholar
Cross Ref
- Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan Bachrach, and Koushik Sen. 2018. RFUZZ: Coverage-Directed Fuzz Testing of RTL on FPGAs. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE Press, 1–8. https://doi.org/10.1145/3240765.3240842
Google Scholar
Digital Library
- Sumit Lahiri and Subhajit Roy. 2022. Almost Correct Invariants: Synthesizing Inductive Invariants by Fuzzing Proofs. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA. 352–364. isbn:9781450393799 https://doi.org/10.1145/3533767.3534381
Google Scholar
Digital Library
- Shuvendu K. Lahiri, Akash Lal, Sridhar Gopinath, Alexander Nutz, Vladimir Levin, Rahul Kumar, Nate Deisinger, Jakob Lichtenberg, and Chetan Bansal. 2020. Angelic Checking within Static Driver Verifier: Towards high-precision defects without (modeling) cost. In 2020 Formal Methods in Computer Aided Design, FMCAD 2020, Haifa, Israel, September 21-24, 2020. IEEE, 169–178. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_24
Google Scholar
Cross Ref
- K Rustan M Leino. 2008. This is boogie 2. manuscript KRML, 178, 131 (2008), 9.
Google Scholar
- Caroline Lemieux, Rohan Padhye, Koushik Sen, and Dawn Song. 2018. PerfFuzz: Automatically Generating Pathological Inputs. ISSTA 2018. Association for Computing Machinery, New York, NY, USA. isbn:9781450356992 https://doi.org/10.1145/3213846.3213874
Google Scholar
Digital Library
- Alan Leung, John Sarracino, and Sorin Lerner. 2015. Interactive Parser Synthesis by Example. SIGPLAN Not., 50, 6 (2015), jun, 565–574. issn:0362-1340 https://doi.org/10.1145/2813885.2738002
Google Scholar
Digital Library
- Daniel Liew, Cristian Cadar, Alastair F. Donaldson, and J. Ryan Stinnett. 2019. Just Fuzz It: Solving Floating-Point Constraints Using Coverage-Guided Fuzzing. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA. 521–532. isbn:9781450355728 https://doi.org/10.1145/3338906.3338921
Google Scholar
Digital Library
- Sergey Mechtaev, Alberto Griggio, Alessandro Cimatti, and Abhik Roychoudhury. 2018. Symbolic execution with existential second-order constraints. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 389–399. https://doi.org/10.1145/3236024.3236049
Google Scholar
Digital Library
- Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). 691–701. https://doi.org/10.1145/2884781.2884807
Google Scholar
Digital Library
- Sujit Kumar Muduli and Subhajit Roy. 2022. Satisfiability Modulo Fuzzing: A Synergistic Combination of SMT Solving and Fuzzing (Artifact). https://doi.org/10.5281/zenodo.7066264
Google Scholar
Digital Library
- Sujit Kumar Muduli, Gourav Takhar, and Pramod Subramanyan. 2020. Hyperfuzzing for SoC Security Validation. In Proceedings of the 39th International Conference on Computer-Aided Design (ICCAD ’20). Association for Computing Machinery, New York, NY, USA. isbn:9781450380263 https://doi.org/10.1145/3400302.3415709
Google Scholar
Digital Library
- Greg Nelson and Derek C Oppen. 1979. Simplification by cooperating decision procedures. ACM Transactions on Programming Languages and Systems (TOPLAS), 1, 2 (1979), 245–257.
Google Scholar
Digital Library
- Robert Nieuwenhuis, Albert Oliveras, and Cesare Tinelli. 2006. Solving SAT and SAT Modulo Theories: From an Abstract Davis–Putnam–Logemann–Loveland Procedure to DPLL(T). J. ACM, 53, 6 (2006), nov, 937–977. issn:0004-5411 https://doi.org/10.1145/1217856.1217859
Google Scholar
Digital Library
- Shirin Nilizadeh, Yannic Noller, and Corina S. Pasareanu. 2019. DifFuzz: Differential Fuzzing for Side-Channel Analysis. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 176–187. https://doi.org/10.1109/ICSE.2019.00034
Google Scholar
Digital Library
- Rohan Padhye, Caroline Lemieux, Koushik Sen, Laurent Simon, and Hayawardh Vijayakumar. 2019. FuzzFactory: Domain-Specific Fuzzing with Waypoints. Proc. ACM Program. Lang., 3, OOPSLA (2019), Article 174, oct, 29 pages. https://doi.org/10.1145/3360600
Google Scholar
Digital Library
- Awanish Pandey, Phani Raj Goutham Kotcharlakota, and Subhajit Roy. 2019. Deferred Concretization in Symbolic Execution via Fuzzing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). https://doi.org/10.1145/3293882.3330554
Google Scholar
Digital Library
- Elizabeth Polgreen, Andrew Reynolds, and Sanjit A. Seshia. 2022. Satisfiability and Synthesis Modulo Oracles. In Verification, Model Checking, and Abstract Interpretation. https://doi.org/10.1007/978-3-030-94583-1_13
Google Scholar
Digital Library
- Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program Synthesis from Polymorphic Refinement Types. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16). Association for Computing Machinery, New York, NY, USA. 522–538. isbn:9781450342612 https://doi.org/10.1145/2908080.2908093
Google Scholar
Digital Library
- Nadia Polikarpova and Ilya Sergey. 2019. Structuring the Synthesis of Heap-Manipulating Programs. Proc. ACM Program. Lang., 3, POPL (2019), Article 72, jan, 30 pages. https://doi.org/10.1145/3290385
Google Scholar
Digital Library
- Corina S. Păsăreanu, Neha Rungta, and Willem Visser. 2011. Symbolic Execution with Mixed Concrete-Symbolic Solving. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11). Association for Computing Machinery, New York, NY, USA. 34–44. isbn:9781450305624 https://doi.org/10.1145/2001420.2001425
Google Scholar
Digital Library
- Subhajit Roy. 2013. From Concrete Examples to Heap Manipulating Programs. In Static Analysis - 20th International Symposium, SAS 2013, Seattle, WA, USA, June 20-22, 2013. Proceedings, Francesco Logozzo and Manuel Fähndrich (Eds.) (Lecture Notes in Computer Science, Vol. 7935). Springer, 126–149. https://doi.org/10.1007/978-3-642-38856-9_9
Google Scholar
Cross Ref
- Subhajit Roy, Justin Hsu, and Aws Albarghouthi. 2021. Learning Differentially Private Mechanisms. In 2021 IEEE Symposium on Security and Privacy (SP). 852–865. https://doi.org/10.1109/SP40001.2021.00060
Google Scholar
Cross Ref
- Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitt, and Yu Hu. 2018. Bug synthesis: Challenging bug-finding tools with deep faults. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (Eds.). ACM, 224–234. https://doi.org/10.1145/3236024.3236084
Google Scholar
Digital Library
- K Serebryany. 2015. libFuzzer a library for coverage-guided fuzz testing. LLVM project.
Google Scholar
- Dhruv Singal, Palak Agarwal, Saket Jhunjhunwala, and Subhajit Roy. 2018. Parse Condition: Symbolic Encoding of LL(1) Parsing. In LPAR-22. 22nd International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Gilles Barthe, Geoff Sutcliffe, and Margus Veanes (Eds.) (EPiC Series in Computing, Vol. 57). EasyChair, 637–655. issn:2398-7340 https://doi.org/10.29007/2ndp
Google Scholar
Cross Ref
- Armando Solar-Lezama. 2013. Program sketching. Int. J. Softw. Tools Technol. Transf., 15, 5-6 (2013), 475–495.
Google Scholar
Digital Library
- Armando Solar-Lezama, Rodric Rabbah, Rastislav Bodík, and Kemal Ebcioğlu. 2005. Programming by Sketching for Bit-Streaming Programs. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’05). Association for Computing Machinery, New York, NY, USA. isbn:1595930566 https://doi.org/10.1145/1065010.1065045
Google Scholar
Digital Library
- Gourav Takhar, Ramesh Karri, Christian Pilato, and Subhajit Roy. 2022. HOLL: Program Synthesis for Higher Order Logic Locking. In Tools and Algorithms for the Construction and Analysis of Systems - 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, April 2-7, 2022, Proceedings, Part I, Dana Fisman and Grigore Rosu (Eds.) (Lecture Notes in Computer Science, Vol. 13243). Springer, 3–24. https://doi.org/10.1007/978-3-030-99524-9_1
Google Scholar
Digital Library
- Emina Torlak and Rastislav Bodik. 2013. Growing Solver-Aided Languages with Rosette. In Proceedings of the 2013 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2013). Association for Computing Machinery, New York, NY, USA. 135–152. isbn:9781450324724 https://doi.org/10.1145/2509578.2509586
Google Scholar
Digital Library
- Aakanksha Verma, Pankaj Kumar Kalita, Awanish Pandey, and Subhajit Roy. 2020. Interactive debugging of concurrent programs under relaxed memory models. In CGO ’20: 18th ACM/IEEE International Symposium on Code Generation and Optimization, San Diego, CA, USA, February, 2020. ACM, 68–80. https://doi.org/10.1145/3368826.3377910
Google Scholar
Digital Library
- Sahil Verma and Subhajit Roy. 2017. Synergistic debug-repair of heap manipulations. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 163–173. https://doi.org/10.1145/3106237.3106263
Google Scholar
Digital Library
- Sahil Verma and špace 0mm Subhajit Roy. 2021. Debug-localize-repair: A symbiotic construction for heap manipulations. Formal Methods Syst. Des., 58, 3 (2021), 399–439. https://doi.org/10.1007/s10703-021-00387-z
Google Scholar
Digital Library
- Yuxin Wang, Zeyu Ding, Yingtai Xiao, Daniel Kifer, and Danfeng Zhang. 2021. DPGen: Automated Program Synthesis for Differential Privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (CCS ’21). Association for Computing Machinery, New York, NY, USA. 393–411. isbn:9781450384544 https://doi.org/10.1145/3460120.3484781
Google Scholar
Digital Library
- Michal Zalewski. 2019. American Fuzzy Lop. http://lcamtuf.coredump.cx/afl
Google Scholar
Index Terms
Satisfiability modulo fuzzing: a synergistic combination of SMT solving and fuzzing
Recommendations
Design and results of the 2nd annual satisfiability modulo theories competition (SMT-COMP 2006)
The Satisfiability Modulo Theories Competition (SMT-COMP) arose from the SMT-LIB initiative to spur adoption of common, community-designed formats, and to spark further advances in satisfiability modulo theories (SMT). The first SMT-COMP was held in ...
Incremental Linearization for Satisfiability and Verification Modulo Nonlinear Arithmetic and Transcendental Functions
Satisfiability Modulo Theories (SMT) is the problem of deciding the satisfiability of a first-order formula with respect to some theory or combination of theories; Verification Modulo Theories (VMT) is the problem of analyzing the reachability for ...
Using Higher Levels of Abstraction for Solving Optimization Problems by Boolean Satisfiability
ISVLSI '08: Proceedings of the 2008 IEEE Computer Society Annual Symposium on VLSIOptimization problems can be solved using Boolean Satisfiability by mapping them to a sequence of decision problems. Therefore, in the last years several encodings have been developed. Independently, also new solvers have been introduced lifting Boolean ...






Comments