Enhancing symbolic execution with veritesting

ABSTRACT

We present MergePoint, a new binary-only symbolic execution system for large-scale and fully unassisted testing of commodity off-the-shelf (COTS) software. MergePoint introduces veritesting, a new technique that employs static symbolic execution to amplify the effect of dynamic symbolic execution. Veritesting allows MergePoint to find twice as many bugs, explore orders of magnitude more paths, and achieve higher code coverage than previous dynamic symbolic execution systems. MergePoint is currently running daily on a 100 node cluster analyzing 33,248 Linux binaries; has generated more than 15 billion SMT queries, 200 million test cases, 2,347,420 crashes, and found 11,687 bugs in 4,379 distinct applications.

References

1. Online Bibliography for Symbolic Execution. http:// sites.google.com/site/symexbib.
2. A. V. Aho, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986.
3. J. R. Allen, K. Kennedy, C. Porterfield, and J. Warren. Conversion of Control Dependence to Data Dependence. In Proceedings of the 10th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pages 177–189, New York, NY, USA, 1983. ACM Press.
4. S. Anand, P. Godefroid, and N. Tillmann. Demand-Driven Compositional Symbolic Execution. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 367–381, Berlin, Heidelberg, 2008. Springer-Verlag.
5. D. Babic. Exploiting structure for scalable software verification. PhD thesis, University of British Columbia, Vancouver, Canada, 2008.
6. D. Babic and A. J. Hu. Calysto: Scalable and Precise Extended Static Checking. In Proceedings of the 30th International Conference on Software Engineering, pages 211–220, New York, NY, USA, 2008. ACM.
7. S. Bardin, P. Herrmann, J. Leroux, O. Ly, R. Tabary, and A. Vincent. The BINCOA Framework for Binary Code Analysis. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 165– 170, Berlin, Heidelberg, 2011. Springer-Verlag.
8. D. Beyer, T. A. Henzinger, and G. Theoduloz. Configurable Software Verification: Concretizing the Convergence of Model Checking and Program Analysis. In Proceedings of the 19th International Conference on Computer Aided Verification, pages 504–518, Berlin, Heidelberg, 2007. Springer-Verlag.
9. P. Boonstoppel, C. Cadar, and D. Engler. RWset: Attacking Path Explosion in Constraint-Based Test Generation. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 351–366, Berlin, Heidelberg, 2008. Springer-Verlag.
10. E. Bounimova, P. Godefroid, and D. Molnar. Billions and Billions of Constraints: Whitebox Fuzz Testing in Production. In Proceedings of the 35th IEEE International Conference on Software Engineering, pages 122–131, Piscataway, NJ, USA, 2013. IEEE Press.
11. R. S. Boyer, B. Elspas, and K. N. Levitt. SELECT—a formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Notices, 10(6): 234–245, 1975.
12. D. Brumley, I. Jager, T. Avgerinos, and E. J. Schwartz. BAP: A Binary Analysis Platform. In Proceedings of the 23rd International Conference on Computer Aided Verification, pages 463–469. Springer-Verlag, 2011.
13. S. Bucur, V. Ureche, C. Zamfir, and G. Candea. Parallel symbolic execution for automated real-world software testing. In Proceedings of the 6th ACM SIGOPS European Conference on Computer Systems, pages 183–198. ACM Press, 2011.
14. C. Cadar and K. Sen. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90, 2013.
15. C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. EXE : Automatically Generating Inputs of Death. In Proceedings of the 13th ACM Conference on Computer and Communications Security, New York, NY, USA, 2006. ACM.
16. C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In Proceedings of the 8th USENIX Symposium on Operating System Design and Implementation, pages 209–224, Berkeley, CA, USA, 2008. USENIX Association.
17. C. Cadar, D. Dunbar, and D. R. Engler. KLEE Coreutils Experiment. http://klee.github.io/klee/ CoreutilsExperiments.html, 2008.
18. S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley. Unleashing Mayhem on Binary Code. In Proceedings of the 33rd IEEE Symposium on Security and Privacy, pages 380–394, Washington, DC, USA, 2012. IEEE Computer Society.
19. V. Chipounov, V. Kuznetsov, and G. Candea. S2E: A platform for in-vivo multi-path analysis of software systems. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems, pages 265–278, New York, NY, USA, 2011. ACM.
20. P. Collingbourne, C. Cadar, and P. H. Kelly. Symbolic crosschecking of floating-point and SIMD code. Proceedings of the 6th ACM SIGOPS European conference on Computer Systems, pages 315–328, 2011.
21. L. De Moura and N. Bjørner. Z3: An Efficient SMT Solver. In Proceedings of 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340, Berlin, Heidelberg, 2008. Springer-Verlag.
22. I. Dillig, T. Dillig, and A. Aiken. Sound, Complete and Scalable Path-Sensitive Analysis. In Proceedings of the 29th ACM Conference on Programming Language Design and Implementation, pages 270–280, New York, NY, USA, 2008. ACM.
23. J. Filliˆ atre and S. Conchon. Type-safe modular hashconsing. In Proceedings of the Workshop on ML, pages 12–19, New York, NY, USA, 2006. ACM.
24. C. Flanagan and J. Saxe. Avoiding exponential explosion: Generating compact verification conditions. In Proceedings of the 28th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 193–205, New York, NY, USA, 2001. ACM.
25. P. Godefroid. Compositional Dynamic Test Generation. In Proceedings of the 34th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 47–54, New York, NY, USA, 2007. ACM.
26. P. Godefroid, N. Klarlund, and K. Sen. DART : Directed Automated Random Testing. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, New York, NY, USA, 2005. ACM.
27. P. Godefroid, M. Y. Levin, and D. Molnar. Automated Whitebox Fuzz Testing. In Proceedings of the 15th Network and Distributed System Security Symposium. The Internet Society, 2008.
28. P. Godefroid, M. Y. Levin, and D. Molnar. SAGE: Whitebox Fuzzing for Security Testing. Communications of the ACM, 55(3):40–44, 2012.
29. E. Goto. Monocopy and Associative Algorithms in Extended Lisp. Technical Report TR-74-03, University of Tokyo, 1974.
30. T. Hansen, P. Schachte, and H. Søndergaard. State Joining and Splitting for the Symbolic Execution of Binaries. Runtime Verification, pages 76–92, 2009.
31. W. Howden. Methodology for the Generation of Program Test Data. IEEE Transactions on Computers, C-24(5):554–560, 1975.
32. J. Kinder and H. Veith. Jakstab: A Static Analysis Platform for Binaries. In Proceedings of the 20th International Conference on Computer Aided Verification, pages 423–427, Berlin, Heidelberg, 2008. Springer-Verlag.
33. J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976.
34. A. Koelbl and C. Pixley. Constructing Efficient Formal Models from High-Level Descriptions Using Symbolic Simulation. International Journal of Parallel Programming, 33(6):645–666, Dec. 2005.
35. V. Kuznetsov, J. Kinder, S. Bucur, and G. Candea. Efficient state merging in symbolic execution. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 193–204, New York, NY, USA, 2012. ACM.
36. C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In Proceedings of the International Symposium on Code Generation and Optimization: Feedback-directed and Runtime Optimization, pages 75–86, Washington, DC, USA, 2004. IEEE Computer Society.
37. K. R. M. Leino. Efficient weakest preconditions. Information Processing Letters, 93(6):281–288, 2005.
38. C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: Building Customized Program Analysis Tols with Dynamic Instrumentation. In Proceedings of the 26th ACM Conference on Programming Language Design and Implementation, pages 190–200, New York, NY, USA, 2005. ACM.
39. P. D. Marinescu and C. Cadar. Make test-zesti: A symbolic execution solution for improving regression testing. In Proceedings of the 34th International Conference on Software Engineering, pages 716–726, Piscataway, NJ, USA, 2012. IEEE Press.
40. Mayhem. 1.2K Crashes in Debian, 2013. URL http://lists.debian.org/debian-devel/2013/06/ msg00720.html.
41. Mayhem. Open Source Statistics & Analysis, 2013. URL http://www.forallsecure.com/summaries.
42. D. Molnar, X. Li, and D. Wagner. Dynamic test generation to find integer bugs in x86 binary linux programs. In Proceedings of the USENIX Security Symposium, pages 67–82, 2009.
43. C. S. Păsăreanu and W. Visser. A survey of new trends in symbolic execution for software testing and analysis. International Journal on Software Tools for Technology Transfer, 11(4):339–353, Aug. 2009.
44. A. J. Romano. Linux Bug Release, July 2013. URL http://www.bugsdujour.com/release/.
45. E. J. Schwartz, T. Avgerinos, and D. Brumley. All You Ever Wanted to Know about Dynamic Taint Analysis and Forward Symbolic Execution (but Might Have Been Afraid to Ask). In Proceedings of the 31st IEEE Symposium on Security and Privacy, pages 317–331, Washington, DC, USA, 2010. IEEE Computer Society.
46. K. Sen, D. Marinov, and G. Agha. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 263–272, New York, NY, USA, 2005. ACM.
47. P. Tu and D. Padua. Efficient building and placing of gating functions. In Proceedings of the 16th ACM Conference on Programming Language Design and Implementation, pages 47–55, New York, NY, USA, 1995. ACM.
48. Y. Xie and A. Aiken. Scalable error detection using boolean satisfiability. In Proceedings of the 32nd ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 351–363, New York, NY, USA, 2005. ACM.
49. H. Zhu, P. A. V. Hall, and J. H. R. May. Software unit test coverage and adequacy. ACM Computing Surveys, 29(4):366–427, 1997.

Index Terms

• Software and its engineering

  • Software creation and management

    • Software verification and validation

      • Software defect analysis

        • Software testing and debugging

• Theory of computation

  • Semantics and reasoning

    • Program semantics