skip to main content
research-article

Prototyping symbolic execution engines for interpreted languages

Authors Info & Claims
Published:24 February 2014Publication History
Skip Abstract Section

Abstract

Symbolic execution is being successfully used to automatically test statically compiled code. However, increasingly more systems and applications are written in dynamic interpreted languages like Python. Building a new symbolic execution engine is a monumental effort, and so is keeping it up-to-date as the target language evolves. Furthermore, ambiguous language specifications lead to their implementation in a symbolic execution engine potentially differing from the production interpreter in subtle ways.

We address these challenges by flipping the problem and using the interpreter itself as a specification of the language semantics. We present a recipe and tool (called Chef) for turning a vanilla interpreter into a sound and complete symbolic execution engine. Chef symbolically executes the target program by symbolically executing the interpreter's binary while exploiting inferred knowledge about the program's high-level structure.

Using Chef, we developed a symbolic execution engine for Python in 5 person-days and one for Lua in 3 person-days. They offer complete and faithful coverage of language features in a way that keeps up with future language versions at near-zero cost. Chef-produced engines are up to 1000 times more performant than if directly executing the interpreter symbolically without Chef.

References

  1. Al Danial. Cloc.footnotesizehttp://cloc.sourceforge.net/.Google ScholarGoogle Scholar
  2. S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst. Finding bugs in dynamic web applications. In Intl. Symp. on Software Testing and Analysis, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. T. F. Bissyandé, F. Thung, D. Lo, L. Jiang, and L. Réveillère. Popularity, interoperability, and impact of programming languages in 100,000 open source projects. In Computer Software & Applications Conference, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. E. Bounimova, P. Godefroid, and D. Molnar. Billions and billions of constraints: Whitebox fuzz testing in production. Technical Report MSR-TR-2012-55, Microsoft Research, 2012.Google ScholarGoogle Scholar
  5. S. Bucur, J. Kinder, and G. Candea. Making automated testing of cloud applications an integral component of PaaS. In Proc. 4th Asia-Pacific Workshop on Systems (APSYS 2013). USENIX, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. J. Burnim and K. Sen. Heuristics for scalable dynamic test generation. In Intl. Conf. on Automated Software Engineering, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Symp. on Operating Sys. Design and Implem., 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. M. Canini, D. Venzano, P. Peresini, D. Kostic, and J. Rexford. A NICE way to test openflow applications. In Symp. on Networked Systems Design and Implem., 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. V. Chipounov, V. Kuznetsov, and G. Candea. S2E: A platform for in-vivo multi-path analysis of software systems. In Intl. Conf. on Architectural Support for Programming Languages and Operating Systems, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. K. Claessen and J. Hughes. QuickCheck: A lightweight tool for random testing of haskell programs. In ACM SIGPLAN International Conference on Functional Programming, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. A. T. Clements, M. F. Kaashoek, N. Zeldovich, R. T. Morris, and E. Kohler. The scalable commutativity rule: Designing scalable software for multicore processors. In Symp. on Operating Systems Principles, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. L. de Moura and N. Bjorner. Generalized, efficient array decision procedures. In Intl. Conf. on Formal Methods in Computer-Aided Design, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  13. V. Ganesh and D. L. Dill. A decision procedure for bit-vectors and arrays. In Intl. Conf. on Computer Aided Verification, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. P. Godefroid. Compositional dynamic test generation. In Symp. on Principles of Programming Languages, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Intl. Conf. on Programming Language Design and Implem., 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. P. Godefroid, M. Y. Levin, and D. Molnar. Automated whitebox fuzz testing. In Network and Distributed System Security Symp., 2008.Google ScholarGoogle Scholar
  17. A. Kiezun, P. J. Guo, K. Jayaraman, and M. D. Ernst. Automatic creation of SQL injection and cross-site scripting attacks. In Intl. Conf. on Software Engineering, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. R. S. King. The top 10 programming languages. IEEE Spectrum, 48 (10): 84, 2011.Google ScholarGoogle ScholarCross RefCross Ref
  19. V. Kuznetsov, V. Chipounov, and G. Candea. Testing closed-source binary device drivers with DDT. In USENIX Annual Technical Conf., 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. V. Kuznetsov, J. Kinder, S. Bucur, and G. Candea. Efficient state merging in symbolic execution. In Intl. Conf. on Programming Language Design and Implem., 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis and transformation. In Intl. Symp. on Code Generation and Optimization, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. The Python Language Reference. Python Software Foundation. http://docs.python.org/3/reference/.Google ScholarGoogle Scholar
  23. S. Sapra, M. Minea, S. Chaki, A. Gurfinkel, and E. M. Clarke. Finding errors in python programs using dynamic symbolic execution. In Intl. Conf. on Testing Software and Systems, 2013.Google ScholarGoogle ScholarCross RefCross Ref
  24. P. Saxena, D. Akhawe, S. Hanna, F. Mao, S. McCamant, and D. Song. A symbolic execution framework for JavaScript. In IEEE Symp. on Security and Privacy, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. D. Song, D. Brumley, H. Yin, J. Caballero, I. Jager, M. G. Kang, Z. Liang, J. Newsome, P. Poosankam, and P. Saxena. Bitblaze: A new approach to computer security via binary analysis. In Intl. Conf. on Information Systems Security, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. N. Tillmann and W. Schulte. Parameterized unit tests. In Symp. on the Foundations of Software Eng., 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. Wagner, V. Kuznetsov, and G. Candea. -OVERIFY: Optimizing programs for fast verification. In Workshop on Hot Topics in Operating Systems, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. Fitness-guided path exploration in dynamic symbolic execution. In Intl. Conf. on Dependable Systems and Networks, 2009.Google ScholarGoogle ScholarCross RefCross Ref
  29. C. Zamfir and G. Candea. Execution synthesis: A technique for automated debugging. In ACM EuroSys European Conf. on Computer Systems, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Prototyping symbolic execution engines for interpreted languages

            Recommendations

            Reviews

            Jacques Carette

            While extensive testing is undoubtedly useful, it is also tedious. Automating this task as much as is feasible is an active research area; it has been more successfully applied (until now) to statically typed compiled languages. This work, while not the first of its kind, significantly advances what can be realistically done for interpreted languages. By slightly instrumenting a language's own interpreter, correct coverage of the full language is almost trivial to obtain. A few quite clever optimizations then make the results fairly practical. The main point is to be able to differentiate between low-level branching of the interpreter and high-level branching of the interpreted program: the goal is good coverage of the program, not the interpreter. All of the necessary concepts (symbolic execution, instrumentation, what interpreter optimizations to turn off, and others) for understanding the presented ideas are clearly explained; related work is cited when appropriate. The evaluation section is thorough yet informative. The paper itself is beautifully written, using clear, concrete, and precise language. At all times, the reader sees the high-level picture that the authors are trying to paint, as well as the details that are presented in each section. It is thoroughly motivated, with deftly analyzed experimental results. This is a significant paper, yet the authors do not overstate their case, carefully inserting appropriate caveats where necessary. This paper is an absolute must-read for anyone remotely interested in interpreted languages. Online Computing Reviews Service

            Access critical reviews of Computing literature here

            Become a reviewer for Computing Reviews.

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!