skip to main content
research-article

Compiler validation via equivalence modulo inputs

Published:09 June 2014Publication History
Skip Abstract Section

Abstract

We introduce equivalence modulo inputs (EMI), a simple, widely applicable methodology for validating optimizing compilers. Our key insight is to exploit the close interplay between (1) dynamically executing a program on some test inputs and (2) statically compiling the program to work on all possible inputs. Indeed, the test inputs induce a natural collection of the original program's EMI variants, which can help differentially test any compiler and specifically target the difficult-to-find miscompilations.

To create a practical implementation of EMI for validating C compilers, we profile a program's test executions and stochastically prune its unexecuted code. Our extensive testing in eleven months has led to 147 confirmed, unique bug reports for GCC and LLVM alone. The majority of those bugs are miscompilations, and more than 100 have already been fixed.

Beyond testing compilers, EMI can be adapted to validate program transformation and analysis systems in general. This work opens up this exciting, new direction.

References

  1. ACE. SuperTest compiler test and validation suite. http://www.ace.nl/compiler/supertest.html.Google ScholarGoogle Scholar
  2. A. Balestrat. CCG: A random C code generator. https://github.com/Merkil/ccg/.Google ScholarGoogle Scholar
  3. S. Bansal and A. Aiken. Automatic generation of peephole superoptimizers. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 394--403, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Y. Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, and J. Regehr. Taming compiler fuzzers. In Proceedings of the 2013 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 197--208, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. P. Cuoq, B. Monate, A. Pacalet, V. Prevosto, J. Regehr, B. Yakobowski, and X. Yang. Testing static analyzers with randomly generated programs. In A. Goodloe and S. Person, editors, NASA Formal Methods, volume 7226 of Lecture Notes in Computer Science, pages 120--125. Springer Berlin Heidelberg, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. Ellison and G. Rosu. An executable formal semantics of C with applications. In Proceedings of the 39th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 533--544, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. GNU Compiler Collection. gcov. http://gcc.gnu.org/onlinedocs/gcc/Gcov.html.Google ScholarGoogle Scholar
  8. A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr. Swarm testing. In International Symposium on Software Testing and Analysis (ISSTA), pages 78--88, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. T. Hoare. The verifying compiler: A grand challenge for computing research. In Modular Programming Languages, pages 25--35. Springer, 2003.Google ScholarGoogle ScholarCross RefCross Ref
  10. C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In Proceedings of the 21st USENIX Security Symposium, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. R. Joshi, G. Nelson, and K. H. Randall. Denali: A goal-directed superoptimizer. In Proceedings of the 2002 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 304--314, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. X. Leroy. Formal certification of a compiler back-end, or: programming a compiler with a proof assistant. In Proceedings of the 33rd ACM Symposium on Principles of Programming Languages (POPL), pages 42--54. ACM Press, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. X. Leroy. A formally verified compiler back-end. Journal of Automated Reasoning, 43(4):363--446, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. G. Malecha, G. Morrisett, A. Shinnar, and R. Wisnesky. Toward a verified relational database management system. In Proceedings of the 37th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 237--248, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. H. Massalin. Superoptimizer -- A look at the smallest program. In Proceedings of the Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 122--126, 1987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100--107, 1998.Google ScholarGoogle Scholar
  17. S. McPeak, D. S. Wilkerson, and S. Goldsmith. Berkeley Delta. http://delta.tigris.org/.Google ScholarGoogle Scholar
  18. E. Nagai, H. Awazu, N. Ishiura, and N. Takeda. Random testing of C compilers targeting arithmetic optimization. In Workshop on Synthesis And System Integration of Mixed Information Technologies (SASIMI 2012), pages 48--53, 2012.Google ScholarGoogle Scholar
  19. E. Nagai, A. Hashimoto, and N. Ishiura. Scaling up size and number of expressions in random testing of arithmetic optimization of C compilers. In Workshop on Synthesis And System Integration of Mixed Information Technologies (SASIMI 2013), pages 88--93, 2013.Google ScholarGoogle Scholar
  20. G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the 2000 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 83--94, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Plum Hall, Inc. The Plum Hall Validation Suite for C. http://www.plumhall.com/stec.html.Google ScholarGoogle Scholar
  22. A. Pnueli, M. Siegel, and E. Singerman. Translation validation. In Proceedings of the 4th International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS), pages 151--166, London, UK, UK, 1998. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. J. Regehr. Embedded in academia. http://blog.regehr.org/.Google ScholarGoogle Scholar
  24. J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-case reduction for C compiler bugs. In Proceedings of the 2012 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 335--346, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. R. Tate, M. Stepp, Z. Tatlock, and S. Lerner. Equality saturation: a new approach to optimization. In Proceedings of the 36th ACM SIGPLAN-SIGACT symposium on Principles of Programming Languages, pages 264--276, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. The Clang Team. Clang 3.4 documentation: LibTooling. http://clang.llvm.org/docs/LibTooling.html.Google ScholarGoogle Scholar
  27. J.-B. Tristan, P. Govereau, and G. Morrisett. Evaluating value-graph translation validation for LLVM. In Proceedings of the 2011 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 295--305, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Proceedings of the 2011 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 283--294, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. C. Zhao, Y. Xue, Q. Tao, L. Guo, and Z. Wang. Automated test program generation for an industrial optimizing compiler. In ICSE Workshop on Automation of Software Test (AST), pages 36--43, 2009.Google ScholarGoogle Scholar
  30. J. Zhao, S. Nagarakatte, M. M. K. Martin, and S. Zdancewic. Formal verification of SSA-based optimizations for LLVM. In Proceedings of the 2013 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 175--186, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Compiler validation via equivalence modulo inputs

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM SIGPLAN Notices
        ACM SIGPLAN Notices  Volume 49, Issue 6
        PLDI '14
        June 2014
        598 pages
        ISSN:0362-1340
        EISSN:1558-1160
        DOI:10.1145/2666356
        • Editor:
        • Andy Gill
        Issue’s Table of Contents
        • cover image ACM Conferences
          PLDI '14: Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
          June 2014
          619 pages
          ISBN:9781450327848
          DOI:10.1145/2594291

        Copyright © 2014 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 9 June 2014

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!