skip to main content
article

Coverage-directed differential testing of JVM implementations

Published:02 June 2016Publication History
Skip Abstract Section

Abstract

Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to differentially test JVMs by blindly mutating seeding classfiles and then executing the resulting mutants on different JVM binaries for revealing inconsistent behaviors. However, this blind approach is not cost effective in practice because most of the mutants are invalid and redundant. This paper tackles this challenge by introducing classfuzz, a coverage-directed fuzzing approach that focuses on representative classfiles for differential testing of JVMs’ startup processes. Our core insight is to (1) mutate seeding classfiles using a set of predefined mutation operators (mutators) and employ Markov Chain Monte Carlo (MCMC) sampling to guide mutator selection, and (2) execute the mutants on a reference JVM implementation and use coverage uniqueness as a discipline for accepting representative ones. The accepted classfiles are used as inputs to differentially test different JVM implementations and find defects. We have implemented classfuzz and conducted an extensive evaluation of it against existing fuzz testing algorithms. Our evaluation results show that classfuzz can enhance the ratio of discrepancy-triggering classfiles from 1.7% to 11.9%. We have also reported 62 JVM discrepancies, along with the test classfiles, to JVM developers. Many of our reported issues have already been confirmed as JVM defects, and some even match recent clarifications and changes to the Java SE 8 edition of the JVM specification.

References

  1. https://lcamtuf.coredump.cx/afl/.Google ScholarGoogle Scholar
  2. https://gcc.gnu.org/onlinedocs/gcj/index. html.Google ScholarGoogle Scholar
  3. A. Bartel, J. Klein, Y. L. Traon, and M. Monperrus. Dexpler: converting Android Dalvik bytecode to Jimple for static analysis with Soot. In Proceedings of the ACM SIGPLAN International Workshop on State of the Art in Java Program analysis (SOAP 2012), pages 27–38, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A. Calvagna and E. Tramontana. Automated conformance testing of Java virtual machines. In Proceedings of the 7th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS), pages 547–552, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. A. Calvagna and E. Tramontana. Combinatorial validation testing of Java Card byte code verifiers. In Proceedings of the 2013 Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, pages 347–352, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. Calvagna, A. Fornaia, and E. Tramontana. Combinatorial interaction testing of a Java Card static verifier. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), pages 84–87, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Y. Chen and Z. Su. Guided differential testing of certificate validation in SSL/TLS implementations. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Y. Chen, A. Groce, C. Zhang, W. Wong, X. Fern, E. Eide, and J. Regehr. Taming compiler fuzzers. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13), pages 197–208, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. S. Chib and E. Greenberg. Understanding the Metropolis-Hastings algorithm. The American Statistician, 49(4):327–335, Nov. 1995.Google ScholarGoogle Scholar
  10. A. Gauthier, C. Mazin, J. Iguchi-Cartigny, and J. Lanet. Enhancing fuzzing technique for OKL4 syscalls testing. In Proceedings of the Sixth International Conference on Availability, Reliability and Security (ARES 2011), pages 728–733, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI 2005), pages 213–223, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2008, 2008.Google ScholarGoogle Scholar
  13. J. Gosling, B. Joy, G. Steele, G. Bracha, and A. Buckley. The Java Language Specification, Java SE 8 Edition. 2015. URL http://docs.oracle.com/javase/ specs/jls/se8/jls8.pdf. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. A. Groce, G. J. Holzmann, and R. Joshi. Randomized differential testing as a prelude to formal verification. In Proceedings of the International Conference on Software Engineering (ICSE 2007), pages 621–631, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In Proceedings of the 21st USENIX Security Symposium (USENIX Security 2012), 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. G. Kondoh and T. Onodera. Finding bugs in Java native interface programs. In Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), pages 109–118, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. S. C. Kyle, H. Leather, B. Franke, D. Butcher, and S. Monteith. Application of domain-aware binary fuzzing to aid Android virtual machine testing. In Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2015), pages 121–132, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. V. Le, M. Afshari, and Z. Su. Compiler validation via equivalence modulo inputs. In Proceedings of the Conference on Programming Language Design and Implementation (PLDI 2014), page 25, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. V. Le, C. Sun, and Z. Su. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2015), pages 386–399, 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. T. Lindholm, F. Yellin, G. Bracha, and A. Buckley. The Java Virtual Machine Specification: Java SE 7 Edition. 2013. URL http://docs.oracle.com/javase/ specs/jvms/se7/html/index.html. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. T. Lindholm, F. Yellin, G. Bracha, and A. Buckley. The Java Virtual Machine Specification: Java SE 8 Edition. 2015. URL http://docs.oracle.com/javase/ specs/jvms/se8/html/index.html. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. L. Martignoni, R. Paleari, G. F. Roglia, and D. Bruschi. Testing system virtual machines. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis (ISSTA 2010), pages 171–182, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100–107, 1998.Google ScholarGoogle Scholar
  24. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21:1087– 1092, 1953.Google ScholarGoogle Scholar
  25. J. Meyer and T. Downing. Java Virtual Machine. O’Reilly, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. G. Misherghi and Z. Su. HDD: hierarchical delta debugging. In Proceedings of the 28th International Conference on Software Engineering (ICSE 2006), pages 142–151, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-case reduction for C compiler bugs. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2012), pages 335–346, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. Ruderman. Introducing jsfunfuzz. URL http://www.squarefree.com/2007/08/02/ introducing-jsfunfuzz/.Google ScholarGoogle Scholar
  29. E. Schkufza, R. Sharma, and A. Aiken. Stochastic superoptimization. In Proceedings of the Architectural Support for Programming Languages and Operating Systems (ASPLOS 2013), pages 305–316, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. E. G. Sirer and B. N. Bershad. Using production grammars in software testing. In Proceedings of the Second Conference on Domain-Specific Languages (DSL 1999), pages 1–13, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. G. Tan. JNI light: An operational model for the core JNI. In Proceedings of the 8th Asian Symposium on Programming Languages and Systems (APLAS 2010), pages 114–130, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. P. Tsankov, M. T. Dashti, and D. A. Basin. SECFUZZ: fuzz-testing security protocols. In Proceedings of the 7th International Workshop on Automation of Software Test (AST 2012), pages 1–7, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. P. Tsankov, M. T. Dashti, and D. A. Basin. Semi-valid input coverage for fuzz testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 2013), pages 56–66, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. S. T.V. Oracle JRockit Diagnostics and Troubleshooting Guide, Release R28. 2011. URL http://docs.oracle.com/ cd/E15289_01/doc.40/e15059.pdf.Google ScholarGoogle Scholar
  35. R. Vallée-Rai, P. Co, E. Gagnon, L. J. Hendren, P. Lam, and V. Sundaresan. Soot - a Java bytecode optimization framework. In Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative Research, page 13, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. V. M. Weaver and D. Jones. perf fuzzer: Targeted fuzzing of the perf event open() system call. Technical Report UMAINEVMW-TR-PERF-FUZZER, University of Maine, July 2015.Google ScholarGoogle Scholar
  37. X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2011), pages 283–294, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. T. Yoshikawa, K. Shimura, and T. Ozawa. Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software (QSIC 2003), page 20, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. B. Zhou, H. Okamura, and T. Dohi. Markov Chain Monte Carlo random testing. In Advances in Computer Science and Information Technology, pages 447–456, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Coverage-directed differential testing of JVM implementations

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGPLAN Notices
          ACM SIGPLAN Notices  Volume 51, Issue 6
          PLDI '16
          June 2016
          726 pages
          ISSN:0362-1340
          EISSN:1558-1160
          DOI:10.1145/2980983
          • Editor:
          • Andy Gill
          Issue’s Table of Contents
          • cover image ACM Conferences
            PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation
            June 2016
            726 pages
            ISBN:9781450342612
            DOI:10.1145/2908080
            • General Chair:
            • Chandra Krintz,
            • Program Chair:
            • Emery Berger

          Copyright © 2016 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 2 June 2016

          Check for updates

          Qualifiers

          • article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!