Abstract
Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to differentially test JVMs by blindly mutating seeding classfiles and then executing the resulting mutants on different JVM binaries for revealing inconsistent behaviors. However, this blind approach is not cost effective in practice because most of the mutants are invalid and redundant. This paper tackles this challenge by introducing classfuzz, a coverage-directed fuzzing approach that focuses on representative classfiles for differential testing of JVMs’ startup processes. Our core insight is to (1) mutate seeding classfiles using a set of predefined mutation operators (mutators) and employ Markov Chain Monte Carlo (MCMC) sampling to guide mutator selection, and (2) execute the mutants on a reference JVM implementation and use coverage uniqueness as a discipline for accepting representative ones. The accepted classfiles are used as inputs to differentially test different JVM implementations and find defects. We have implemented classfuzz and conducted an extensive evaluation of it against existing fuzz testing algorithms. Our evaluation results show that classfuzz can enhance the ratio of discrepancy-triggering classfiles from 1.7% to 11.9%. We have also reported 62 JVM discrepancies, along with the test classfiles, to JVM developers. Many of our reported issues have already been confirmed as JVM defects, and some even match recent clarifications and changes to the Java SE 8 edition of the JVM specification.
- https://lcamtuf.coredump.cx/afl/.Google Scholar
- https://gcc.gnu.org/onlinedocs/gcj/index. html.Google Scholar
- A. Bartel, J. Klein, Y. L. Traon, and M. Monperrus. Dexpler: converting Android Dalvik bytecode to Jimple for static analysis with Soot. In Proceedings of the ACM SIGPLAN International Workshop on State of the Art in Java Program analysis (SOAP 2012), pages 27–38, 2012. Google Scholar
Digital Library
- A. Calvagna and E. Tramontana. Automated conformance testing of Java virtual machines. In Proceedings of the 7th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS), pages 547–552, 2013. Google Scholar
Digital Library
- A. Calvagna and E. Tramontana. Combinatorial validation testing of Java Card byte code verifiers. In Proceedings of the 2013 Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises, pages 347–352, 2013. Google Scholar
Digital Library
- A. Calvagna, A. Fornaia, and E. Tramontana. Combinatorial interaction testing of a Java Card static verifier. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), pages 84–87, 2014. Google Scholar
Digital Library
- Y. Chen and Z. Su. Guided differential testing of certificate validation in SSL/TLS implementations. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), 2015. Google Scholar
Digital Library
- Y. Chen, A. Groce, C. Zhang, W. Wong, X. Fern, E. Eide, and J. Regehr. Taming compiler fuzzers. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13), pages 197–208, 2013. Google Scholar
Digital Library
- S. Chib and E. Greenberg. Understanding the Metropolis-Hastings algorithm. The American Statistician, 49(4):327–335, Nov. 1995.Google Scholar
- A. Gauthier, C. Mazin, J. Iguchi-Cartigny, and J. Lanet. Enhancing fuzzing technique for OKL4 syscalls testing. In Proceedings of the Sixth International Conference on Availability, Reliability and Security (ARES 2011), pages 728–733, 2011. Google Scholar
Digital Library
- P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI 2005), pages 213–223, 2005. Google Scholar
Digital Library
- P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated whitebox fuzz testing. In Proceedings of the Network and Distributed System Security Symposium, NDSS 2008, 2008.Google Scholar
- J. Gosling, B. Joy, G. Steele, G. Bracha, and A. Buckley. The Java Language Specification, Java SE 8 Edition. 2015. URL http://docs.oracle.com/javase/ specs/jls/se8/jls8.pdf. Google Scholar
Digital Library
- A. Groce, G. J. Holzmann, and R. Joshi. Randomized differential testing as a prelude to formal verification. In Proceedings of the International Conference on Software Engineering (ICSE 2007), pages 621–631, 2007. Google Scholar
Digital Library
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In Proceedings of the 21st USENIX Security Symposium (USENIX Security 2012), 2012. Google Scholar
Digital Library
- G. Kondoh and T. Onodera. Finding bugs in Java native interface programs. In Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), pages 109–118, 2008. Google Scholar
Digital Library
- S. C. Kyle, H. Leather, B. Franke, D. Butcher, and S. Monteith. Application of domain-aware binary fuzzing to aid Android virtual machine testing. In Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2015), pages 121–132, 2015. Google Scholar
Digital Library
- V. Le, M. Afshari, and Z. Su. Compiler validation via equivalence modulo inputs. In Proceedings of the Conference on Programming Language Design and Implementation (PLDI 2014), page 25, 2014. Google Scholar
Digital Library
- V. Le, C. Sun, and Z. Su. Finding deep compiler bugs via guided stochastic program mutation. In Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2015), pages 386–399, 2015. Google Scholar
Digital Library
- T. Lindholm, F. Yellin, G. Bracha, and A. Buckley. The Java Virtual Machine Specification: Java SE 7 Edition. 2013. URL http://docs.oracle.com/javase/ specs/jvms/se7/html/index.html. Google Scholar
Digital Library
- T. Lindholm, F. Yellin, G. Bracha, and A. Buckley. The Java Virtual Machine Specification: Java SE 8 Edition. 2015. URL http://docs.oracle.com/javase/ specs/jvms/se8/html/index.html. Google Scholar
Digital Library
- L. Martignoni, R. Paleari, G. F. Roglia, and D. Bruschi. Testing system virtual machines. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis (ISSTA 2010), pages 171–182, 2010. Google Scholar
Digital Library
- W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100–107, 1998.Google Scholar
- N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state calculations by fast computing machines. Journal of Chemical Physics, 21:1087– 1092, 1953.Google Scholar
- J. Meyer and T. Downing. Java Virtual Machine. O’Reilly, 1997. Google Scholar
Digital Library
- G. Misherghi and Z. Su. HDD: hierarchical delta debugging. In Proceedings of the 28th International Conference on Software Engineering (ICSE 2006), pages 142–151, 2006. Google Scholar
Digital Library
- J. Regehr, Y. Chen, P. Cuoq, E. Eide, C. Ellison, and X. Yang. Test-case reduction for C compiler bugs. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2012), pages 335–346, 2012. Google Scholar
Digital Library
- J. Ruderman. Introducing jsfunfuzz. URL http://www.squarefree.com/2007/08/02/ introducing-jsfunfuzz/.Google Scholar
- E. Schkufza, R. Sharma, and A. Aiken. Stochastic superoptimization. In Proceedings of the Architectural Support for Programming Languages and Operating Systems (ASPLOS 2013), pages 305–316, 2013. Google Scholar
Digital Library
- E. G. Sirer and B. N. Bershad. Using production grammars in software testing. In Proceedings of the Second Conference on Domain-Specific Languages (DSL 1999), pages 1–13, 1999. Google Scholar
Digital Library
- G. Tan. JNI light: An operational model for the core JNI. In Proceedings of the 8th Asian Symposium on Programming Languages and Systems (APLAS 2010), pages 114–130, 2010. Google Scholar
Digital Library
- P. Tsankov, M. T. Dashti, and D. A. Basin. SECFUZZ: fuzz-testing security protocols. In Proceedings of the 7th International Workshop on Automation of Software Test (AST 2012), pages 1–7, 2012. Google Scholar
Digital Library
- P. Tsankov, M. T. Dashti, and D. A. Basin. Semi-valid input coverage for fuzz testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 2013), pages 56–66, 2013. Google Scholar
Digital Library
- S. T.V. Oracle JRockit Diagnostics and Troubleshooting Guide, Release R28. 2011. URL http://docs.oracle.com/ cd/E15289_01/doc.40/e15059.pdf.Google Scholar
- R. Vallée-Rai, P. Co, E. Gagnon, L. J. Hendren, P. Lam, and V. Sundaresan. Soot - a Java bytecode optimization framework. In Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative Research, page 13, 1999. Google Scholar
Digital Library
- V. M. Weaver and D. Jones. perf fuzzer: Targeted fuzzing of the perf event open() system call. Technical Report UMAINEVMW-TR-PERF-FUZZER, University of Maine, July 2015.Google Scholar
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in C compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2011), pages 283–294, 2011. Google Scholar
Digital Library
- T. Yoshikawa, K. Shimura, and T. Ozawa. Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software (QSIC 2003), page 20, 2003. Google Scholar
Digital Library
- B. Zhou, H. Okamura, and T. Dohi. Markov Chain Monte Carlo random testing. In Advances in Computer Science and Information Technology, pages 447–456, 2010. Google Scholar
Digital Library
Index Terms
Coverage-directed differential testing of JVM implementations
Recommendations
Coverage-directed differential testing of JVM implementations
PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and ImplementationJava virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to ...
Deep differential testing of JVM implementations
ICSE '19: Proceedings of the 41st International Conference on Software EngineeringThe Java Virtual Machine (JVM) is the cornerstone of the widely-used Java platform. Thus, it is critical to ensure the reliability and robustness of popular JVM implementations. However, little research exists on validating production JVMs. One notable ...
History-driven test program synthesis for JVM testing
ICSE '22: Proceedings of the 44th International Conference on Software EngineeringJava Virtual Machine (JVM) provides the runtime environment for Java programs, which allows Java to be "write once, run anywhere". JVM plays a decisive role in the correctness of all Java programs running on it. Therefore, ensuring the correctness and ...







Comments