ABSTRACT
A managed runtime environment, such as the Java virtual machine, is non-trivial to benchmark. Java performance is affected in various complex ways by the application and its input, as well as by the virtual machine (JIT optimizer, garbage collector, thread scheduler, etc.). In addition, non-determinism due to timer-based sampling for JIT optimization, thread scheduling, and various system effects further complicate the Java performance benchmarking process.
Replay compilation is a recently introduced Java performance analysis methodology that aims at controlling non-determinism to improve experimental repeatability. The key idea of replay compilation is to control the compilation load during experimentation by inducing a pre-recorded compilation plan at replay time. Replay compilation also enables teasing apart performance effects of the application versus the virtual machine.
This paper argues that in contrast to current practice which uses a single compilation plan at replay time, multiple compilation plans add statistical rigor to the replay compilation methodology. By doing so, replay compilation better accounts for the variability observed in compilation load across compilation plans. In addition, we propose matched-pair comparison for statistical data analysis. Matched-pair comparison considers the performance measurements per compilation plan before and after an innovation of interest as a pair, which enables limiting the number of compilation plans needed for accurate performance analysis compared to statistical analysis assuming unpaired measurements.
- M. Arnold, S. Fink, D. Grove, M. Hind, and P. F. Sweeney. Adaptive optimization in the Jalapeno JVM. In OOPSLA, pages 47--65, Oct. 2000. Google Scholar
Digital Library
- BEA. BEA JRockit: Java for the enterprise. Technical white paper. http://www.bea.com, Jan. 2006.Google Scholar
- S. Blackburn, P. Cheng, and K. S. McKinley. Myths and reality: The performance impact of garbage collection. In SIGMETRICS, pages 25--36, June 2004. Google Scholar
Digital Library
- S. Blackburn, P. Cheng, and K. S. McKinley. Oil and water? High performance garbage collection in Java with JMTk. In ICSE, pages 137--146, May 2004. Google Scholar
Digital Library
- S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovic, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In OOPSLA, pages 169--190, Oct. 2006. Google Scholar
Digital Library
- S. M. Blackburn, M. Hertz, K. S. McKinley, J. E. B. Moss, and T. Yang. Profile-based pretenuring. ACM Trans. Program. Lang. Syst., 29(1):2, 2007. Google Scholar
Digital Library
- S. M. Blackburn and A. L. Hosking. Barriers: Friend or foe? In ISMM, pages 143--151, Oct. 2004. Google Scholar
Digital Library
- M. D. Bond and K. S. McKinley. Continuous path and edge profiling. In MICRO, pages 130--140, Dec. 2005. Google Scholar
Digital Library
- M. D. Bond and K. S. McKinley. Probabilistic calling context. In OOPSLA, pages 97--112, Oct. 2007. Google Scholar
Digital Library
- M. D. Bond and K. S. McKinley. Bell: Bit-encoding online memory leak detection. In ASPLOS, pages 61--72, Oct. 2006. Google Scholar
Digital Library
- D. Buytaert, A. Georges, M. Hind, M. Arnold, L. Eeckhout, and K. De Bosschere. Using HPM-sampling to drive dynamic compilation. In OOPSLA, pages 553--568, Oct. 2007. Google Scholar
Digital Library
- J. Cavazos and J. E. B. Moss. Inducing heuristics to decide whether to schedule. In PLDI, pages 183--194, June 2004. Google Scholar
Digital Library
- M. Cierniak, M. Eng, N. Glew, B. Lewis, and J. Stichnoth. The open runtime platform: A flexible high-performance managed runtime environment. Intel Technology Journal, 7(1):5--18, 2003.Google Scholar
- L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In OOPSLA, pages 169--186, Nov. 2003. Google Scholar
Digital Library
- A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In OOPSLA, pages 57--76, Oct. 2007. Google Scholar
Digital Library
- D. Gu, C. Verbrugge, and E. M. Gagnon. Relative factors in performance analysis of Java virtual machines. In VEE, pages 111--121, June 2006. Google Scholar
Digital Library
- S. Z. Guyer, K. S. McKinley, and D. Frampton. Free-me: A static analysis for automatic individual object reclamation. In PLDI, pages 364--375, June 2006. Google Scholar
Digital Library
- M. Hauswirth, P. F. Sweeney, A. Diwan, and M. Hind. Vertical profiling: Understanding the behavior of object-oriented applications. In OOPSLA, pages 251--269, Oct. 2004. Google Scholar
Digital Library
- X. Huang, S. M. Blackburn, K. S. McKinley, J. E. B. Moss, Z. Wang, and P. Cheng. The garbage collection advantage: Improving program locality. In OOPSLA, pages 69--80, Oct. 2004. Google Scholar
Digital Library
- R. A. Johnson and D. W. Wichern. Applied Multivariate Statistical Analysis. Prentice Hall, 2002. Google Scholar
Digital Library
- C. Krintz and B. Calder. Using annotations to reduce dynamic optimization time. In PLDI, pages 156--167, May 2001. Google Scholar
Digital Library
- B. Lee, K. Resnick, M. D. Bond, and K. S. McKinley. Correcting the dynamic call graph using control-flow constraints. In CC, pages 80--95, March 2007. Google Scholar
Digital Library
- D. J. Lilja. Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press, 2000. Google Scholar
Digital Library
- J. Maebe, D. Buytaert, L. Eeckhout, and K. De Bosschere. Javana: A system for building customized Java program analysis tools. In OOPSLA, pages 153--168, Oct. 2006. Google Scholar
Digital Library
- V. Sundaresan, D. Maier, P. Ramarao, and M. Stoodley. Experiences with multithreading and dynamic class loading in a Java just-in-time compiler. In CGO, pages 87--97, Mar. 2006. Google Scholar
Digital Library
- J. Neter, M. H. Kutner, W. Wasserman, and C. J. Nachtsheim. Applied Linear Statistical Models. McGraw-Hill, 1996.Google Scholar
- K. Ogata, T. Onodera, K. Kawachiya, H. Komatsu, and T. Nakatani. Replay compilation: Improving debuggability of a just-in-time compiler. In OOPSLA, pages 241--252, Oct. 2006. Google Scholar
Digital Library
- M. Paleczny, C. Vick, and C. Click. The Java Hotspot server compiler. In JVM, pages 1--12, Apr. 2001. Google Scholar
Digital Library
- N. Sachindran, and J. E. B. Moss. Mark-copy: fast copying GC with less space overhead. In OOPSLA, pages 326--343, Nov. 2003. Google Scholar
Digital Library
- N. Sachindran, J. E. B. Moss, and E. D. Berger. MC2: high-performance garbage collection for memory-constrained environments. In OOPSLA, pages 81--98, Oct. 2004. Google Scholar
Digital Library
- F. T. Schneider, M. Payer, and T. R. Gross. Online optimizations driven by hardware performance monitoring. In PLDI, pages 373--382, June 2007. Google Scholar
Digital Library
- S. Soman, C. Krintz, and D. F. Bacon. Dynamic selection of application-specific garbage collectors. In ISMM, pages 49--60, June 2004. Google Scholar
Digital Library
- Standard Performance Evaluation Corporation. SPECjvm98 Benchmarks. http://www.spec.org/jvm98.Google Scholar
- T. Suganuma, T. Yasue, M. Kawahito, H. Komatsu, and T. Nakatani. Design and evaluation of dynamic optimizations for a Java just-in-time compiler. In TOPLAS, 27(4):732--785, July 2005. Google Scholar
Digital Library
- P. F. Sweeney, M. Hauswirth, B. Cahoon, P. Cheng, A. Diwan, D. Grove, and M. Hind. Using hardware performance monitors to understand the behavior of Java applications. In VM, pages 57--72, May 2004. Google Scholar
Digital Library
- J. Whaley. A portable sampling--based profiler for Java virtual machines. In Proceedings of the ACM 2000 Conference on Java Grande, pages 78--87, June 2000. Google Scholar
Digital Library
- T. Yang, M. Hertz, E. D. Berger, S. F. Kaplan, and J. E. B. Moss. Automatic heap sizing: taking real memory into account. In ISMM, pages 61--72, June 2004. Google Scholar
Digital Library
Index Terms
Java performance evaluation through rigorous replay compilation
Recommendations
Statistically rigorous java performance evaluation
OOPSLA '07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems, languages and applicationsJava performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes ...
Java performance evaluation through rigorous replay compilation
A managed runtime environment, such as the Java virtual machine, is non-trivial to benchmark. Java performance is affected in various complex ways by the application and its input, as well as by the virtual machine (JIT optimizer, garbage collector, ...
Replay compilation: improving debuggability of a just-in-time compiler
OOPSLA '06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applicationsThe performance of Java has been tremendously improved by the advance of Just-in-Time (JIT) compilation technologies. However, debugging such a dynamic compiler is much harder than a static compiler. Recompiling the problematic method to produce a ...







Comments