skip to main content
10.1145/1449764.1449794acmconferencesArticle/Chapter ViewAbstractPublication PagessplashConference Proceedingsconference-collections
research-article

Java performance evaluation through rigorous replay compilation

Published:19 October 2008Publication History

ABSTRACT

A managed runtime environment, such as the Java virtual machine, is non-trivial to benchmark. Java performance is affected in various complex ways by the application and its input, as well as by the virtual machine (JIT optimizer, garbage collector, thread scheduler, etc.). In addition, non-determinism due to timer-based sampling for JIT optimization, thread scheduling, and various system effects further complicate the Java performance benchmarking process.

Replay compilation is a recently introduced Java performance analysis methodology that aims at controlling non-determinism to improve experimental repeatability. The key idea of replay compilation is to control the compilation load during experimentation by inducing a pre-recorded compilation plan at replay time. Replay compilation also enables teasing apart performance effects of the application versus the virtual machine.

This paper argues that in contrast to current practice which uses a single compilation plan at replay time, multiple compilation plans add statistical rigor to the replay compilation methodology. By doing so, replay compilation better accounts for the variability observed in compilation load across compilation plans. In addition, we propose matched-pair comparison for statistical data analysis. Matched-pair comparison considers the performance measurements per compilation plan before and after an innovation of interest as a pair, which enables limiting the number of compilation plans needed for accurate performance analysis compared to statistical analysis assuming unpaired measurements.

References

  1. M. Arnold, S. Fink, D. Grove, M. Hind, and P. F. Sweeney. Adaptive optimization in the Jalapeno JVM. In OOPSLA, pages 47--65, Oct. 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. BEA. BEA JRockit: Java for the enterprise. Technical white paper. http://www.bea.com, Jan. 2006.Google ScholarGoogle Scholar
  3. S. Blackburn, P. Cheng, and K. S. McKinley. Myths and reality: The performance impact of garbage collection. In SIGMETRICS, pages 25--36, June 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Blackburn, P. Cheng, and K. S. McKinley. Oil and water? High performance garbage collection in Java with JMTk. In ICSE, pages 137--146, May 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovic, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In OOPSLA, pages 169--190, Oct. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. S. M. Blackburn, M. Hertz, K. S. McKinley, J. E. B. Moss, and T. Yang. Profile-based pretenuring. ACM Trans. Program. Lang. Syst., 29(1):2, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. S. M. Blackburn and A. L. Hosking. Barriers: Friend or foe? In ISMM, pages 143--151, Oct. 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. M. D. Bond and K. S. McKinley. Continuous path and edge profiling. In MICRO, pages 130--140, Dec. 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. M. D. Bond and K. S. McKinley. Probabilistic calling context. In OOPSLA, pages 97--112, Oct. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. M. D. Bond and K. S. McKinley. Bell: Bit-encoding online memory leak detection. In ASPLOS, pages 61--72, Oct. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. D. Buytaert, A. Georges, M. Hind, M. Arnold, L. Eeckhout, and K. De Bosschere. Using HPM-sampling to drive dynamic compilation. In OOPSLA, pages 553--568, Oct. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. J. Cavazos and J. E. B. Moss. Inducing heuristics to decide whether to schedule. In PLDI, pages 183--194, June 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. M. Cierniak, M. Eng, N. Glew, B. Lewis, and J. Stichnoth. The open runtime platform: A flexible high-performance managed runtime environment. Intel Technology Journal, 7(1):5--18, 2003.Google ScholarGoogle Scholar
  14. L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In OOPSLA, pages 169--186, Nov. 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In OOPSLA, pages 57--76, Oct. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. Gu, C. Verbrugge, and E. M. Gagnon. Relative factors in performance analysis of Java virtual machines. In VEE, pages 111--121, June 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. S. Z. Guyer, K. S. McKinley, and D. Frampton. Free-me: A static analysis for automatic individual object reclamation. In PLDI, pages 364--375, June 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. M. Hauswirth, P. F. Sweeney, A. Diwan, and M. Hind. Vertical profiling: Understanding the behavior of object-oriented applications. In OOPSLA, pages 251--269, Oct. 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. X. Huang, S. M. Blackburn, K. S. McKinley, J. E. B. Moss, Z. Wang, and P. Cheng. The garbage collection advantage: Improving program locality. In OOPSLA, pages 69--80, Oct. 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. R. A. Johnson and D. W. Wichern. Applied Multivariate Statistical Analysis. Prentice Hall, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. C. Krintz and B. Calder. Using annotations to reduce dynamic optimization time. In PLDI, pages 156--167, May 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. B. Lee, K. Resnick, M. D. Bond, and K. S. McKinley. Correcting the dynamic call graph using control-flow constraints. In CC, pages 80--95, March 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. D. J. Lilja. Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. J. Maebe, D. Buytaert, L. Eeckhout, and K. De Bosschere. Javana: A system for building customized Java program analysis tools. In OOPSLA, pages 153--168, Oct. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. V. Sundaresan, D. Maier, P. Ramarao, and M. Stoodley. Experiences with multithreading and dynamic class loading in a Java just-in-time compiler. In CGO, pages 87--97, Mar. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. J. Neter, M. H. Kutner, W. Wasserman, and C. J. Nachtsheim. Applied Linear Statistical Models. McGraw-Hill, 1996.Google ScholarGoogle Scholar
  27. K. Ogata, T. Onodera, K. Kawachiya, H. Komatsu, and T. Nakatani. Replay compilation: Improving debuggability of a just-in-time compiler. In OOPSLA, pages 241--252, Oct. 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. M. Paleczny, C. Vick, and C. Click. The Java Hotspot server compiler. In JVM, pages 1--12, Apr. 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. N. Sachindran, and J. E. B. Moss. Mark-copy: fast copying GC with less space overhead. In OOPSLA, pages 326--343, Nov. 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. N. Sachindran, J. E. B. Moss, and E. D. Berger. MC2: high-performance garbage collection for memory-constrained environments. In OOPSLA, pages 81--98, Oct. 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. F. T. Schneider, M. Payer, and T. R. Gross. Online optimizations driven by hardware performance monitoring. In PLDI, pages 373--382, June 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. S. Soman, C. Krintz, and D. F. Bacon. Dynamic selection of application-specific garbage collectors. In ISMM, pages 49--60, June 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Standard Performance Evaluation Corporation. SPECjvm98 Benchmarks. http://www.spec.org/jvm98.Google ScholarGoogle Scholar
  34. T. Suganuma, T. Yasue, M. Kawahito, H. Komatsu, and T. Nakatani. Design and evaluation of dynamic optimizations for a Java just-in-time compiler. In TOPLAS, 27(4):732--785, July 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. P. F. Sweeney, M. Hauswirth, B. Cahoon, P. Cheng, A. Diwan, D. Grove, and M. Hind. Using hardware performance monitors to understand the behavior of Java applications. In VM, pages 57--72, May 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. J. Whaley. A portable sampling--based profiler for Java virtual machines. In Proceedings of the ACM 2000 Conference on Java Grande, pages 78--87, June 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. T. Yang, M. Hertz, E. D. Berger, S. F. Kaplan, and J. E. B. Moss. Automatic heap sizing: taking real memory into account. In ISMM, pages 61--72, June 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Java performance evaluation through rigorous replay compilation

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!