skip to main content
article

Cross-language compiler benchmarking: are we fast yet?

Published:01 November 2016Publication History
Skip Abstract Section

Abstract

Comparing the performance of programming languages is difficult because they differ in many aspects including preferred programming abstractions, available frameworks, and their runtime systems. Nonetheless, the question about relative performance comes up repeatedly in the research community, industry, and wider audience of enthusiasts.

This paper presents 14 benchmarks and a novel methodology to assess the compiler effectiveness across language implementations. Using a set of common language abstractions, the benchmarks are implemented in Java, JavaScript, Ruby, Crystal, Newspeak, and Smalltalk. We show that the benchmarks exhibit a wide range of characteristics using language-agnostic metrics. Using four different languages on top of the same compiler, we show that the benchmarks perform similarly and therefore allow for a comparison of compiler effectiveness across languages. Based on anecdotes, we argue that these benchmarks help language implementers to identify performance bugs and optimization potential by comparing to other language implementations.

References

  1. S. M. Blackburn, R. Garner, C. Hoffmann, A. M. Khang, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, B. Moss, A. Phansalkar, D. Stefanovi´c, T. Van-Drunen, D. von Dincklage, and B. Wiedermann. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. In Proc. of OOPSLA, pages 169–190. ACM, 2006. doi: 10.1145/1167473.1167488. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. H.-J. Boehm. Space Efficient Conservative Garbage Collection. In Proc. of PLDI, pages 197–206. ACM, 1993. doi: 10.1145/155090.155109. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. B. Dufour, K. Driesen, L. Hendren, and C. Verbrugge. Dynamic Metrics for Java. In Proc. of OOPSLA, pages 149–168. ACM, 2003. doi: 10.1145/949305.949320. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. P. Havlak. Nesting of Reducible and Irreducible Loops. ACM Trans. Program. Lang. Syst., 19(4):557–567, 1997. doi: 10.1145/262004.262005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. Hoste and L. Eeckhout. Microarchitecture-Independent Workload Characterization. IEEE Micro, 27(3):63–72, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. doi: 10.1109/MM.2007.56.Google ScholarGoogle Scholar
  7. R. Hundt. Loop Recognition in C++/Java/Go/Scala. In Proc. of Scala Days, 2011.Google ScholarGoogle Scholar
  8. T. Kalibera, J. Hagelberg, P. Maj, F. Pizlo, B. Titzer, and J. Vitek. CDx: A Family of Real-time Java Benchmarks. Concurrency and Computation: Practice and Experience, 23 (14):1679–1700, 2011. doi: 10.1002/cpe.1677. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. W. H. Li, D. R. White, and J. Singer. JVM-hosted Languages: They Talk the Talk, but Do They Walk the Walk? In Proc. of PPPJ, pages 101–112. ACM, 2013. doi: 10.1145/ 2500828.2500838. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. P. Ratanaworabhan, B. Livshits, and B. G. Zorn. JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web Applications. In Proc. of WebApps. USENIX, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. G. Richards, S. Lebresne, B. Burg, and J. Vitek. An Analysis of the Dynamic Behavior of JavaScript Programs. In Proc. of PLDI, pages 1–12. ACM, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. doi: 10.1145/ 1809028.1806598.Google ScholarGoogle Scholar
  13. M. Richards. Bench, 1999.Google ScholarGoogle Scholar
  14. A. Sarimbekov, A. Podzimek, L. Bulej, Y. Zheng, N. Ricci, and W. Binder. Characteristics of Dynamic JVM Languages. In Proc. of VMIL, pages 11–20. ACM, 2013. doi: 10.1145/ 2542142.2542144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. A. Sewe, M. Mezini, A. Sarimbekov, and W. Binder. Da Capo Con Scala: Design and Analysis of a Scala Benchmark Suite for the Java Virtual Machine. In Proc. of OOPSLA, pages 657–676. ACM, 2011. doi: 10.1145/2048066.2048118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. K. Shiv, K. Chow, Y. Wang, and D. Petrochenko. SPECjvm2008 Performance Characterization. In Proc. of SPEC Benchmark Workshop, pages 17–35. Springer, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. doi: 10.1007/978-3-540-93799-9 2.Google ScholarGoogle Scholar
  18. M. Wolczko. Benchmarking Java with Richards and Deltablue, 2013.Google ScholarGoogle Scholar
  19. T. Würthinger, C. Wimmer, A. Wöß, L. Stadler, G. Duboscq, C. Humer, G. Richards, D. Simon, and M. Wolczko. One VM to Rule Them All. In Proc. of Onward!, pages 187–204. ACM, 2013. doi: 10.1145/2509578.2509581. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. A. Wöß, C. Wirth, D. Bonetta, C. Seaton, C. Humer, and H. Mössenböck. An Object Storage Model for the Truffle Language Implementation Framework. In Proc. of PPPJ, pages 133–144. ACM, 2014. doi: 10.1145/2647508.2647517. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. T. Würthinger, A. Wöß, L. Stadler, G. Duboscq, D. Simon, and C. Wimmer. Self-Optimizing AST Interpreters. In Proc. of DLS, pages 73–82, 2012. doi: 10.1145/2384577.2384587. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Cross-language compiler benchmarking: are we fast yet?

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM SIGPLAN Notices
      ACM SIGPLAN Notices  Volume 52, Issue 2
      DLS '16
      February 2017
      131 pages
      ISSN:0362-1340
      EISSN:1558-1160
      DOI:10.1145/3093334
      Issue’s Table of Contents
      • cover image ACM Conferences
        DLS 2016: Proceedings of the 12th Symposium on Dynamic Languages
        November 2016
        131 pages
        ISBN:9781450344456
        DOI:10.1145/2989225

      Copyright © 2016 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 1 November 2016

      Check for updates

      Qualifiers

      • article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!