skip to main content
keynote

The case for the three R's of systems research: repeatability, reproducibility and rigor

Published:01 March 2014Publication History
Skip Abstract Section

Abstract

Computer systems research spans sub-disciplines that include embedded systems, programming languages, network- ing, and operating systems. In this talk my contention is that a number of structural factors inhibit quality systems re- search. Symptoms of the problem include unrepeatable and unreproduced results as well as results that are either devoid of meaning or that measure the wrong thing. I will illustrate the impact of these issues on our research output with examples from the development and empirical evaluation of the Schism real-time garbage collection algorithm that is shipped with the FijiVM -- a Java virtual machine for embedded and mobile devices. I will argue that our field should fos- ter: repetition of results, independent reproduction, as well as rigorous evaluation. I will outline some baby steps taken by several computer conferences. In particular I will focus on the introduction of Artifact Evaluation Committees or AECs to ECOOP, OOPLSA, PLDI and soon POPL. The goal of the AECs is to encourage author to package the soft- ware artifacts that they used to support the claims made in their paper and to submit these artifacts for evaluation. AECs were carefully designed to provide positive feedback to the authors that take the time to create repeatable research.

References

  1. J. Vitek, T. Kalibera. Repeatability, Reproducibility and Rigor in Systems Research. In EMSOFT, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. T. Kalibera, F. Pizlo, A. Hosking, J. Vitek. Scheduling real-time garbage collection on uniprocessors. ACM Trans. Comput. Syst. 29 (3), 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. F. Pizlo, E. Blanton, A. Hosking, P. Maj, J. Vitek, L. Ziarek. Schis: Fragmentation-tolerant real-time garbage collection. In PLDI 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. The case for the three R's of systems research: repeatability, reproducibility and rigor

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!