Abstract
Computer systems research spans sub-disciplines that include embedded systems, programming languages, network- ing, and operating systems. In this talk my contention is that a number of structural factors inhibit quality systems re- search. Symptoms of the problem include unrepeatable and unreproduced results as well as results that are either devoid of meaning or that measure the wrong thing. I will illustrate the impact of these issues on our research output with examples from the development and empirical evaluation of the Schism real-time garbage collection algorithm that is shipped with the FijiVM -- a Java virtual machine for embedded and mobile devices. I will argue that our field should fos- ter: repetition of results, independent reproduction, as well as rigorous evaluation. I will outline some baby steps taken by several computer conferences. In particular I will focus on the introduction of Artifact Evaluation Committees or AECs to ECOOP, OOPLSA, PLDI and soon POPL. The goal of the AECs is to encourage author to package the soft- ware artifacts that they used to support the claims made in their paper and to submit these artifacts for evaluation. AECs were carefully designed to provide positive feedback to the authors that take the time to create repeatable research.
- J. Vitek, T. Kalibera. Repeatability, Reproducibility and Rigor in Systems Research. In EMSOFT, 2011. Google Scholar
Digital Library
- T. Kalibera, F. Pizlo, A. Hosking, J. Vitek. Scheduling real-time garbage collection on uniprocessors. ACM Trans. Comput. Syst. 29 (3), 2001. Google Scholar
Digital Library
- F. Pizlo, E. Blanton, A. Hosking, P. Maj, J. Vitek, L. Ziarek. Schis: Fragmentation-tolerant real-time garbage collection. In PLDI 2010. Google Scholar
Digital Library
Index Terms
The case for the three R's of systems research: repeatability, reproducibility and rigor
Recommendations
The case for the three R's of systems research: repeatability, reproducibility and rigor
VEE '14: Proceedings of the 10th ACM SIGPLAN/SIGOPS international conference on Virtual execution environmentsComputer systems research spans sub-disciplines that include embedded systems, programming languages, network- ing, and operating systems. In this talk my contention is that a number of structural factors inhibit quality systems re- search. Symptoms of ...
Evaluating the artifacts of SIGCOMM papers
A growing fraction of the papers published by CCR and at SIGCOMM-sponsored conferences include artifacts such as software or datasets. Besides CCR, these artifacts were rarely evaluated. During the last months of 2018, we organised two different ...







Comments