The ACM SIGSOFT Paper and Peer Review Quality Initiative: Status Report

Abstract

Scholarly peer review is crucial to science: it not only determines what is published where, but also, indirectly, who is hired, funded and promoted. Yet, virtually every academic has peer review horror stories. Empirical evidence suggests that "peer review is prejudiced, capricious, inefficient, ineffective, and generally unscientific" [1]. An experiment at a major machine learning conference found that peer review was unreliable highlighted that the outcome of peer review can be very noisy [2, 3].

In May 2019, ACM SIGSOFT launched an initiative to improve the quality of research papers and peer reviews at software engineering venues. It has two main components: empirical standards and recommendations for improving review processes.

References

1. P. Ralph. 2015. Practical Suggestions for Improving Scholarly Peer Review Quality and Reducing Cycle Times. Communications of the Association for Information Systems, 38. DOI: https://doi.org/10.17705/1CAIS.03813
2. N. Lawrence and C. Cortes. 2014. The NIPS Experiment. http://inverseprobability.com/2014/12/16/the-nips-experiment.
3. E. Price. 2014. The NIPS experiment. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html.
4. J. McGrath, E. McLachlan and R. Zeller. 2015. Transparency in Research involving Animals: The Basel Declaration and new principles for reporting research in BJP manuscripts. British Journal of Pharmacology, 172, 10, 2427--2432. DOI: https://doi.org/10.1111/bph.12956
5. S. Blackburn et al. 2016. The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations. ACM Trans. Program. Lang. Syst., 38, Article 15. DOI: https://doi.org/10.1145/2983574