Abstract
In this paper, we identify trends about, benefits from, and barriers to performing user evaluations in software engineering research. From a corpus of over 3,000 papers spanning ten years, we report on various subtypes of user evaluations (e.g., coding tasks vs. questionnaires) and relate user evaluations to paper topics (e.g., debugging vs. technology transfer). We identify the external measures of impact, such as best paper awards and citation counts, that are correlated with the presence of user evaluations. We complement this with a survey of over 100 researchers from over 40 different universities and labs in which we identify a set of perceived barriers to performing user evaluations.
- J. W. Backus. The IBM 701 Speedcoding system. J. ACM, 1:4--6, January 1954. Google Scholar
Digital Library
- V. Basili, R. Selby Jr, and D. Hutchens. Experimentation in SE. In P. Oman and S. L. Pfleeger, editors, Applying Software Metrics. Wiley-IEEE Computer Society, 1997.Google Scholar
- I. Campbell. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine, 26:3661--3675, 2007.Google Scholar
Cross Ref
- T. Couronne, M. Zelkowitz, and D. Wallace. Experimental validation in software engineering. Information and Software Technology, 39(11):735--743, 1997.Google Scholar
Digital Library
- I. Deligiannis, M. Shepperd, S. Webster, and M. Roumeliotis. A review of experimental investigations into object-oriented technology. ESM, 7(3):193--231, 2002. Google Scholar
Digital Library
- T. Dybå and T. Dingsøyr. Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9--10):833--859, 2008. Google Scholar
Digital Library
- A. Egyed. Instant consistency checking for the uml. In International Conference on Software Engineering (ICSE), pages 381--390, 2006. Google Scholar
Digital Library
- R. Glass, I. Vessey, and V. Ramesh. Research in software engineering: an analysis of the literature. Information and Software Technology, 44(8):491--506, 2002.Google Scholar
Cross Ref
- S. Hanenberg. Faith, hope, and love: an essay on software science's neglect of human factors. In Symposium on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2010. Google Scholar
Digital Library
- M. Host, C. Wohlin, and T. Thelin. Experimental context classification: incentives and experience of subjects. In International Conference on Software Engineering (ICSE), 2005. Google Scholar
Digital Library
- IDEO. Method Cards: 51 Ways to Inspire Design. William Stout, 2003.Google Scholar
- M. Jørgensen. A review of studies on expert estimation of software development effort. Journal of Systems and Software, 70(1--2):37--60, 2004. Google Scholar
Digital Library
- N. Juristo, A. Moreno, and S. Vegas. Reviewing 25 years of testing technique experiments. Empirical Software Engineering, 9(1):7--44, 2004. Google Scholar
Digital Library
- B. Kitchenham. Evaluating software engineering methods and tool part 1: The evaluation context and evaluation methods. ACM SIGSOFT Software Engineering Notes, 21(1):11--14, 1996. Google Scholar
Digital Library
- H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Seven Guiding Scenarios for Information Visualization Evaluation. Technical Report 2011--992-04, University of Calgary, 2011.Google Scholar
- J. Lazar, J. H. Feng, and H. Hochheiser. Research Methods in Human Computer Interaction. Wiley, 2010. Google Scholar
Digital Library
- J. Lung, J. Aranda, S. M. Easterbrook, and G. V. Wilson. On the difficulty of replicating human subjects studies in software engineering. In International Conference on Software Engineering (ICSE), 2008. Google Scholar
Digital Library
- J. McGrath. Methodology matters: Doing research in the behavioral and social sciences. In Human-computer interaction, pages 152--169. Morgan Kaufmann Publishers Inc., 1995. Google Scholar
Digital Library
- J. R. Quinlan. C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc., 1993. Google Scholar
Digital Library
- M. Robnik-\vSikonja and I. Kononenko. Theoretical and empirical analysis of ReliefF and RReliefF. Machine Learning, 53:23--69, October 2003. Google Scholar
Digital Library
- M. Shaw. Writing good software engineering research papers: minitutorial. In International Conference on Software Engineering (ICSE), 2003. Google Scholar
Digital Library
- D. Sjoberg, B. Anda, E. Arisholm, T. Dyba, M. Jorgensen, A. Karahasanovic, E. Koren, and M. Vokác. Conducting realistic experiments in software engineering. In International Symposium on Empirical Software Engineering and Measurement (ESEM), 2003.Google Scholar
- D. Sjoberg, T. Dyba, and M. Jorgensen. The future of empirical methods in software engineering research. In Future of Software Engineering Workshop (FoSER), 2007. Google Scholar
Digital Library
- D. Sjøeberg, J. Hannay, O. Hansen, V. Kampenes, A. Karahasanovic, N.-K. Liborg, and A. Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733 -- 753, 2005. Google Scholar
Digital Library
- M. Svahnberg, A. Aurum, and C. Wohlin. Using students as subjects-an empirical evaluation. In International Symposium on Empirical Software Engineering and Measurement (ESEM), 2008. Google Scholar
Digital Library
- L. Teo and B. John. Cogtool-Explorer: towards a tool for predicting user interaction. In Conference on Human factors In computing systems (CHI), 2008. Google Scholar
Digital Library
- W. Tichy, P. Lukowicz, L. Prechelt, and E. Heinz. Experimental evaluation in computer science: A quantitative study. Journal of Systems and Software, 28(1):9--18, 1995. Google Scholar
Digital Library
- M. Zelkowitz. Techniques for Empirical validation. In V. Basili, D. Rombach, K. Schneider, B. Kitchenham, D. Pfahl, and R. Selby, editors, Empirical Software Engineering Issues. Critical Assessment and Future Directions, pages 4--9. Springer, 2007. Google Scholar
Digital Library
Index Terms
Benefits and barriers of user evaluation in software engineering research
Recommendations
Benefits and barriers of user evaluation in software engineering research
OOPSLA '11: Proceedings of the 2011 ACM international conference on Object oriented programming systems languages and applicationsIn this paper, we identify trends about, benefits from, and barriers to performing user evaluations in software engineering research. From a corpus of over 3,000 papers spanning ten years, we report on various subtypes of user evaluations (e.g., coding ...
A Software Maintainability Evaluation Methodology
This paper describes a conceptual framework of software maintainability and an implemented procedure for evaluating a program's documentation and source code for maintainability characteristics. The evaluation procedure includes use of closed-form ...
The diversity of gamification evaluation in the software engineering education and industry: trends, comparisons and gaps
ICSE-JSEET '21: Proceedings of the 43rd International Conference on Software Engineering: Joint Track on Software Engineering Education and TrainingContext: gamification has been used to motivate and engage participants in software engineering education and practice activities. Problem: There is a significant demand for empirical studies for the understanding of the impacts and efficacy of ...







Comments