Experiences from using snowballing and database searches in systematic literature studies

ABSTRACT

Background: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward snowballing - BSB) and citations (forward snowballing - FSB) of relevant papers are reviewed to identify new papers whereas in a database search, different databases are searched using predefined search strings to identify new papers. Objective: Snowballing has not been in use as extensively as database search. Hence it is important to evaluate its efficiency and reliability when being used as a search strategy in literature studies. Moreover, it is important to compare it to database searches. Method: In this paper, we applied snowballing in a literature study, and reflected on the outcome. We also compared database search with backward and forward snowballing. Database search and snowballing were conducted independently by different researchers. The searches of our literature study were compared with respect to the efficiency and reliability of the findings. Results: Out of the total number of papers found, snowballing identified 83% of the papers in comparison to 46% of the papers for the database search. Snowballing failed to identify a few relevant papers, which potentially could have been addressed by identifying a more comprehensive start set. Conclusion: The efficiency of snowballing is comparable to database search. It can potentially be more reliable than a database search however, the reliability is highly dependent on the creation of a suitable start set.

References

1. L. Chen, M. Ali Babar, and N. Ali. Variability management in software product lines: a systematic review. In Proceedings of the 13th International Software Product Line Conference, pages 81--90. Carnegie Mellon University, 2009.
2. T. Greenhalgh and R. Peacock. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. Bmj, 331(7524): 1064--1065, 2005.
3. S. Jalali and C. Wohlin. Systematic literature studies: database searches vs. backward snowballing. In Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement, pages 29--38. ACM, 2012.
4. B. A. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. Technical report, Technical report, EBSE Technical Report EBSE-2007-01, 2007.
5. B. A. Kitchenham, T. Dyba, and M. Jorgensen. Evidence-based software engineering. In Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on, pages 273--281. IEEE, 2004.
6. S. MacDonell, M. Shepperd, B. Kitchenham, and E. Mendes. How reliable are systematic reviews in empirical software engineering? Software Engineering, IEEE Transactions on, 36(5): 676--687, 2010.
7. K. Petersen and N. B. Ali. Identifying strategies for study selection in systematic reviews and maps. In Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement, ESEM 2011, Banff, AB, Canada, September 22--23, 2011, pages 351--354, 2011.
8. M. Riaz, E. Mendes, and E. Tempero. A systematic review of software maintainability prediction and metrics. In Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement, pages 367--377. IEEE, 2009.
9. M. Skoglund and P. Runeson. Reference-based search strategies in systematic reviews. In the Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering, Durham, England, 2009.
10. J. Webster and R. T. Watson. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly, 26(2): 3, 2002.
11. R. Wieringa, N. A. M. Maiden, N. R. Mead, and C. Rolland. Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir. Eng., 11(1): 102--107, 2006.
12. C. Wohlin. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In 8th International Conference on Evaluation and Assessment in Software Engineering (EASE 2014), pages 321--330. ACM, 2014.
13. C. Wohlin, P. Runeson, P. A. da Mota Silveira Neto, E. Engström, I. do Carmo Machado, and E. S. de Almeida. On the reliability of mapping studies in software engineering. Journal of Systems and Software, 86(10): 2594--2610, 2013.

Index Terms

• Mathematics of computing

  • Probability and statistics

    • Statistical paradigms

      • Exploratory data analysis

• Social and professional topics

  • Professional topics

    • Management of computing and information systems

      • Project and people management

• Software and its engineering

  • Software creation and management

    • Software development process management

  • Software notations and tools

    • Software configuration management and version control systems