Abstract

Software bots are used to streamline tasks in Open Source Software (OSS) projects' pull requests, saving development cost, time, and effort. However, their presence can be disruptive to the community. We identified several challenges caused by bots in pull request interactions by interviewing 21 practitioners, including project maintainers, contributors, and bot developers. In particular, our findings indicate noise as a recurrent and central problem. Noise affects both human communication and development workflow by overwhelming and distracting developers. Our main contribution is a theory of how human developers perceive annoying bot behaviors as noise on social coding platforms. This contribution may help practitioners understand the effects of adopting a bot, and researchers and tool designers may leverage our results to better support human-bot interaction on social coding platforms.
- Norah Abokhodair, Daisy Yoo, and David W. McDonald. 2015. Dissecting a Social Botnet: Growth, Content and Influence in Twitter. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). ACM, New York, NY, USA, 839--851. https://doi.org/10.1145/2675133.2675208Google Scholar
- David Bawden and Lyn Robinson. 2009. The dark side of information: overload, anxiety and other paradoxes and pathologies. Journal of information science, Vol. 35, 2 (2009), 180--191.Google Scholar
Digital Library
- Chris Brown and Chris Parnin. 2019. Sorry to Bother You: Designing Bots for Effective Recommendations. In Proceedings of the 1st International Workshop on Bots in Software Engineering (BotSE).Google Scholar
Digital Library
- A. Carvalho, W. Luz, D. Marcílio, R. Bonifácio, G. Pinto, and E. Dias Canedo. 2020. C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests. In Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 161--171.Google Scholar
- Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative analysis .sage.Google Scholar
- Dan Cosley, Dan Frankowski, Loren Terveen, and John Riedl. 2007. SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia. In Proceedings of the 12th International Conference on Intelligent User Interfaces (Honolulu, Hawaii, USA) (IUI '07). ACM, New York, NY, USA, 32--41. https://doi.org/10.1145/1216295.1216309Google Scholar
Digital Library
- Robert Dale. 2016. The return of the chatbots. Natural Language Engineering, Vol. 22, 5 (2016), 811--817.Google Scholar
- Thomas Ellwart, Christian Happ, Andrea Gurtner, and Oliver Rack. 2015. Managing information overload in virtual teams: Effects of a structured online team adaptation on cognition and performance. European Journal of Work and Organizational Psychology, Vol. 24, 5 (2015), 812--826.Google Scholar
Cross Ref
- Linda Erlenhov, Francisco Gomes de Oliveira Neto, Riccardo Scandariato, and Philipp Leitner. 2019. Current and Future Bots in Software Development. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 7--11. https://doi.org/10.1109/BotSE.2019.00009Google Scholar
Digital Library
- Linda Erlenhov, Francisco Gomes de Oliveira Neto, and Philipp Leitner. 2016. An Empirical Study of Bots in Software Development--Characteristics and Challenges from a Practitioner's Perspective. In Proceedings of the 2020 28th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2020).Google Scholar
- Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM, Vol. 59, 7 (2016), 96--104.Google Scholar
Digital Library
- R. Stuart Geiger and Aaron Halfaker. 2017. Operationalizing Conflict and Cooperation Between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'. Proceedings of the ACM Human-Computer Interaction, Vol. 1, CSCW, Article 49 (Dec. 2017), 33 pages. https://doi.org/10.1145/3134684Google Scholar
- Supratip Ghose and Jagat Joyti Barua. 2013. Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor. In Proceedings of the 2013 International Conference on Informatics, Electronics & Vision (ICIEV). IEEE, Washington, DC,USA, 1--5.Google Scholar
Cross Ref
- Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research .Routledge.Google Scholar
- Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2017. Towards Designing Cooperative and Social Conversational Agents for Customer Service. In Proceedings of the International Conference on Information Systems (ICIS). AIS.Google Scholar
- Mehdi Golzadeh, Alexandre Decan, Damien Legay, and Tom Mens. 2021. A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments. Journal of Systems and Software, Vol. 175 (2021), 110911. https://doi.org/10.1016/j.jss.2021.110911Google Scholar
Cross Ref
- Georgios Gousios, Margaret-Anne Storey, and Alberto Bacchelli. 2016. Work Practices and Challenges in Pull-based Development: The Contributor's Perspective. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE '16). ACM, New York, NY, USA, 285--296. https://doi.org/10.1145/2884781.2884826Google Scholar
Digital Library
- Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie van Deursen. 2015. Work Practices and Challenges in Pull-based Development: The Integrator's Perspective. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (Florence, Italy) (ICSE '15). IEEE Press, Piscataway, NJ, USA, 358--368. http://dl.acm.org/citation.cfm?id=2818754.2818800Google Scholar
Digital Library
- Raman Goyal, Gabriel Ferreira, Christian K"astner, and James Herbsleb. 2018. Identifying unusual commits on GitHub. Journal of Software: Evolution and Process, Vol. 30, 1 (2018), e1893.Google Scholar
Cross Ref
- Tim Healy. 2012. The unanticipated consequences of technology. Nanotechnology: ethical and social Implications (2012), 155--173.Google Scholar
- Siw Elisabeth Hove and Bente Anda. 2005. Experiences from conducting semi-structured interviews in empirical software engineering research. In Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS'05). IEEE, 10--pp.Google Scholar
Digital Library
- Shamsi T Iqbal and Eric Horvitz. 2010. Notifications and awareness: a field study of alert usage and preferences. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. 27--30.Google Scholar
Digital Library
- Quentin Jones, Gilad Ravid, and Sheizaf Rafaeli. 2004. Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Information systems research, Vol. 15, 2 (2004), 194--210.Google Scholar
- D. Kavaler, A. Trockman, B. Vasilescu, and V. Filkov. 2019. Tool Choice Matters: JavaScript Quality Assurance Tools and Usage Outcomes in GitHub Projects. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 476--487.Google Scholar
- Timothy Kinsman, Mairieli Wessel, Marco Gerosa, and Christoph Treude. 2021. How Do Software Developers Use GitHub Actions to Automate Their Workflows?. In Proceedings of the IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE.Google Scholar
Cross Ref
- Annabel M Latham, Keeley A Crockett, David A McLean, Bruce Edmonds, and Karen O'Shea. 2010. Oscar: An intelligent conversational agent tutor to estimate learning styles. In Proceedings of the International Conference on Fuzzy Systems. IEEE, Washington, DC, USA, 1--8.Google Scholar
Cross Ref
- Carlene Lebeuf, Margaret-Anne Storey, and Alexey Zagalsky. 2018. Software Bots. IEEE Software, Vol. 35, 1 (2018), 18--23.Google Scholar
- Bin Lin, Alexey Zagalsky, Margaret-Anne Storey, and Alexander Serebrenik. 2016. Why developers are slacking off: Understanding how software teams use slack. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. ACM, 333--336.Google Scholar
Digital Library
- Dongyu Liu, Micah J. Smith, and Kalyan Veeramachaneni. 2020. Understanding User-Bot Interactions for Small-Scale Automation in Open-Source Development. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1--8. https://doi.org/10.1145/3334480.3382998Google Scholar
Digital Library
- Gustavo López and Luis A Guerrero. 2016. Ubiquitous notification mechanism to provide user awareness. In Advances in Ergonomics in Design. Springer, 689--700.Google Scholar
- C. Mendez, H. S. Padala, Z. Steine-Hanson, C. Hildebrand, A. Horvath, C. Hill, L. Simpson, N. Patil, A. Sarma, and M. Burnett. 2018. Open Source Barriers to Entry, Revisited: A Sociotechnical Perspective. In Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 1004--1015.Google Scholar
- Sharan B Merriam. 1998. Qualitative Research and Case Study Applications in Education. Revised and Expanded from" Case Study Research in Education.". ERIC.Google Scholar
- George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, Vol. 63, 2 (1956), 81.Google Scholar
- Samim Mirhosseini and Chris Parnin. 2017. Can Automated Pull Requests Encourage Software Developers to Upgrade Out-of-date Dependencies?. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (Urbana-Champaign, IL, USA) (ASE 2017). IEEE Press, Piscataway, NJ, USA, 84--94. http://dl.acm.org/citation.cfm?id=3155562.3155577Google Scholar
Digital Library
- Martin Monperrus. 2019. Explainable Software Bot Contributions: Case Study of Automated Bug Fixes. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 12--15. https://doi.org/10.1109/BotSE.2019.00010Google Scholar
Digital Library
- KF Mulder. 2013. Impact of new technologies: how to assess the intended and unintended effects of new technologies. Handb. Sustain. Eng.(2013) (2013).Google Scholar
- Claudia Müller-Birn, Leonhard Dobusch, and James D Herbsleb. 2013. Work-to-rule: the emergence of algorithmic governance in Wikipedia. In 6th International Conference on Communities and Technologies. ACM, 80--89.Google Scholar
Digital Library
- Alessandro Murgia, Daan Janssens, Serge Demeyer, and Bogdan Vasilescu. 2016. Among the machines: Human-bot interaction on social q&a websites. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1272--1279.Google Scholar
Digital Library
- Kazuaki Nakamura, Koh Kakusho, Tetsuo Shoji, and Michihiko Minoh. 2012. Investigation of a Method to Estimate Learners' Interest Level for Agent-based Conversational e-Learning. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems. Springer, Berlin, Heidelberg, 425--433.Google Scholar
Cross Ref
- Azadeh Nematzadeh, Giovanni Luca Ciampaglia, Yong-Yeol Ahn, and Alessandro Flammini. 2016. Information overload in group communication: From conversation to cacophony in the twitch chat. Royal Society open science, Vol. 6, 10 (2016).Google Scholar
- Elahe Paikari and André van der Hoek. 2018. A Framework for Understanding Chatbots and Their Future. In Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering (Gothenburg, Sweden) (CHASE '18). ACM, New York, NY, USA, 13--16. https://doi.org/10.1145/3195836.3195859Google Scholar
- Michael Quinn Patton. 2014. Qualitative research & evaluation methods: Integrating theory and practice .Sage publications.Google Scholar
- Zhenhui Peng and Xiaojuan Ma. 2019. Exploring how software developers work with mention bot in GitHub. CCF Transactions on Pervasive Computing and Interaction, Vol. 1, 3 (01 Nov 2019), 190--203. https://doi.org/10.1007/s42486-019-00013--2Google Scholar
Cross Ref
- Zhenhui Peng, Jeehoon Yoo, Meng Xia, Sunghun Kim, and Xiaojuan Ma. 2018. Exploring How Software Developers Work with Mention Bot in GitHub. In Proceedings of the Sixth International Symposium of Chinese CHI (Montreal, QC, Canada) (ChineseCHI '18). ACM, New York, NY, USA, 152--155. https://doi.org/10.1145/3202667.3202694Google Scholar
Digital Library
- Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, New York, NY, USA, 813--822.Google Scholar
Digital Library
- C. E. Shannon. 2001. A Mathematical Theory of Communication. SIGMOBILE Mob. Comput. Commun. Rev., Vol. 5, 1 (Jan. 2001), 3--55. https://doi.org/10.1145/584091.584093Google Scholar
Digital Library
- Carla Simone, Monica Divitini, and Kjeld Schmidt. 1995. A notation for malleable and interoperable coordination mechanisms for CSCW systems. In Proceedings of conference on Organizational computing systems. 44--54.Google Scholar
Digital Library
- Igor Steinmacher, Ana Paula Chaves, and Marco Aurélio Gerosa. 2013. Awareness support in distributed software development: A systematic review and mapping of the literature. Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW), Vol. 22, 2--3 (2013), 113--158.Google Scholar
Digital Library
- Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015. Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). ACM, New York, NY, USA, 1379--1392. https://doi.org/10.1145/2675133.2675215Google Scholar
Digital Library
- Igor Steinmacher, Tayana Uchoa Conte, Christoph Treude, and Marco Aurelio Gerosa. 2016. Overcoming Open Source Project Entry Barriers with a Portal for Newcomers. In Proceedings of the 38th International Conference on Software Engineering (ICSE).Google Scholar
Digital Library
- Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering. 120--131.Google Scholar
Digital Library
- Margaret-Anne Storey, Alexander Serebrenik, Carolyn Penstein Rosé, Thomas Zimmermann, and James D. Herbsleb. 2020. BOTse: Bots in Software Engineering (Dagstuhl Seminar 19471). Dagstuhl Reports, Vol. 9, 11 (2020), 84--96.Google Scholar
- Margaret-Anne Storey and Alexey Zagalsky. 2016. Disrupting Developer Productivity One Bot at a Time. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). ACM, New York, NY, USA, 928--931. https://doi.org/10.1145/2950290.2983989Google Scholar
Digital Library
- Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.Google Scholar
- ANSELM L Strauss and JM Corbin. 1998. Basics of qualitative research: Techniques and procedures for developing grounded theory Sage Publications. SAGE Publications (1998).Google Scholar
- Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Let's Talk About It: Evaluating Contributions Through Discussion in GitHub. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014). ACM, New York, NY, USA, 144--154. https://doi.org/10.1145/2635868.2635882Google Scholar
Digital Library
- Simon Urli, Zhongxing Yu, Lionel Seinturier, and Martin Monperrus. 2018. How to Design a Program Repair Bot?: Insights from the Repairnator Project. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (Gothenburg, Sweden) (ICSE-SEIP '18). ACM, New York, NY, USA, 95--104. https://doi.org/10.1145/3183519.3183540Google Scholar
Digital Library
- Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 805--816. https://doi.org/10.1145/2786805.2786850Google Scholar
Digital Library
- Alessandro Vinciarelli, Anna Esposito, Elisabeth André, Francesca Bonin, Mohamed Chetouani, Jeffrey F Cohn, Marco Cristani, Ferdinand Fuhrmann, Elmer Gilmartin, Zakia Hammal, et al. 2015. Open challenges in modelling, analysis and synthesis of human behaviour in human-human and human--machine interactions. Cognitive Computation, Vol. 7, 4 (2015), 397--413.Google Scholar
Cross Ref
- Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe Steine-Hanson, and Margaret Burnett. 2019. From Gender Biases to Gender-Inclusive Design: An Empirical Investigation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3290605.3300283Google Scholar
Digital Library
- Mairieli Wessel, Bruno Mendes de Souza, Igor Steinmacher, Igor S. Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A. Gerosa. 2018. The Power of Bots: Characterizing and Understanding Bots in OSS Projects. Proceedings of the ACM Conference on Computer Supported Cooperative Work Social Computing, Vol. 2, CSCW, Article 182 (Nov. 2018), 19 pages. https://doi.org/10.1145/3274451Google Scholar
- Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2020 a. What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. In Proceedings of the SBES 2020 - Ideias Inovadoras e Resultados Emergentes .Google Scholar
- Mairieli Wessel, Alexander Serebrenik, Igor Scaliante Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2020 b. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE Computer Society.Google Scholar
Cross Ref
- Mairieli Wessel and Igor Steinmacher. 2020. The Inconvenient Side of Software Bots on Pull Requests. In Proceedings of the 2nd International Workshop on Bots in Software Engineering (BotSE). https://doi.org/10.1145/3387940.3391504Google Scholar
Digital Library
- David D Woods and Emily S Patterson. 2001. How unexpected events produce an escalation of cognitive and coordinative demands. PA Hancock, & PA Desmond, Stress, workload, and fatigue. Mahwah, NJ: L. Erlbaum (2001).Google Scholar
- Marvin Wyrich and Justus Bogner. 2019. Towards an Autonomous Bot for Automatic Source Code Refactoring. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 24--28. https://doi.org/10.1109/BotSE.2019.00015Google Scholar
Digital Library
- Bin Xu, Tina Chien-Wen Yuan, Susan R. Fussell, and Dan Cosley. 2014. SoBot: Facilitating Conversation Using Social Media Data and a Social Agent. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW Companion '14). ACM, New York, NY, USA, 41--44. https://doi.org/10.1145/2556420.2556789Google Scholar
Digital Library
- Lei Zheng, Christopher M Albano, and Jeffrey V Nickerson. 2018. Steps toward Understanding the Design and Evaluation Spaces of Bot and Human Knowledge Production Systems. In Proceedings of the Wiki Workshop'19.Google Scholar
- Victor W Zue and James R Glass. 2000. Conversational interfaces: Advances and challenges. Proc. IEEE, Vol. 88, 8 (2000), 1166--1180.Google Scholar
Cross Ref
Index Terms
Don't Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects
Recommendations
The Inconvenient Side of Software Bots on Pull Requests
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering WorkshopsSoftware bots are applications that integrate their work with humans' tasks, serving as conduits between users and other tools. Due to their ability to automate tasks, bots have been widely adopted by Open Source Software (OSS) projects hosted on ...
Software bots in software engineering: benefits and challenges
MSR '22: Proceedings of the 19th International Conference on Mining Software RepositoriesSoftware bots are becoming increasingly popular in software engineering (SE). In this tutorial, we define what a bot is and present several examples. We also discuss the many benefits bots provide to the SE community, including helping in development ...
Summary of the Third International Workshop on Bots in Software Engineering (BotSE 2021)
Bots automate tasks in software engineering projects and interact with software developers. Bots have been proposed, for example, for testing, maintenance, and automating bug fixes. The research community has been discussing these bots in the ...






Comments