skip to main content
research-article
Public Access

Don't Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects

Authors Info & Claims
Published:18 October 2021Publication History
Skip Abstract Section

Abstract

Software bots are used to streamline tasks in Open Source Software (OSS) projects' pull requests, saving development cost, time, and effort. However, their presence can be disruptive to the community. We identified several challenges caused by bots in pull request interactions by interviewing 21 practitioners, including project maintainers, contributors, and bot developers. In particular, our findings indicate noise as a recurrent and central problem. Noise affects both human communication and development workflow by overwhelming and distracting developers. Our main contribution is a theory of how human developers perceive annoying bot behaviors as noise on social coding platforms. This contribution may help practitioners understand the effects of adopting a bot, and researchers and tool designers may leverage our results to better support human-bot interaction on social coding platforms.

References

  1. Norah Abokhodair, Daisy Yoo, and David W. McDonald. 2015. Dissecting a Social Botnet: Growth, Content and Influence in Twitter. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). ACM, New York, NY, USA, 839--851. https://doi.org/10.1145/2675133.2675208Google ScholarGoogle Scholar
  2. David Bawden and Lyn Robinson. 2009. The dark side of information: overload, anxiety and other paradoxes and pathologies. Journal of information science, Vol. 35, 2 (2009), 180--191.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Chris Brown and Chris Parnin. 2019. Sorry to Bother You: Designing Bots for Effective Recommendations. In Proceedings of the 1st International Workshop on Bots in Software Engineering (BotSE).Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. A. Carvalho, W. Luz, D. Marcílio, R. Bonifácio, G. Pinto, and E. Dias Canedo. 2020. C-3PR: A Bot for Fixing Static Analysis Violations via Pull Requests. In Proceedings of the 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). 161--171.Google ScholarGoogle Scholar
  5. Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative analysis .sage.Google ScholarGoogle Scholar
  6. Dan Cosley, Dan Frankowski, Loren Terveen, and John Riedl. 2007. SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia. In Proceedings of the 12th International Conference on Intelligent User Interfaces (Honolulu, Hawaii, USA) (IUI '07). ACM, New York, NY, USA, 32--41. https://doi.org/10.1145/1216295.1216309Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Robert Dale. 2016. The return of the chatbots. Natural Language Engineering, Vol. 22, 5 (2016), 811--817.Google ScholarGoogle Scholar
  8. Thomas Ellwart, Christian Happ, Andrea Gurtner, and Oliver Rack. 2015. Managing information overload in virtual teams: Effects of a structured online team adaptation on cognition and performance. European Journal of Work and Organizational Psychology, Vol. 24, 5 (2015), 812--826.Google ScholarGoogle ScholarCross RefCross Ref
  9. Linda Erlenhov, Francisco Gomes de Oliveira Neto, Riccardo Scandariato, and Philipp Leitner. 2019. Current and Future Bots in Software Development. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 7--11. https://doi.org/10.1109/BotSE.2019.00009Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Linda Erlenhov, Francisco Gomes de Oliveira Neto, and Philipp Leitner. 2016. An Empirical Study of Bots in Software Development--Characteristics and Challenges from a Practitioner's Perspective. In Proceedings of the 2020 28th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2020).Google ScholarGoogle Scholar
  11. Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM, Vol. 59, 7 (2016), 96--104.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. R. Stuart Geiger and Aaron Halfaker. 2017. Operationalizing Conflict and Cooperation Between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'. Proceedings of the ACM Human-Computer Interaction, Vol. 1, CSCW, Article 49 (Dec. 2017), 33 pages. https://doi.org/10.1145/3134684Google ScholarGoogle Scholar
  13. Supratip Ghose and Jagat Joyti Barua. 2013. Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor. In Proceedings of the 2013 International Conference on Informatics, Electronics & Vision (ICIEV). IEEE, Washington, DC,USA, 1--5.Google ScholarGoogle ScholarCross RefCross Ref
  14. Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research .Routledge.Google ScholarGoogle Scholar
  15. Ulrich Gnewuch, Stefan Morana, and Alexander Maedche. 2017. Towards Designing Cooperative and Social Conversational Agents for Customer Service. In Proceedings of the International Conference on Information Systems (ICIS). AIS.Google ScholarGoogle Scholar
  16. Mehdi Golzadeh, Alexandre Decan, Damien Legay, and Tom Mens. 2021. A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments. Journal of Systems and Software, Vol. 175 (2021), 110911. https://doi.org/10.1016/j.jss.2021.110911Google ScholarGoogle ScholarCross RefCross Ref
  17. Georgios Gousios, Margaret-Anne Storey, and Alberto Bacchelli. 2016. Work Practices and Challenges in Pull-based Development: The Contributor's Perspective. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE '16). ACM, New York, NY, USA, 285--296. https://doi.org/10.1145/2884781.2884826Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie van Deursen. 2015. Work Practices and Challenges in Pull-based Development: The Integrator's Perspective. In Proceedings of the 37th International Conference on Software Engineering - Volume 1 (Florence, Italy) (ICSE '15). IEEE Press, Piscataway, NJ, USA, 358--368. http://dl.acm.org/citation.cfm?id=2818754.2818800Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Raman Goyal, Gabriel Ferreira, Christian K"astner, and James Herbsleb. 2018. Identifying unusual commits on GitHub. Journal of Software: Evolution and Process, Vol. 30, 1 (2018), e1893.Google ScholarGoogle ScholarCross RefCross Ref
  20. Tim Healy. 2012. The unanticipated consequences of technology. Nanotechnology: ethical and social Implications (2012), 155--173.Google ScholarGoogle Scholar
  21. Siw Elisabeth Hove and Bente Anda. 2005. Experiences from conducting semi-structured interviews in empirical software engineering research. In Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS'05). IEEE, 10--pp.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Shamsi T Iqbal and Eric Horvitz. 2010. Notifications and awareness: a field study of alert usage and preferences. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. 27--30.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Quentin Jones, Gilad Ravid, and Sheizaf Rafaeli. 2004. Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Information systems research, Vol. 15, 2 (2004), 194--210.Google ScholarGoogle Scholar
  24. D. Kavaler, A. Trockman, B. Vasilescu, and V. Filkov. 2019. Tool Choice Matters: JavaScript Quality Assurance Tools and Usage Outcomes in GitHub Projects. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 476--487.Google ScholarGoogle Scholar
  25. Timothy Kinsman, Mairieli Wessel, Marco Gerosa, and Christoph Treude. 2021. How Do Software Developers Use GitHub Actions to Automate Their Workflows?. In Proceedings of the IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  26. Annabel M Latham, Keeley A Crockett, David A McLean, Bruce Edmonds, and Karen O'Shea. 2010. Oscar: An intelligent conversational agent tutor to estimate learning styles. In Proceedings of the International Conference on Fuzzy Systems. IEEE, Washington, DC, USA, 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  27. Carlene Lebeuf, Margaret-Anne Storey, and Alexey Zagalsky. 2018. Software Bots. IEEE Software, Vol. 35, 1 (2018), 18--23.Google ScholarGoogle Scholar
  28. Bin Lin, Alexey Zagalsky, Margaret-Anne Storey, and Alexander Serebrenik. 2016. Why developers are slacking off: Understanding how software teams use slack. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion. ACM, 333--336.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Dongyu Liu, Micah J. Smith, and Kalyan Veeramachaneni. 2020. Understanding User-Bot Interactions for Small-Scale Automation in Open-Source Development. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1--8. https://doi.org/10.1145/3334480.3382998Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Gustavo López and Luis A Guerrero. 2016. Ubiquitous notification mechanism to provide user awareness. In Advances in Ergonomics in Design. Springer, 689--700.Google ScholarGoogle Scholar
  31. C. Mendez, H. S. Padala, Z. Steine-Hanson, C. Hildebrand, A. Horvath, C. Hill, L. Simpson, N. Patil, A. Sarma, and M. Burnett. 2018. Open Source Barriers to Entry, Revisited: A Sociotechnical Perspective. In Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 1004--1015.Google ScholarGoogle Scholar
  32. Sharan B Merriam. 1998. Qualitative Research and Case Study Applications in Education. Revised and Expanded from" Case Study Research in Education.". ERIC.Google ScholarGoogle Scholar
  33. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, Vol. 63, 2 (1956), 81.Google ScholarGoogle Scholar
  34. Samim Mirhosseini and Chris Parnin. 2017. Can Automated Pull Requests Encourage Software Developers to Upgrade Out-of-date Dependencies?. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (Urbana-Champaign, IL, USA) (ASE 2017). IEEE Press, Piscataway, NJ, USA, 84--94. http://dl.acm.org/citation.cfm?id=3155562.3155577Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Martin Monperrus. 2019. Explainable Software Bot Contributions: Case Study of Automated Bug Fixes. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 12--15. https://doi.org/10.1109/BotSE.2019.00010Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. KF Mulder. 2013. Impact of new technologies: how to assess the intended and unintended effects of new technologies. Handb. Sustain. Eng.(2013) (2013).Google ScholarGoogle Scholar
  37. Claudia Müller-Birn, Leonhard Dobusch, and James D Herbsleb. 2013. Work-to-rule: the emergence of algorithmic governance in Wikipedia. In 6th International Conference on Communities and Technologies. ACM, 80--89.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Alessandro Murgia, Daan Janssens, Serge Demeyer, and Bogdan Vasilescu. 2016. Among the machines: Human-bot interaction on social q&a websites. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1272--1279.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Kazuaki Nakamura, Koh Kakusho, Tetsuo Shoji, and Michihiko Minoh. 2012. Investigation of a Method to Estimate Learners' Interest Level for Agent-based Conversational e-Learning. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems. Springer, Berlin, Heidelberg, 425--433.Google ScholarGoogle ScholarCross RefCross Ref
  40. Azadeh Nematzadeh, Giovanni Luca Ciampaglia, Yong-Yeol Ahn, and Alessandro Flammini. 2016. Information overload in group communication: From conversation to cacophony in the twitch chat. Royal Society open science, Vol. 6, 10 (2016).Google ScholarGoogle Scholar
  41. Elahe Paikari and André van der Hoek. 2018. A Framework for Understanding Chatbots and Their Future. In Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering (Gothenburg, Sweden) (CHASE '18). ACM, New York, NY, USA, 13--16. https://doi.org/10.1145/3195836.3195859Google ScholarGoogle Scholar
  42. Michael Quinn Patton. 2014. Qualitative research & evaluation methods: Integrating theory and practice .Sage publications.Google ScholarGoogle Scholar
  43. Zhenhui Peng and Xiaojuan Ma. 2019. Exploring how software developers work with mention bot in GitHub. CCF Transactions on Pervasive Computing and Interaction, Vol. 1, 3 (01 Nov 2019), 190--203. https://doi.org/10.1007/s42486-019-00013--2Google ScholarGoogle ScholarCross RefCross Ref
  44. Zhenhui Peng, Jeehoon Yoo, Meng Xia, Sunghun Kim, and Xiaojuan Ma. 2018. Exploring How Software Developers Work with Mention Bot in GitHub. In Proceedings of the Sixth International Symposium of Chinese CHI (Montreal, QC, Canada) (ChineseCHI '18). ACM, New York, NY, USA, 152--155. https://doi.org/10.1145/3202667.3202694Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, New York, NY, USA, 813--822.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. C. E. Shannon. 2001. A Mathematical Theory of Communication. SIGMOBILE Mob. Comput. Commun. Rev., Vol. 5, 1 (Jan. 2001), 3--55. https://doi.org/10.1145/584091.584093Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Carla Simone, Monica Divitini, and Kjeld Schmidt. 1995. A notation for malleable and interoperable coordination mechanisms for CSCW systems. In Proceedings of conference on Organizational computing systems. 44--54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Igor Steinmacher, Ana Paula Chaves, and Marco Aurélio Gerosa. 2013. Awareness support in distributed software development: A systematic review and mapping of the literature. Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (CSCW), Vol. 22, 2--3 (2013), 113--158.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015. Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). ACM, New York, NY, USA, 1379--1392. https://doi.org/10.1145/2675133.2675215Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Igor Steinmacher, Tayana Uchoa Conte, Christoph Treude, and Marco Aurelio Gerosa. 2016. Overcoming Open Source Project Entry Barriers with a Portal for Newcomers. In Proceedings of the 38th International Conference on Software Engineering (ICSE).Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering. 120--131.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Margaret-Anne Storey, Alexander Serebrenik, Carolyn Penstein Rosé, Thomas Zimmermann, and James D. Herbsleb. 2020. BOTse: Bots in Software Engineering (Dagstuhl Seminar 19471). Dagstuhl Reports, Vol. 9, 11 (2020), 84--96.Google ScholarGoogle Scholar
  53. Margaret-Anne Storey and Alexey Zagalsky. 2016. Disrupting Developer Productivity One Bot at a Time. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). ACM, New York, NY, USA, 928--931. https://doi.org/10.1145/2950290.2983989Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.Google ScholarGoogle Scholar
  55. ANSELM L Strauss and JM Corbin. 1998. Basics of qualitative research: Techniques and procedures for developing grounded theory Sage Publications. SAGE Publications (1998).Google ScholarGoogle Scholar
  56. Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Let's Talk About It: Evaluating Contributions Through Discussion in GitHub. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014). ACM, New York, NY, USA, 144--154. https://doi.org/10.1145/2635868.2635882Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Simon Urli, Zhongxing Yu, Lionel Seinturier, and Martin Monperrus. 2018. How to Design a Program Repair Bot?: Insights from the Repairnator Project. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (Gothenburg, Sweden) (ICSE-SEIP '18). ACM, New York, NY, USA, 95--104. https://doi.org/10.1145/3183519.3183540Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 805--816. https://doi.org/10.1145/2786805.2786850Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Alessandro Vinciarelli, Anna Esposito, Elisabeth André, Francesca Bonin, Mohamed Chetouani, Jeffrey F Cohn, Marco Cristani, Ferdinand Fuhrmann, Elmer Gilmartin, Zakia Hammal, et al. 2015. Open challenges in modelling, analysis and synthesis of human behaviour in human-human and human--machine interactions. Cognitive Computation, Vol. 7, 4 (2015), 397--413.Google ScholarGoogle ScholarCross RefCross Ref
  60. Mihaela Vorvoreanu, Lingyi Zhang, Yun-Han Huang, Claudia Hilderbrand, Zoe Steine-Hanson, and Margaret Burnett. 2019. From Gender Biases to Gender-Inclusive Design: An Empirical Investigation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3290605.3300283Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Mairieli Wessel, Bruno Mendes de Souza, Igor Steinmacher, Igor S. Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A. Gerosa. 2018. The Power of Bots: Characterizing and Understanding Bots in OSS Projects. Proceedings of the ACM Conference on Computer Supported Cooperative Work Social Computing, Vol. 2, CSCW, Article 182 (Nov. 2018), 19 pages. https://doi.org/10.1145/3274451Google ScholarGoogle Scholar
  62. Mairieli Wessel, Alexander Serebrenik, Igor Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2020 a. What to Expect from Code Review Bots on GitHub? A Survey with OSS Maintainers. In Proceedings of the SBES 2020 - Ideias Inovadoras e Resultados Emergentes .Google ScholarGoogle Scholar
  63. Mairieli Wessel, Alexander Serebrenik, Igor Scaliante Wiese, Igor Steinmacher, and Marco Aurelio Gerosa. 2020 b. Effects of Adopting Code Review Bots on Pull Requests to OSS Projects. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE Computer Society.Google ScholarGoogle ScholarCross RefCross Ref
  64. Mairieli Wessel and Igor Steinmacher. 2020. The Inconvenient Side of Software Bots on Pull Requests. In Proceedings of the 2nd International Workshop on Bots in Software Engineering (BotSE). https://doi.org/10.1145/3387940.3391504Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. David D Woods and Emily S Patterson. 2001. How unexpected events produce an escalation of cognitive and coordinative demands. PA Hancock, & PA Desmond, Stress, workload, and fatigue. Mahwah, NJ: L. Erlbaum (2001).Google ScholarGoogle Scholar
  66. Marvin Wyrich and Justus Bogner. 2019. Towards an Autonomous Bot for Automatic Source Code Refactoring. In Proceedings of the 1st International Workshop on Bots in Software Engineering (Montreal, Quebec, Canada) (BotSE '19). IEEE Press, Piscataway, NJ, USA, 24--28. https://doi.org/10.1109/BotSE.2019.00015Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Bin Xu, Tina Chien-Wen Yuan, Susan R. Fussell, and Dan Cosley. 2014. SoBot: Facilitating Conversation Using Social Media Data and a Social Agent. In Proceedings of the Companion Publication of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW Companion '14). ACM, New York, NY, USA, 41--44. https://doi.org/10.1145/2556420.2556789Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. Lei Zheng, Christopher M Albano, and Jeffrey V Nickerson. 2018. Steps toward Understanding the Design and Evaluation Spaces of Bot and Human Knowledge Production Systems. In Proceedings of the Wiki Workshop'19.Google ScholarGoogle Scholar
  69. Victor W Zue and James R Glass. 2000. Conversational interfaces: Advances and challenges. Proc. IEEE, Vol. 88, 8 (2000), 1166--1180.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Don't Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!