The Agile Success Model: A Mixed-methods Study of a Large-scale Agile Transformation

Abstract

Organizations are increasingly adopting Agile frameworks for their internal software development. Cost reduction, rapid deployment, requirements and mental model alignment are typical reasons for an Agile transformation. This article presents an in-depth field study of a large-scale Agile transformation in a mission-critical environment, where stakeholders’ commitment was a critical success factor. The goal of such a transformation was to implement mission-oriented features, reducing costs and time to operate in critical scenarios. The project lasted several years and involved over 40 professionals. We report how a hierarchical and plan-driven organization exploited Agile methods to develop a Command & Control (C2) system. Accordingly, we first abstract our experience, inducing a success model of general use for other comparable organizations by performing a post-mortem study. The goal of the inductive research process was to identify critical success factors and their relations. Finally, we validated and generalized our model through Partial Least Squares - Structural Equation Modelling, surveying 200 software engineers involved in similar projects. We conclude the article with data-driven recommendations concerning the management of Agile projects.

References

1. A. Ahmed, S. Ahmad, N. Ehsan, E. Mirza, and S. Z. Sarwar. 2010. Agile software development: Impact on productivity and quality. In International Conference on Management of Innovation & Technology. IEEE, 287–291.
2. D. S. Alberts, J. J. Garstka, and F. P. Stein. 2000. Network Centric Warfare: Developing and Leveraging Information Superiority. Technical Report. DTIC Document.
3. A. Alorwu, N. van Berkel, J. Goncalves, J. Oppenlaender, M. B. Lopez, M. Seetharaman, and S. Hosio. 2020. Crowdsourcing sensitive data using public displays: Opportunities, challenges, and considerations. Person. Ubiq. Comput. (2020). https://books.google.it/books?hl=it&lr=&id=CwvBEKsCY2gC&oi=fnd&pg=PR11&dq=.+W.+Ambler+and+M.+Lines.+2012.+Disciplined+Agile+Delivery:+A+Practitioner%E2%80%99s+Guide+to+Agile+Software+Delivery+in+the+1152+Enterprise.+IBM+Press.&ots=lHT53hT1OF&sig=vMB6rZM2TiL0pLIlMv8GX03fwOk#v=onepage&q&f=false.
4. S. W. Ambler and M. Lines. 2012. Disciplined Agile Delivery: A Practitioner’s Guide to Agile Software Delivery in the Enterprise. IBM Press.
5. T. Anning-Dorson. 2018. Customer involvement capability and service firm performance: The mediating role of innovation. J. Bus. Res. 86 (2018), 269–280.
6. S. Baltes and P. Ralph. 2020. Sampling in software engineering research: A critical review and guidelines. arxiv:2002.07764 (2020).
7. A. Barcomb, K.-J. Stol, D. Riehle, and B. Fitzgerald. 2019. Why do episodic volunteers stay in FLOSS communities? In International Conference on Software Engineering. IEEE, 948–959.
8. L. Benedicenti, A. Messina, and A. Sillitti. 2017. iAgile: Mission critical military software development. In International Conference on High Performance Computing & Simulation. IEEE, 545–552.
9. B. Boehm. 2002. Get ready for agile methods, with care. Computer 35, 1 (2002), 64–69.
10. B. Boehm and R. Turner. 2005. Management challenges to implementing agile processes in traditional development organizations. IEEE Softw. 22, 5 (2005), 30–39.
11. A. Campanelli and F. Parreiras. 2015. Agile methods tailoring–A systematic literature review. J. Syst. Softw. 110 (2015), 85–100.
12. W. W. Chin. 1998. The partial least squares approach to structural equation modeling. Mod. Meth. Bus. Res. 295, 2 (1998), 295–336.
13. M.-W. Chung and B. Drummond. 2009. Agile at Yahoo! from the trenches. In AGILE Conference. IEEE, 113–118.
14. P. Ciancarini, M. Missiroli, and D. Russo. 2019. Cooperative thinking: Analyzing a new framework for software engineering education. J. Syst. Softw. 157 (2019), 110401.
15. J. Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Routledge.
16. M. Cohn and D. Ford. 2003. Introducing an agile process to an organization [software development]. Computer 36, 6 (2003), 74–78.
17. M. Conway. 1968. How do committees invent. Datamation 14, 4 (1968), 28–31.
18. J. Corbin and A. Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitat. Sociol. 13, 1 (1990), 3–21.
19. F. R. Cotugno. 2016. Managing increasing user needs complexity within the ITA Army Agile Framework. In International Conference in Software Engineering for Defence Applications. Springer, 1–11.
20. J. Creswell. 2013. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage.
21. J. Dibbern, T. Goles, R. Hirschheim, and B. Jayatilaka. 2004. Information systems outsourcing: A survey and analysis of the literature. ACM Sigmis Datab. 35, 4 (2004), 6–102.
22. K. Dikert, M. Paasivaara, and C. Lassenius. 2016. Challenges and success factors for large-scale agile transformations: A systematic literature review. J. Syst. Softw. 119 (2016), 87–108.
23. T. Dybå and T. Dingsøyr. 2008. Empirical studies of agile software development: A systematic review. Inf. Softw. Technol. 50, 9-10 (2008), 833–859.
24. A. C. Edmondson and S. E. McManus. 2007. Methodological fit in management field research. Acad. Manag. Rev. 32, 4 (2007), 1246–1264.
25. J. R. Edwards and R. P. Bagozzi. 2000. On the nature and direction of relationships between constructs and measures.Psychol. Meth. 5, 2 (2000), 155.
26. K. Eisenhardt. 1989. Building theories from case study research. Acad. Manag. Rev. 14, 4 (1989), 532–550.
27. K. Eisenhardt and M. Graebner. 2007. Theory building from cases: Opportunities and challenges. Acad. Manag. J. 50, 1 (2007), 25–32.
28. K. El Emam and A. G. Koru. 2008. A replicated survey of IT software project failures. IEEE Softw. 25, 5 (2008), 84–90.
29. F. Faul, E. Erdfelder, A. Buchner, and A. G. Lang. 2009. Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behav. Res. Meth. 41, 4 (2009), 1149–1160.
30. D. J. Fernandez and J. D. Fernandez. 2008. Agile project management agilism versus traditional approaches. J. Comput. Inf. Syst. 49, 2 (2008), 10–17.
31. D. M. Fetterman. 1998. Ethnography. Sage.
32. B. Fitzgerald and D. Howcroft. 1998. Towards dissolution of the IS research debate: From polarization to polarity. J. Inf. Technol. 13, 4 (1998), 313–326.
33. I. Gat. 2006. How BMC is scaling agile development. In AGILE Conference. IEEE, 6–16.
34. D. Gefen, D. Straub, and M.-C. Boudreau. 2000. Structural equation modeling and regression: Guidelines for research practice. Commun. Assoc. Inf. Syst. 4, 1 (2000), 7.
35. S. Geisser. 1974. A predictive approach to the random effect model. Biometrika 61, 1 (1974), 101–107.
36. D. Gioia, K. Corley, and A. Hamilton. 2013. Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organiz. Res. Meth. 16, 1 (2013), 15–31.
37. D. Gioia, J. Thomas, S. Clark, and K. Chittipeddi. 1994. Symbolism and strategic change in academia: The dynamics of sensemaking and influence. Organiz. Sci. 5, 3 (1994), 363–383.
38. B. Glaser and A. Strauss. 2017. Discovery of Grounded Theory: Strategies for Qualitative Research. Routledge.
39. B. G. Glaser. 1992. Basics of Grounded Theory Analysis. Vol. 386. Sociology Press.
40. J. H. Goldthorpe. 2000. On Sociology: Numbers, Narratives, and the Integration of Research and Theory. Oxford University Press.
41. D. L. Goodhue, W. Lewis, and R. Thompson. 2012. Does PLS have advantages for small sample size or non-normal data?MIS Quart. 36, 3 (2012), 981–1001.
42. F. J. Gravetter and L.-A. B. Forzano. 2018. Research Methods for the Behavioral Sciences. Cengage Learning.
43. D. Greening. 2013. Release duration and enterprise agility. In Hawaii International Conference on System Sciences. IEEE, 4835–4841.
44. E. Guba. 1981. Criteria for assessing the trustworthiness of naturalistic inquiries. Educ. Commun. Technol. J. 29, 2 (1981), 75–91.
45. J. F. Hair Jr, G. T. M. Hult, C. Ringle, and M. Sarstedt. 2016. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). Sage.
46. J. Hannay, D. Sjoberg, and T. Dyba. 2007. A systematic review of theory use in software engineering experiments. IEEE Trans. Softw. Eng. 33, 2 (2007), 87–107.
47. J. Henseler, C. M. Ringle, and M. Sarstedt. 2015. A new criterion for assessing discriminant validity in variance-based structural equation modeling. J. Acad. Market. Sci. 43, 1 (2015), 115–135.
48. J. Henseler, C. M. Ringle, and R. R. Sinkovics. 2009. The use of partial least squares path modeling in international marketing. In New Challenges to International Marketing. Emerald, 277–319.
49. C. A. Higgins, L. E. Duxbury, and R. H. Irving. 1992. Work-family conflict in the dual-career family. Organiz. Behav. Hum. Decis. Process. 51, 1 (1992), 51–75.
50. J. Highsmith and M. Fowler. 2001. The Agile manifesto. Softw. Devel. Mag. 9, 8 (2001), 29–30.
51. R. Hoda, N. Salleh, J. Grundy, and H. Tee. 2017. Systematic literature reviews in agile software development: A tertiary study. Inf. Softw. Technol. 85 (2017), 60–70.
52. S. Hosio, N. van Berkel, J. Oppenlaender, and J. Goncalves. 2020. Crowdsourcing personalized weight loss diets. Computer 53, 1 (2020), 63–71.
53. J. Hulland. 1999. Use of partial least squares (PLS) in strategic management research: A review of four recent studies. Strat. Manag. J. 20, 2 (1999), 195–204.
54. L. Isabella. 1990. Evolving interpretations as a change unfolds: How managers construe key organizational events. Acad. Manag. J. 33, 1 (1990), 7–41.
55. M. Jovanović, A. Mas, A.-L. Mesquida, and B. Lalić. 2017. Transition of organizational roles in Agile transformation process: A grounded theory approach. J. Syst. Softw. 133 (2017), 174–194.
56. D. Karlstrom and P. Runeson. 2005. Combining agile methods with stage-gate project management. IEEE Softw. 22, 3 (2005), 43–49.
57. T. R. Kayworth and D. E. Leidner. 2002. Leadership effectiveness in global virtual teams. J. Manag. Inf. Syst. 18, 3 (2002), 7–40.
58. M. Khalifa and J. M. Verner. 2000. Drivers for software development method usage. IEEE Trans. Eng. Manag. 47, 3 (2000), 360–369.
59. U. Kulkarni, S. Ravindran, and R. Freeze. 2006. A knowledge management success model: Theoretical development and empirical validation. J. Manag. Inf. Syst. 23, 3 (2006), 309–347.
60. C. Larman and B. Vodde. 2015. LeSS Framework. https://less.works/.
61. G. Lee and W. Xia. 2010. Toward agile: An integrated analysis of quantitative and qualitative field data on software development agility. MIS Quart. 34, 1 (2010), 87–114.
62. D. Leffingwell et al. 2015. SAFe–Scaled Agile Framework. https://www.scaledagileframework.com/.
63. T. O. Lehtinen, M. V Mäntylä, J. Vanhanen, J. Itkonen, and C. Lassenius. 2014. Perceived causes of software project failures–An analysis of their relationships. Inf. Softw. Technol. 56, 6 (2014), 623–643.
64. W. Lewis, R. Agarwal, and V. Sambamurthy. 2003. Sources of influence on beliefs about information technology use: An empirical study of knowledge workers. MIS Quart. 27, 4 (2003), 657–678.
65. Y. Lincoln and E. Guba. 1985. Naturalistic Inquiry. Sage.
66. M. Lindvall et al. 2004. Agile software development in large organizations. Computer 37, 12 (2004), 26–34.
67. J. A. Livermore. 2008. Factors that significantly impact the implementation of an agile software development methodology. J. Softw. 3, 4 (2008), 31–36.
68. K. Locke. 1996. Rewriting the discovery of grounded theory after 25 years?J. Manag. Inq. 5, 3 (1996), 239–245.
69. L. S. Lüscher and M. W. Lewis. 2008. Organizational change and managerial sensemaking: Working through paradox. Acad. Manag. J. 51, 2 (2008), 221–240.
70. R. P. Marble. 2003. A system implementation study: Management commitment to project management. Inf. Management 41, 1 (2003), 111–123.
71. S. McDowell and N. Dourambeis. 2007. British Telecom experience report: Agile intervention–BT’s joining the dots events for organizational change. In International Conference on Extreme Programming and Agile Processes in Software Engineering. Springer, 17–23.
72. A. Messina, F. Fiore, M. Ruggiero, P. Ciancarini, and D. Russo. 2016. A New Agile paradigm for mission-critical software development. J. Defense Softw. Eng.6 (2016), 25–30.
73. J. Miles. 2014. Tolerance and Variance Inflation Factor. American Cancer Society.
74. H. Mintzberg. 1978. Patterns in strategy formation. Manag. Sci. 24, 9 (1978), 934–948.
75. S. C. Misra, V. Kumar, and U. Kumar. 2010. Identifying some critical changes required in adopting agile practices in traditional software development projects. Int. J. Qual. Reliab. Manag. 27, 4 (2010).
76. N. B. Moe, T. Dingsøyr, and T. Dybå. 2008. Understanding self-organizing teams in agile software development. In Australian Conference on Software Engineering. IEEE, 76–85.
77. A. Munns and B. Bjeirmi. 1996. The role of project management in achieving project success. Int. J. Project Manag. 14, 2 (1996), 81–87.
78. T. Nagel. 1986. The View from Nowhere. Oxford University Press.
79. NATO-RTA. 2004. NATO Code of Best Practice for Command and Control Assessment. Technical Report RTO Technical Report TR-081. NATO.
80. R. R. Nelson and S. G. Winter. 2009. An Evolutionary Theory of Economic Change. Harvard University Press.
81. S. Nerur, R. Mahapatra, and G. Mangalaraj. 2005. Challenges of migrating to agile methodologies. Commun. ACM 48, 5 (2005), 72–78.
82. I. Nonaka. 1994. A dynamic theory of organizational knowledge creation. Organiz. Sci. 5, 1 (1994), 14–37.
83. J. Nunnally. 1978. Psychometric Methods. McGraw-Hill.
84. M. Paasivaara, B. Behm, C. Lassenius, and M. Hallikainen. 2018. Large-scale agile transformation at Ericsson: A case study. Empir. Softw. Eng. 23, 5 (2018), 2550–2596.
85. M. Paasivaara, C. Lassenius, and V. Heikkilä. 2012. Inter-team coordination in large-scale globally distributed scrum: Do scrum-of-scrums really work? In International Symposium on Empirical Software Engineering and Measurement. ACM/IEEE, 235–238.
86. S. Palan and C. Schitter. 2018. Prolific. A subject pool for online experiments. J. Behav. Exper. Finan. 17 (2018), 22–27.
87. D. Paulhus. 1991. Measurement and control of response bias. In Measures of Personality and Social Psychological Attitudes. Academic Press.
88. E. Peer, L. Brandimarte, S. Samat, and A. Acquisti. 2017. Beyond the turk: Alternative platforms for crowdsourcing behavioral research. J. Experim. Soc. Psychol. 70 (2017), 153–163.
89. K. Petersen and C. Wohlin. 2010. The effect of moving from a plan-driven to an incremental software development approach with agile practices. Empir. Softw. Eng. 15, 6 (2010), 654–693.
90. J. Piaget. 1967. Logique et connaissance scientifique. Encyclopédie de la Pléiade (1967).
91. J. K. Pinto and J. E. Prescott. 1990. Planning and tactical factors in the project implementation process. J. Manag. Stud. 27, 3 (1990), 305–327.
92. K. Popper. 2005. The Logic of Scientific Discovery. Routledge.
93. D. Port and M. Korte. 2008. Comparative studies of the model evaluation criterions MMRE and PRED in software cost estimation research. In International Symposium on Empirical Software Engineering and Measurement. 51–60.
94. M. E Porter. 1997. Competitive strategy. Meas. Bus. Excell. 1, 2 (1997).
95. C. Prause and Z. Durdik. 2012. Architectural design and documentation: Waste in agile development? In International Conference on Software and System Process. IEEE, 130–134.
96. A. Putta, M. Paasivaara, and C. Lassenius. 2019. How are agile release trains formed in practice? A case study in a large financial corporation. In International Conference on Agile Software Development. Springer, 154–170.
97. H. S. Qiu, A. Nolte, A. Brown, A. Serebrenik, and B. Vasilescu. 2019. Going farther together: The impact of social capital on sustained participation in open source. In International Conference on Software Engineering. IEEE, 688–699.
98. Ragioneria Generale dello Stato. 2013–2018. Bilancio della Difesa. Retrieved from https://www.difesa.it/Amministrazionetrasparente/bilandife/Pagine/Bilanciopreventivoeconsuntivo.aspx.
99. P. Ralph et al. 2020. ACM SIGSOFT empirical standards. arxiv:2010.03525 (2020).
100. M. Ramkumar, T. Schoenherr, S. Wagner, and M. Jenamani. 2019. Q-TAM: A quality technology acceptance model for predicting organizational buyers’ continuance intentions for e-procurement services. Int. J. Product. Econ. 216 (2019), 333–348.
101. T. Ravichandran, C. Lertwongsatien, and C. Lertwongsatien. 2005. Effect of information systems resources and capabilities on firm performance: A resource-based perspective. J. Manag. Inf. Syst. 21, 4 (2005), 237–276.
102. D. Reifer. 2004. Industry software cost, quality and productivity benchmarks. DoD Softw. Tech News 7, 2 (2004), 3–19.
103. C. M. Ringle and M. Sarstedt. 2016. Gain more insight from your PLS-SEM results: The importance-performance map analysis. Industr. Manag. Data Syste. 116, 9 (2016), 1865–1886.
104. C. M. Ringle, S. Wende, and J.-M. Becker. 2015. SmartPLS 3. https://www.smartpls.com/.
105. D. Russo, P. Ciancarini, T. Falasconi, and M. Tomasi. 2018. A meta model for information systems quality: A mixed-study of the financial sector. ACM Trans. Manag. Inf. Syst. 9, 3 (2018).
106. D. Russo, P. H. P. Hanel, S. Altnickel, and N. van Berkel. 2021. The daily life of software engineers during the COVID-19 pandemic. In International Conference on Software Engineering.
107. D. Russo, P. H. P. Hanel, S. Altnickel, and N. van Berkel. 2021. Predictors of well-being and productivity of software professionals during the COVID-19 pandemic–A longitudinal study. Empir. Softw. Eng. 26, 4 (2021), 62. DOI:10.1007/s10664-021-09945-9
108. D. Russo and K.-J. Stol. 2019. Soft theory: A pragmatic alternative to conduct quantitative empirical studies. In Joint International Workshop on Conducting Empirical Studies in Industry and International Workshop on Software Engineering Research and Industrial Practice. IEEE, 1–4.
109. D. Russo and K.-J. Stol. 2020. Gender differences in personality traits of software engineers. IEEE Trans. Softw. Eng. In Press (2020), 16.
110. D. Russo and K.-J. Stol. 2021. PLS-SEM for software engineering research: An introduction and survey. Comput. Surv. 54, 4 (2021).
111. D. Russo, G. Taccogna, P. Ciancarini, A. Messina, and G. Succi. 2018. Contracting Agile developments for mission critical systems in the public sector. In International Conference on Software Engineering. ACM.
112. M. Sarstedt, J. Henseler, and C. Ringle. 2011. Multigroup analysis in partial least squares (PLS) path modeling: Alternative methods and empirical results. In Measurement and Research Methods in International Marketing. Emerald, 195–218.
113. K. Schwaber and J. Sutherland. 2020. The Scrum Guide. Scrum Alliance. Technical Report.
114. B. Selic. 2009. Agile documentation, anyone?IEEE Softw. 26, 6 (2009), 11–12.
115. G. G. Sharma and K.-J. Stol. 2020. Exploring onboarding success, organizational fit, and turnover intention of software professionals. J. Syst. Softw. 159 (2020), 110442.
116. Galit Shmueli, Soumya Ray, Juan Manuel Velasquez Estrada, and Suneel Babu Chatla. 2016. The elephant in the room: Predictive performance of PLS models. J. Bus. Res. 69, 10 (2016), 4552–4564.
117. G. Shmueli, M. Sarstedt, J. F. Hair, J Cheah, H. Ting, S. Vaithilingam, and C. M. Ringle. 2019. Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. Eur. J. Market. 53, 11 (2019).
118. A. Sillitti and G. Succi. 2005. Requirements engineering for agile methods. In Engineering and Managing Software Requirements. Springer, 309–326.
119. B. Slife and R. Williams. 1995. What’s Behind the Research?: Discovering Hidden Assumptions in the Behavioral Sciences. Sage.
120. B. Snyder and B. Curtis. 2017. Using analytics to guide improvement during an Agile–DevOps transformation. IEEE Softw. 35, 1 (2017), 78–83.
121. K.-J. Stol and B. Fitzgerald. 2018. The ABC of software engineering research. ACM Trans. Softw. Eng. Methodol. 27, 3 (2018), 11.
122. M. Stone. 1974. Cross-validatory choice and assessment of statistical predictions. J. Royal Statist. Soc. Series B (Methodol.) 36, 2 (1974), 111–147.
123. A. Strauss and J. Corbin. 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques.Sage.
124. A. Strauss and J. Corbin. 1997. Grounded Theory in Practice. Sage.
125. V. Stray, N. B. Moe, and R. Hoda. 2018. Autonomous agile teams: Challenges and future directions for research. In International Conference on Agile Software Development. 1–5.
126. J. Sutherland. 2001. Agile can scale: Inventing and reinventing Scrum in five companies. Cutter IT J. 14, 12 (2001), 5–11.
127. D. Talby, A. Keren, O. Hazzan, and Y. Dubinsky. 2006. Agile software testing in a large-scale project. IEEE Softw. 23, 4 (2006), 30–37.
128. P. E. Tesluk and J. E. Mathieu. 1999. Overcoming roadblocks to effectiveness: Incorporating management of performance barriers into models of work group effectiveness.J. Appl. Psychol. 84, 2 (1999), 200.
129. A. Tiwana and E. Mclean. 2005. Expertise integration and creativity in information systems development. J. Manag. Inf. Syst. 22, 1 (2005), 13–43.
130. J. Van Maanen. 1979. The fact of fiction in organizational ethnography. Admin. Sci. Quart. 24, 4 (1979), 539–550.
131. B. Vasilescu, A. Capiluppi, and A. Serebrenik. 2014. Gender, representation and online participation: A quantitative study. Interact. Comput. 26, 5 (2014), 488–511.
132. VersionOne. 2019. 13th Annual State of Agile Survey. Retrieved from http://stateofagile.versionone.com/.
133. L. M. Wagner, E. Capezuti, and J. C. Rice. 2009. Nurses’ perceptions of safety culture in long-term care settings. J. Nurs. Scholar. 41, 2 (2009), 184–192.
134. B. C. Wheeler. 2002. NEBIC: A dynamic capabilities theory for assessing net-enablement. Inf. Syst. Res. 13, 2 (2002), 125–146.
135. C. Wohlin, P. Runeson, M. Höst, M. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer Science & Business Media.
136. R. K. Yin. 1994. Case study research: Design and methods, applied social research. Meth. Ser. 5 (1994), 312.
137. R. Young and E. Jordan. 2008. Top management support: Mantra or necessity?Int. J. Proj. Manag. 26, 7 (2008), 713–725.

Index Terms

• Social and professional topics

  • Professional topics

    • Computing industry

      • Sustainability

    • Management of computing and information systems

      • Project and people management

      • Software management

• Software and its engineering

  • Software creation and management

    • Software development process management

      • Software development methods

        • Agile software development