Abstract
Context: End-user service composition (EUSC) is a service-oriented paradigm that aims to empower end users and allow them to compose their own web applications from reusable service components. User studies have been used to evaluate EUSC tools and processes. Such an approach should benefit software development, because incorporating end users’ feedback into software development should make software more useful and usable. Problem: There is a gap in our understanding of what constitutes a user study and how a good user study should be designed, conducted, and reported. Goal: This article aims to address this gap. Method: The article presents a systematic review of 47 selected user studies for EUSC. Guided by a review framework, the article systematically and consistently assesses the focus, methodology and cohesion of each of these studies. Results: The article concludes that the focus of these studies is clear, but their methodology is incomplete and inadequate, their overall cohesion is poor. The findings lead to the development of a design framework and a set of questions for the design, reporting, and review of good user studies for EUSC. The detailed analysis and the insights obtained from the analysis should be applicable to the design of user studies for service-oriented systems as well and indeed for any user studies related to software artifacts.
- F. Daniel and M. Matera. 2014. Mashups: Concepts, Models and Architectures. Springer. Google Scholar
Digital Library
- L. Xuanzhe, M. Yun, H. Gang, Z. Junfeng, M. Hong, and L. Yunxin. 2015. Data-driven composition for service-oriented situational web applications. IEEE Trans. Serv. Comput. 8 (2015), 2--16.Google Scholar
Cross Ref
- B. Nuseibeh and S. Easterbrook. 2000. Requirements engineering: A roadmap. In Proceedings of the Conference on the Future of Software Engineering, 35--46. Google Scholar
Digital Library
- H. Lieberman, F. Paternò, M. Klann, and V. Wulf. 2006. End-user Development: An Emerging Paradigm. Springer.Google Scholar
Digital Library
- D. Tetteroo and P. Markopoulos. 2015. A review of research methods in end user development. In End-User Development, vol. 9083, P. Díaz, V. Pipek, C. Ardito, C. Jensen, I. Aedo, and A. Boden (Eds.). Springer International Publishing, 58--75.Google Scholar
- A. Namoun, T. Nestler, and A. De Angeli. 2010. Conceptual and usability issues in the composable web of software services. In Current Trends in Web Engineering, vol. 6385, F. Daniel and F. Facca (Eds.). Springer, Berlin, 396--407. Google Scholar
Digital Library
- A. Namoun, T. Nestler, and A. De Angeli. 2010. Service composition for non-programmers: Prospects, problems, and design recommendations. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS’10), 123--130. Google Scholar
Digital Library
- Ž. Obrenović and D. Gašević. 2008. End-user service computing: Spreadsheets as a service composition tool. IEEE Trans. Serv. Comput. 1 (2008), 229--242. Google Scholar
Digital Library
- W. M. Newman, M. G. Lamming, and M. Lamming. 1995. Interactive System Design. Addison-Wesley Reading.Google Scholar
- A. Bouguettaya, M. Singh, M. Huhns, Q. Z. Sheng, H. Dong, and Q. Yu et al. 2017. A service computing manifesto: the next 10 years. Commun. ACM 60 (2017), 64--72. Google Scholar
Digital Library
- S. Balasubramaniam, G. Lewis, S. Simanta, and D. B. Smith. 2008. Situated software: Concepts, motivation, technology, and the future. IEEE Softw. 25 (2008), 50--55. Google Scholar
Digital Library
- S. Aghaee and C. Pautasso. 2010. Mashup development with HTML5. In Proceedings of the 3rd and 4th International Workshops on Web APIs and Services Mashups, 10. Google Scholar
Digital Library
- D. Benslimane, S. Dustdar, and A. Sheth. 2008. Services mashups: The new generation of web applications. IEEE Internet Comput. 12, 5 (2008), 13--15. Google Scholar
Digital Library
- C. Cappiello, F. Daniel, M. Matera, M. Picozzi, and M. Weiss. 2011. Enabling end user development through mashups: requirements, abstractions and innovation toolkits. In Proceedings of the International Symposium on End User Development, 9--24. Google Scholar
Digital Library
- J. Yu, B. Benatallah, F. Casati, and F. Daniel. 2008. Understanding mashup development. IEEE Internet Comput. 12, 5 (2008). Google Scholar
Digital Library
- D. Lizcano, J. Soriano, M. Reyes, and J. J. Hierro. 2008. EzWeb/FAST: Reporting on a successful mashup-based solution for developing and deploying composite applications in the upcoming web of services. In Proceedings of the 10th International Conference on Information Integration and Web-based Applications and Services, 15--24. Google Scholar
Digital Library
- F. Daniel and M. Matera. 2014. Mashups and end-user development. In Mashups, Springer, Berlin, 237--268.Google Scholar
- N. Mehandjiev, F. Lecue, U. Wajid, and A. Namoun. 2010. Assisted service composition for end users. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS), 131--138. Google Scholar
Digital Library
- S. S. Minhas, P. Sampaio, and N. Mehandjiev. 2012. A Framework for the evaluation of mashup tools. In Proceedings of the IEEE 9th International Conference on Services Computing (SCC), 431--438. Google Scholar
Digital Library
- E. M. Maximilien, A. Ranabahu, and K. Gomadam. 2008. An online patform for web APIs and service mashups. IEEE Internet Comput. 12 (2008). Google Scholar
Digital Library
- J. Lin, J. Wong, J. Nichols, A. Cypher, and T. A. Lau. 2009. End-user programming of mashups with vegemite. In Proceedings of 14th International Conference on Intelligent User Interfaces, 97--106. Google Scholar
Digital Library
- S. S. Minhas, P. Sampaio, and N. Mehandjiev. 2012. A framework for the evaluation of mashup tools. In Proceedings of 9th IEEE International Conference on Services Computing (SCC’12), 431--438. Google Scholar
Digital Library
- M. C. Jones and E. F. Churchill. 2009. Conversations in developer communities: A preliminary analysis of the yahoo! pipes community. In Proceedings of the 4th International Conference on Communities and Technologies, 195--204. Google Scholar
Digital Library
- D. Lizcano, F. Alonso, J. Soriano, and G. López. 2014. A component- and connector-based approach for end-user composite web applications development. J. Syst. Softw. 94 (2014), 108-128, 8// 2014.Google Scholar
Cross Ref
- N. Mehandjiev, A. Namoune, U. Wajid, L. Macaulay, and A. Sutcliffe. 2010. End user service composition: Perceptions and requirements. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS’10), 139--146. Google Scholar
Digital Library
- I. Weber, H.-Y. Paik, and B. Benatallah. 2013. Form-based web service composition for domain experts. ACM Trans. Web 8 (2013), 1--40. Google Scholar
Digital Library
- R. Tuchinda, C. A. Knoblock, and P. Szekely. 2011. Building mashups by demonstration. ACM Trans. Web 5 (2011), 1--45. Google Scholar
Digital Library
- M. Matera, M. Picozzi, M. Pini, and M. Tonazzo. 2013. PEUDOM: A mashup platform for the end user development of common information spaces. In Web Engineering. Springer, 494--497. Google Scholar
Digital Library
- S. Aghaee and C. Pautasso. 2014. End-user development of mashups with naturalmash. J. Vis. Lang. Comput. 25 (2014), 414--432, 8// 2014. Google Scholar
Digital Library
- ISO/IEC. 2011. Systems and software engineering—Systems and software quality requirements and evaluation (SQuaRE) —System and software quality models. ISO/IEC FDIS 25010. Retrieved on from https://www.iso.org/standard/35733.html.Google Scholar
- P. Runeson, M. Host, A. Rainer, and B. Regnell. 2012. Case Study Research in Software Engineering: Guidelines and Examples. John Wiley 8 Sons. Google Scholar
Digital Library
- H. Sharp, Y. Dittrich, and C. R. B. de Souza. 2016. The role of ethnographic studies in empirical software engineering. IEEE Trans. Softw. Engineer. 42 (2016), 786--804. Google Scholar
Digital Library
- J. Rubin and D. Chisnell. 2008. Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. John Wiley 8 Sons. Google Scholar
Digital Library
- J. Nielsen. 1994. Usability Engineering. Elsevier.Google Scholar
- J. S. Dumas and J. Redish. 1999. A Practical Guide to Usability Testing. Intellect Books. Google Scholar
Digital Library
- G. Salvendy. 2012. Handbook of Human Factors and Ergonomics. John Wiley 8 Sons. Google Scholar
Digital Library
- J. R. Lewis. 2012. Usability testing. In Handbook of Human Factors and Ergonomics, G. Salvendy (Ed.). John Wiley 8 Sons.Google Scholar
- B. Gillham. 2007. The Case Study Handbook. Harvard Business School, Boston, MA.Google Scholar
- J. A. Maxwell. 2008. Designing a qualitative study. SAGE Handbook Appl. Soc. Res. Methods 2 (2008), 214--253.Google Scholar
- S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. 2008. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering, Springer, 285--311.Google Scholar
- P. Runeson and M. Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Engineer. 14 (2009), 131--164. Google Scholar
Digital Library
- C. B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Trans. Softw. Engineer. 25 (1999), 557--572. Google Scholar
Digital Library
- T. C. Lethbridge, S. E. Sim, and J. Singer. 2005. Studying software engineers: Data collection techniques for software field studies. Empir. Softw. Engineer. 10 (2005), 311--341. Google Scholar
Digital Library
- M. Van Den Haak, M. De Jong, and P. Jan Schellens. 2003. Retrospective vs. Concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behav. Info. Technol. 22 (2003), 339--351.Google Scholar
Cross Ref
- J. Kontio, J. Bragge, and L. Lehtola. 2008. The focus group method as an empirical tool in software engineering. In Guide to Advanced Empirical Software Engineering, Springer, 93--116.Google Scholar
- H. Priest, P. Roberts, and L. Woods. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 1: Theoretical issues. Nurse Res. (Through 2013) 10 (2002), 43.Google Scholar
- L. Woods, H. Priest, and P. Roberts. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 2: practical illustrations. Nurse Res. (Through 2013) 10 (2002), 43.Google Scholar
- C. Robson and K. McCartan. 2016. Real World Research. Wiley.Google Scholar
- C. Andersson and P. Runeson. 2007. A spiral process model for case studies on software quality monitoring—method and metrics. Softw. Process: Improve. Pract. 12 (2007), 125--140.Google Scholar
Cross Ref
- B. Kitchenham, L. Pickard, and S. L. Pfleeger. 1995. Case studies for method and tool evaluation. IEEE Softw. 12 (1995), 52. Google Scholar
Digital Library
- B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam et al. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Engineer. 28 (2002), 721--734. Google Scholar
Digital Library
- C. Wohlin, M. Höst, and K. Henningsson. 2003. Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering: Experiences from ESERNET. Springer, 7--23. Google Scholar
Digital Library
- B. Kitchenham, H. Al-Khilidar, M. A. Babar, M. Berry, K. Cox, J. Keung et al. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Engineer. 13 (2008), 97--121. Google Scholar
Digital Library
- D. E. Perry, S. E. Sim, and S. M. Easterbrook. 2005. Case studies for software engineers. In Proceedings of the NASA SW Engineering Workshop Tutorial, 736--738. Google Scholar
Digital Library
- R. K. Yin. 2013. Case Study Research: Design and Methods. Sage Publications.Google Scholar
- B. Kitchenham et al. 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University Keele, Staffs, ST5 5BG, UK Technical Report EBSE-2007-01, 9 July 2007.Google Scholar
- M. Kuhrmann, D. M. Fernández, and M. Daneva. 2017. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir. Softw. Engineer. 22 (2017), 2852--2891. Google Scholar
Digital Library
- P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80 (2007), 571--583. Google Scholar
Digital Library
- R. K. Yin. 1994. Case Study Research: Design and Methods, 2nd ed. Sage Publications.Google Scholar
- D. M. Fernández and J.-H. Passoth. 2018. Empirical software engineering: From discipline to interdiscipline. arXiv preprint arXiv:1805.08302.Google Scholar
- F. Q. B. da Silva, M. Suassuna, A. C. C. França, A. M. Grubb, T. B. Gouveia, C. V. F. Monteiro et al. 2014. Replication of empirical studies in software engineering research: A systematic mapping study. Empir. Softw. Engineer. 19 (2014), 501--557, 2014// 2014. Google Scholar
Digital Library
- K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. Google Scholar
Digital Library
- K. Petersen, S. Vakkalanka, and L. Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Info. Softw. Technol. 64 (2015), 1--18. Google Scholar
Digital Library
- Y. Zhou, H. Zhang, X. Huang, S. Yang, M. A. Babar, and H. Tang. 2015. Quality assessment of systematic reviews in software engineering: A tertiary study. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 14. Google Scholar
Digital Library
- M. Kuhrmann, C. Konopka, P. Nellemann, P. Diebold, and J. Münch. 2015. Software process improvement: where is the evidence?: initial findings from a systematic mapping study. In Proceedings of the International Conference on Software and System Process, 107--116. Google Scholar
Digital Library
- C. Wohlin, P. Runeson, P. A. d. M. S. Neto, E. Engström, I. do Carmo Machado, and E. S. De Almeida. 2013. On the reliability of mapping studies in software engineering. J. Syst. Softw. 86 (2013), 2594--2610.Google Scholar
Cross Ref
- D. I. K. Sjoeberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahasanovic, N. K. Liborg et al. 2005. A survey of controlled experiments in software engineering. IEEE Trans. Softw. Engineer. 31 (2005), 733--753. Google Scholar
Digital Library
Index Terms
User Studies on End-User Service Composition: A Literature Review and a Design Framework
Recommendations
Supporting End-User Service Composition: A Systematic Review of Current Activities and Tools
ICWS '15: Proceedings of the 2015 IEEE International Conference on Web ServicesThis paper presents a systematic literature review of end-user service composition. It reviews current activities performed by end users, and tools and approaches that enable them to compose and develop service systems from Web Services. The paper also ...
Ontology-driven service composition for end-users
Current service composition techniques and tools are mainly designed for use by Service-Oriented Architecture (SOA) professionals to solve business problems. Little attention has been paid to allowing end-users without sufficient service composition ...
Towards End User Service Composition
COMPSAC '07: Proceedings of the 31st Annual International Computer Software and Applications Conference - Volume 01The popularity of Service Oriented Computing (SOC) brings a large number of distributed, well-encapsulated and reusable services all over internet, and makes it possible to create value-added services by means of service composition. Current composition ...






Comments