skip to main content
tutorial

User Studies on End-User Service Composition: A Literature Review and a Design Framework

Authors Info & Claims
Published:26 July 2019Publication History
Skip Abstract Section

Abstract

Context: End-user service composition (EUSC) is a service-oriented paradigm that aims to empower end users and allow them to compose their own web applications from reusable service components. User studies have been used to evaluate EUSC tools and processes. Such an approach should benefit software development, because incorporating end users’ feedback into software development should make software more useful and usable. Problem: There is a gap in our understanding of what constitutes a user study and how a good user study should be designed, conducted, and reported. Goal: This article aims to address this gap. Method: The article presents a systematic review of 47 selected user studies for EUSC. Guided by a review framework, the article systematically and consistently assesses the focus, methodology and cohesion of each of these studies. Results: The article concludes that the focus of these studies is clear, but their methodology is incomplete and inadequate, their overall cohesion is poor. The findings lead to the development of a design framework and a set of questions for the design, reporting, and review of good user studies for EUSC. The detailed analysis and the insights obtained from the analysis should be applicable to the design of user studies for service-oriented systems as well and indeed for any user studies related to software artifacts.

References

  1. F. Daniel and M. Matera. 2014. Mashups: Concepts, Models and Architectures. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. L. Xuanzhe, M. Yun, H. Gang, Z. Junfeng, M. Hong, and L. Yunxin. 2015. Data-driven composition for service-oriented situational web applications. IEEE Trans. Serv. Comput. 8 (2015), 2--16.Google ScholarGoogle ScholarCross RefCross Ref
  3. B. Nuseibeh and S. Easterbrook. 2000. Requirements engineering: A roadmap. In Proceedings of the Conference on the Future of Software Engineering, 35--46. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. H. Lieberman, F. Paternò, M. Klann, and V. Wulf. 2006. End-user Development: An Emerging Paradigm. Springer.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. D. Tetteroo and P. Markopoulos. 2015. A review of research methods in end user development. In End-User Development, vol. 9083, P. Díaz, V. Pipek, C. Ardito, C. Jensen, I. Aedo, and A. Boden (Eds.). Springer International Publishing, 58--75.Google ScholarGoogle Scholar
  6. A. Namoun, T. Nestler, and A. De Angeli. 2010. Conceptual and usability issues in the composable web of software services. In Current Trends in Web Engineering, vol. 6385, F. Daniel and F. Facca (Eds.). Springer, Berlin, 396--407. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. A. Namoun, T. Nestler, and A. De Angeli. 2010. Service composition for non-programmers: Prospects, problems, and design recommendations. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS’10), 123--130. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Ž. Obrenović and D. Gašević. 2008. End-user service computing: Spreadsheets as a service composition tool. IEEE Trans. Serv. Comput. 1 (2008), 229--242. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. W. M. Newman, M. G. Lamming, and M. Lamming. 1995. Interactive System Design. Addison-Wesley Reading.Google ScholarGoogle Scholar
  10. A. Bouguettaya, M. Singh, M. Huhns, Q. Z. Sheng, H. Dong, and Q. Yu et al. 2017. A service computing manifesto: the next 10 years. Commun. ACM 60 (2017), 64--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. S. Balasubramaniam, G. Lewis, S. Simanta, and D. B. Smith. 2008. Situated software: Concepts, motivation, technology, and the future. IEEE Softw. 25 (2008), 50--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. Aghaee and C. Pautasso. 2010. Mashup development with HTML5. In Proceedings of the 3rd and 4th International Workshops on Web APIs and Services Mashups, 10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. D. Benslimane, S. Dustdar, and A. Sheth. 2008. Services mashups: The new generation of web applications. IEEE Internet Comput. 12, 5 (2008), 13--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. C. Cappiello, F. Daniel, M. Matera, M. Picozzi, and M. Weiss. 2011. Enabling end user development through mashups: requirements, abstractions and innovation toolkits. In Proceedings of the International Symposium on End User Development, 9--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. J. Yu, B. Benatallah, F. Casati, and F. Daniel. 2008. Understanding mashup development. IEEE Internet Comput. 12, 5 (2008). Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. Lizcano, J. Soriano, M. Reyes, and J. J. Hierro. 2008. EzWeb/FAST: Reporting on a successful mashup-based solution for developing and deploying composite applications in the upcoming web of services. In Proceedings of the 10th International Conference on Information Integration and Web-based Applications and Services, 15--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. F. Daniel and M. Matera. 2014. Mashups and end-user development. In Mashups, Springer, Berlin, 237--268.Google ScholarGoogle Scholar
  18. N. Mehandjiev, F. Lecue, U. Wajid, and A. Namoun. 2010. Assisted service composition for end users. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS), 131--138. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. S. S. Minhas, P. Sampaio, and N. Mehandjiev. 2012. A Framework for the evaluation of mashup tools. In Proceedings of the IEEE 9th International Conference on Services Computing (SCC), 431--438. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. E. M. Maximilien, A. Ranabahu, and K. Gomadam. 2008. An online patform for web APIs and service mashups. IEEE Internet Comput. 12 (2008). Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. J. Lin, J. Wong, J. Nichols, A. Cypher, and T. A. Lau. 2009. End-user programming of mashups with vegemite. In Proceedings of 14th International Conference on Intelligent User Interfaces, 97--106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. S. S. Minhas, P. Sampaio, and N. Mehandjiev. 2012. A framework for the evaluation of mashup tools. In Proceedings of 9th IEEE International Conference on Services Computing (SCC’12), 431--438. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. M. C. Jones and E. F. Churchill. 2009. Conversations in developer communities: A preliminary analysis of the yahoo! pipes community. In Proceedings of the 4th International Conference on Communities and Technologies, 195--204. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. D. Lizcano, F. Alonso, J. Soriano, and G. López. 2014. A component- and connector-based approach for end-user composite web applications development. J. Syst. Softw. 94 (2014), 108-128, 8// 2014.Google ScholarGoogle ScholarCross RefCross Ref
  25. N. Mehandjiev, A. Namoune, U. Wajid, L. Macaulay, and A. Sutcliffe. 2010. End user service composition: Perceptions and requirements. In Proceedings of the IEEE 8th European Conference on Web Services (ECOWS’10), 139--146. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. I. Weber, H.-Y. Paik, and B. Benatallah. 2013. Form-based web service composition for domain experts. ACM Trans. Web 8 (2013), 1--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. R. Tuchinda, C. A. Knoblock, and P. Szekely. 2011. Building mashups by demonstration. ACM Trans. Web 5 (2011), 1--45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. M. Matera, M. Picozzi, M. Pini, and M. Tonazzo. 2013. PEUDOM: A mashup platform for the end user development of common information spaces. In Web Engineering. Springer, 494--497. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. S. Aghaee and C. Pautasso. 2014. End-user development of mashups with naturalmash. J. Vis. Lang. Comput. 25 (2014), 414--432, 8// 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. ISO/IEC. 2011. Systems and software engineering—Systems and software quality requirements and evaluation (SQuaRE) —System and software quality models. ISO/IEC FDIS 25010. Retrieved on from https://www.iso.org/standard/35733.html.Google ScholarGoogle Scholar
  31. P. Runeson, M. Host, A. Rainer, and B. Regnell. 2012. Case Study Research in Software Engineering: Guidelines and Examples. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. H. Sharp, Y. Dittrich, and C. R. B. de Souza. 2016. The role of ethnographic studies in empirical software engineering. IEEE Trans. Softw. Engineer. 42 (2016), 786--804. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. J. Rubin and D. Chisnell. 2008. Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. J. Nielsen. 1994. Usability Engineering. Elsevier.Google ScholarGoogle Scholar
  35. J. S. Dumas and J. Redish. 1999. A Practical Guide to Usability Testing. Intellect Books. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. G. Salvendy. 2012. Handbook of Human Factors and Ergonomics. John Wiley 8 Sons. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. J. R. Lewis. 2012. Usability testing. In Handbook of Human Factors and Ergonomics, G. Salvendy (Ed.). John Wiley 8 Sons.Google ScholarGoogle Scholar
  38. B. Gillham. 2007. The Case Study Handbook. Harvard Business School, Boston, MA.Google ScholarGoogle Scholar
  39. J. A. Maxwell. 2008. Designing a qualitative study. SAGE Handbook Appl. Soc. Res. Methods 2 (2008), 214--253.Google ScholarGoogle Scholar
  40. S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. 2008. Selecting empirical methods for software engineering research. In Guide to Advanced Empirical Software Engineering, Springer, 285--311.Google ScholarGoogle Scholar
  41. P. Runeson and M. Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Engineer. 14 (2009), 131--164. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. C. B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Trans. Softw. Engineer. 25 (1999), 557--572. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. T. C. Lethbridge, S. E. Sim, and J. Singer. 2005. Studying software engineers: Data collection techniques for software field studies. Empir. Softw. Engineer. 10 (2005), 311--341. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. M. Van Den Haak, M. De Jong, and P. Jan Schellens. 2003. Retrospective vs. Concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behav. Info. Technol. 22 (2003), 339--351.Google ScholarGoogle ScholarCross RefCross Ref
  45. J. Kontio, J. Bragge, and L. Lehtola. 2008. The focus group method as an empirical tool in software engineering. In Guide to Advanced Empirical Software Engineering, Springer, 93--116.Google ScholarGoogle Scholar
  46. H. Priest, P. Roberts, and L. Woods. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 1: Theoretical issues. Nurse Res. (Through 2013) 10 (2002), 43.Google ScholarGoogle Scholar
  47. L. Woods, H. Priest, and P. Roberts. 2002. An overview of three different approaches to the interpretation of qualitative data. Part 2: practical illustrations. Nurse Res. (Through 2013) 10 (2002), 43.Google ScholarGoogle Scholar
  48. C. Robson and K. McCartan. 2016. Real World Research. Wiley.Google ScholarGoogle Scholar
  49. C. Andersson and P. Runeson. 2007. A spiral process model for case studies on software quality monitoring—method and metrics. Softw. Process: Improve. Pract. 12 (2007), 125--140.Google ScholarGoogle ScholarCross RefCross Ref
  50. B. Kitchenham, L. Pickard, and S. L. Pfleeger. 1995. Case studies for method and tool evaluation. IEEE Softw. 12 (1995), 52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam et al. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Engineer. 28 (2002), 721--734. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. C. Wohlin, M. Höst, and K. Henningsson. 2003. Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering: Experiences from ESERNET. Springer, 7--23. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. B. Kitchenham, H. Al-Khilidar, M. A. Babar, M. Berry, K. Cox, J. Keung et al. 2008. Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Engineer. 13 (2008), 97--121. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. D. E. Perry, S. E. Sim, and S. M. Easterbrook. 2005. Case studies for software engineers. In Proceedings of the NASA SW Engineering Workshop Tutorial, 736--738. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. R. K. Yin. 2013. Case Study Research: Design and Methods. Sage Publications.Google ScholarGoogle Scholar
  56. B. Kitchenham et al. 2007. Guidelines for performing systematic literature reviews in software engineering. Keele University Keele, Staffs, ST5 5BG, UK Technical Report EBSE-2007-01, 9 July 2007.Google ScholarGoogle Scholar
  57. M. Kuhrmann, D. M. Fernández, and M. Daneva. 2017. On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir. Softw. Engineer. 22 (2017), 2852--2891. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80 (2007), 571--583. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. R. K. Yin. 1994. Case Study Research: Design and Methods, 2nd ed. Sage Publications.Google ScholarGoogle Scholar
  60. D. M. Fernández and J.-H. Passoth. 2018. Empirical software engineering: From discipline to interdiscipline. arXiv preprint arXiv:1805.08302.Google ScholarGoogle Scholar
  61. F. Q. B. da Silva, M. Suassuna, A. C. C. França, A. M. Grubb, T. B. Gouveia, C. V. F. Monteiro et al. 2014. Replication of empirical studies in software engineering research: A systematic mapping study. Empir. Softw. Engineer. 19 (2014), 501--557, 2014// 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson. 2008. Systematic mapping studies in software engineering. In Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. K. Petersen, S. Vakkalanka, and L. Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Info. Softw. Technol. 64 (2015), 1--18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Y. Zhou, H. Zhang, X. Huang, S. Yang, M. A. Babar, and H. Tang. 2015. Quality assessment of systematic reviews in software engineering: A tertiary study. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. M. Kuhrmann, C. Konopka, P. Nellemann, P. Diebold, and J. Münch. 2015. Software process improvement: where is the evidence?: initial findings from a systematic mapping study. In Proceedings of the International Conference on Software and System Process, 107--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. C. Wohlin, P. Runeson, P. A. d. M. S. Neto, E. Engström, I. do Carmo Machado, and E. S. De Almeida. 2013. On the reliability of mapping studies in software engineering. J. Syst. Softw. 86 (2013), 2594--2610.Google ScholarGoogle ScholarCross RefCross Ref
  67. D. I. K. Sjoeberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahasanovic, N. K. Liborg et al. 2005. A survey of controlled experiments in software engineering. IEEE Trans. Softw. Engineer. 31 (2005), 733--753. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. User Studies on End-User Service Composition: A Literature Review and a Design Framework

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!