skip to main content
research-article

CrowdUI: Supporting Web Design with the Crowd

Published:18 June 2020Publication History
Skip Abstract Section

Abstract

Web design is a complex and challenging task. It involves making many design decisions that materialise preconceived notions of user needs that may or may not be true. In this paper, we investigate supporting the co-design of a website with visual feedback elicited from the website's community of users. Website users can express their needs by re-arranging and modifying the website's layout and design. To explore and validate this idea, we present CrowdUI, a web-based tool that enables members of the community of a website to visually express their design improvement ideas, frustrations and needs, and to send this feedback to the person in charge of designing or maintaining the website.

CrowdUI is validated in a study with 45 users of a popular social media and networking website. Second, our qualitative evaluation with 60 experienced web developers shows that CrowdUI is able to elicit diverse and meaningful feedback. Put together, our results suggest that CrowdUI's approach constitutes a productive setting for eliciting visual feedback from the user community as a complement to traditional ways of eliciting feedback and participatory design. Finally, based on our experiences, we discuss a design space for crowdsourced web design and provide design recommendations for similar future tools.

References

  1. Chadia Abras, Diane Maloney-Krichmar, and Jenny Preece. 2004. User-Centered Design. In Berkshire Encyclopedia of Human-Computer Interaction 2nd ed.), William S. Bainbridge (Ed.). Berkshire Publishing Company, Ltd., Great Barrington, MA, USA, 763--768.Google ScholarGoogle Scholar
  2. Pierre Akiki, Arosha Bandara, and Yijun Yu. 2013. Crowdsourcing User Interface Adaptations for Minimizing the Bloat in Enterprise Applications. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 121--126. https://doi.org/10.1145/2494603.2480319Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Salvatore Andolina, Hendrik Schneider, Joel Chan, Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017. Crowdboard: Augmenting In-Person Idea Generation with Real-Time Crowds. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17). ACM, New York, NY, USA, 106--118. https://doi.org/10.1145/3059454.3059477Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Goetz Botterweck. 2011. Multi Front-End Engineering. In Model-Driven Development of Advanced User Interfaces, Heinrich Hussmann, Gerrit Meixner, and Detlef Zuehlke (Eds.). Springer, Berlin/Heidelberg, Germany, 27--42. https://doi.org/10.1007/978--3--642--14562--9_2Google ScholarGoogle Scholar
  5. Margaret M. Bradley and Peter J. Lang. 2007. The International Affective Picture System (IAPS) in the Study of Emotion and Attention. In Series in Affective Science. Handbook of Emotion Elicitation and Assessment, James A. Coan and John J. B. Allen (Eds.). Oxford University Press, New York, NY, USA, 29--46.Google ScholarGoogle Scholar
  6. Marco Brambilla, Stefano Ceri, Andrea Mauri, and Riccardo Volonterio. 2014. Community-based Crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web (WWW '14 Companion). ACM, New York, NY, USA, 891--896. https://doi.org/10.1145/2567948.2578835Google ScholarGoogle Scholar
  7. Virginia Braun, Victoria Clarke, Nikki Hayfield, and Gareth Terry. 2019. Thematic Analysis. In Handbook of Research Methods in Health Social Sciences, Pranee Liamputtong (Ed.). Springer, Singapore, 843--860. https://doi.org/10.1007/978--981--10--5251--4_103Google ScholarGoogle Scholar
  8. Andrea Bunt, Cristina Conati, and Joanna McGrenere. 2007. Supporting Interface Customization Using a Mixed-initiative Approach. In Proceedings of the 12th International Conference on Intelligent User Interfaces (IUI '07). ACM, New York, NY, USA, 92--101. https://doi.org/10.1145/1216295.1216317Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Javier Caminero, Mari Carmen Rodríguez-Gancedo, Jean Vanderdonckt, Fabio Paternò, Joerg Rett, Dave Raggett, Jean-Loup Comeliau, and Ignacio Marín. 2012. Multidimensional Context-Aware Adaptation of Service Front-Ends. In Context-Aware Adaptation of Service Front-Ends 2012. Proceedings of the Workshop on Context-Aware Adaptation of Service Front-Ends (CASFE 2012), Francisco Javier Caminero Gil, Fabio Paternò, and Jean Vanderdonckt (Eds.). CEUR-WS, 5 pages.Google ScholarGoogle Scholar
  10. Biplab Deka, Zifeng Huang, Chad Franzen, Jeffrey Nichols, Yang Li, and Ranjitha Kumar. 2017. ZIPT: Zero-Integration Performance Testing of Mobile App Designs. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 727--736. https://doi.org/10.1145/3126594.3126647Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Stephen W. Draper and Don A. Norman. 1986. User Centered System Design .Lawrence Erlbaum Associates, Hillsdale, NJ, USA. 85025207Google ScholarGoogle Scholar
  12. Facebook, Inc. 2019. Facebook Reports Second Quarter 2019 Results. Press release, 24 July, 2019.Google ScholarGoogle Scholar
  13. Eureka Foong, Steven P. Dow, Brian P. Bailey, and Elizabeth M. Gerber. 2017a. Online Feedback Exchange: A Framework for Understanding the Socio-Psychological Factors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4454--4467. https://doi.org/10.1145/3025453.3025791Google ScholarGoogle Scholar
  14. Eureka Foong, Darren Gergle, and Elizabeth M. Gerber. 2017b. Novice and Expert Sensemaking of Crowdsourced Design Feedback. Proceedings of the ACM on Human-Computer Interaction, Vol. 1, CSCW (Dec. 2017), 45:1--45:18. https://doi.org/10.1145/3134680Google ScholarGoogle Scholar
  15. Vivian Genaro Motti, Dave Raggett, Sascha Van Cauwelaert, and Jean Vanderdonckt. 2013. Simplifying the Development of Cross-Platform Web User Interfaces by Collaborative Model-Based Design. In Proceedings of the 31st ACM International Conference on Design of Communication (SIGDOC '13). ACM, New York, NY, USA, 55--64. https://doi.org/10.1145/2507065.2507067Google ScholarGoogle Scholar
  16. Michael D. Greenberg, Matthew W. Easterday, and Elizabeth M. Gerber. 2015. Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (C&C '15). ACM, New York, NY, USA, 235--244. https://doi.org/10.1145/2757226.2757249Google ScholarGoogle Scholar
  17. Eszter Hargittai and Gina Walejko. 2008. The Participation Divide: Content Creation and Sharing in the Digital Age. Information, Communication & Society, Vol. 11, 2 (2008), 239--256. https://doi.org/10.1080/13691180801946150Google ScholarGoogle ScholarCross RefCross Ref
  18. Marc Hassenzahl. 2004. The Thing and I: Understanding the Relationship Between User and Product. In Funology, Mark A. Blythe, Kees Overbeeke, Andrew F. Monk, and Peter C. Wright (Eds.). Kluwer Academic Publishers, Norwell, MA, USA, 31--42. https://doi.org/10.1007/1--4020--2967--5_4Google ScholarGoogle Scholar
  19. Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 203--212. https://doi.org/10.1145/1753326.1753357Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh, and Björn Hartmann. 2012. CommunitySourcing: Engaging Local Crowds to Perform Expert Work via Physical Kiosks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1539--1548. https://doi.org/10.1145/2207676.2208619Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hyeonsu B. Kang, Gabriel Amoako, Neil Sengupta, and Steven P. Dow. 2018. Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, 606:1--606:13. https://doi.org/10.1145/3173574.3174180Google ScholarGoogle Scholar
  22. Louise H. Kidder and Michelle Fine. 1987. Qualitative and quantitative methods: When stories converge. New Directions for Program Evaluation, Vol. 1987, 35 (1987), 57--75. https://doi.org/10.1002/ev.1459Google ScholarGoogle Scholar
  23. Nigel King, Joanna Brooks, and Saloomeh Tabari. 2018. Template Analysis in Business and Management Research. In Qualitative Methodologies in Organization Studies: Volume II: Methods and Possibilities, Malgorzata Ciesielska and Dariusz Jemielniak (Eds.). Springer, Cham, Switzerland, 179--206. https://doi.org/10.1007/978--3--319--65442--3_8Google ScholarGoogle Scholar
  24. Yuki Koyama. 2016. Computational Design Driven by Aesthetic Preference. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct). ACM, New York, NY, USA, 1--4. https://doi.org/10.1145/2984751.2984781Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3083--3092. https://doi.org/10.1145/2470654.2466420Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Benjamin Lafreniere, Andrea Bunt, Matthew Lount, Filip Krynicki, and Michael A. Terry. 2011. AdaptableGIMP: Designing a Socially-adaptable Interface. In Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology (UIST '11 Adjunct). ACM, New York, NY, USA, 89--90. https://doi.org/10.1145/2046396.2046437Google ScholarGoogle Scholar
  27. Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1925--1934. https://doi.org/10.1145/2702123.2702565Google ScholarGoogle Scholar
  28. Talia Lavie and Noam Tractinsky. 2004. Assessing Dimensions of Perceived Visual Aesthetics of Web Sites. International Journal of Human-Computer Studies, Vol. 60, 3 (March 2004), 269--298. https://doi.org/10.1016/j.ijhcs.2003.09.002Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O'Keefe, and Walter S. Lasecki. 2018. Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (Nov. 2018), 104:1--104:23. https://doi.org/10.1145/3274373Google ScholarGoogle Scholar
  30. Sang Won Lee, Yujin Zhang, Isabelle Wong, Yiwei Yang, Stephanie D. O'Keefe, and Walter S. Lasecki. 2017. SketchExpress: Remixing Animations for More Effective Crowd-Powered Prototyping of Interactive Interfaces. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 817--828. https://doi.org/10.1145/3126594.3126595Google ScholarGoogle Scholar
  31. Friedrich Leisch. 2008. Visualizing Cluster Analysis and Finite Mixture Models. In Handbook of Data Visualization, Chun-houh Chen, Wolfgang Karl Hardle, and Antony Unwin (Eds.). Springer, Berlin/Heidelberg, Germany, 561--587.Google ScholarGoogle Scholar
  32. Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P. Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 473--485. https://doi.org/10.1145/2675133.2675283Google ScholarGoogle Scholar
  33. Paul A. Mabe and Stephen West. 1982. Validity of Self-evaluation of Ability: A Review and Meta-analysis. Journal of Applied Psychology, Vol. 67, 3 (6 1982), 280--296. https://doi.org/10.1037/0021--9010.67.3.280Google ScholarGoogle ScholarCross RefCross Ref
  34. Wendy E. Mackay. 2004. The Interactive Thread: Exploring Methods for Multi-disciplinary Design. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS '04). ACM, New York, NY, USA, 103--112. https://doi.org/10.1145/1013115.1013131Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Joanna McGrenere and Gale Moore. 2000. Are We All in the Same "Bloat"?. In Proceedings of the Graphics Interface 2000 Conference. 187--196.Google ScholarGoogle Scholar
  36. John Michael Moore. 2007. Design Exploration: Engaging A Larger User Population .Texas A&M University, College Station, TX, USA. Doctoral Dissertation.Google ScholarGoogle Scholar
  37. Michael J. Muller and Sarah Kuhn. 1993. Taxonomy Of PD Practices: A Brief Practitioner's Guide. Commun. ACM, Vol. 36, 6 (1993), 24--28. https://doi.org/10.1145/153571.255960Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Michael Nebeling, Stefania Leone, and Moira C. Norrie. 2012. Crowdsourced Web Engineering and Design. In Web Engineering, Marco Brambilla, Takehiro Tokuda, and Robert Tolksdorf (Eds.). Vol. 7387. Springer, Berlin, Germany, 31--45. https://doi.org/10.1007/978--3--642--31753--8_3Google ScholarGoogle Scholar
  39. Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. 2013a. CrowdAdapt: Enabling Crowdsourced Web Page Adaptation for Individual Viewing Conditions and Preferences. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 23--32. https://doi.org/10.1145/2494603.2480304Google ScholarGoogle Scholar
  40. Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. 2013b. CrowdStudy: General Toolkit for Crowdsourced Evaluation of Web Interfaces. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 255--264. https://doi.org/10.1145/2494603.2480303Google ScholarGoogle Scholar
  41. Jakob Nielsen. 1993. Usability Engineering. Academic Press, Cambridge, MA, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Jakob Nielsen. 2000. Why You Only Need to Test with 5 Users. Jakob Nielsen's Alertbox. http://www.useit.com/alertbox/20000319.htmlGoogle ScholarGoogle Scholar
  43. Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020. Creativity on Paid Crowdsourcing Platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376677Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Judith Alice Redi, Tobias Hoßfeld, Pavel Korshunov, Filippo Mazza, Isabel Povoa, and Christian Keimel. 2013. Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal. In Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia (CrowdMM '13). ACM, New York, NY, USA, 29--34. https://doi.org/10.1145/2506364.2506368Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Christian Rohrer and John Boyd. 2004. The Rise of Intrusive Online Advertising and the Response of User Experience Research at Yahoo!. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). ACM, New York, NY, USA, 1085--1086. https://doi.org/10.1145/985921.985992Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the New Landscapes of Design. CoDesign, Vol. 4, 1 (2008), 5--18. https://doi.org/10.1080/15710880701875068Google ScholarGoogle ScholarCross RefCross Ref
  47. Ugo Braga Sangiorgi, Franccois Beuvens, and Jean Vanderdonckt. 2012. User Interface Design by Collaborative Sketching. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 378--387. https://doi.org/10.1145/2317956.2318013Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Martin Schmettow. 2012. Sample Size in Usability Studies. Commun. ACM, Vol. 55, 4 (April 2012), 64--70. https://doi.org/10.1145/2133806.2133824Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Ben Shneiderman. 2000. Universal Usability. Commun. ACM, Vol. 43, 5 (May 2000), 84--91. https://doi.org/10.1145/332833.332843Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Ben Shneiderman. 2009. Creativity Support Tools: A Grand Challenge for HCI Researchers. In Engineering the User Interface: From Research to Practice, Miguel Redondo, Crescencio Bravo, and Manuel Ortega (Eds.). Springer, London, UK, 1--9. https://doi.org/10.1007/978--1--84800--136--7_1Google ScholarGoogle Scholar
  51. Ben Shneiderman, Gerhard Fischer, Mary Czerwinski, Mitch Resnick, Brad Myers, Linda Candy, Ernest Edmonds, Mike Eisenberg, Elisa Giaccardi, Tom Hewett, Pamela Jennings, Bill Kules, Kumiyo Nakakoji, Jay Nunamaker, Randy Pausch, Ted Selker, Elisabeth Sylvan, and Michael Terry. 2006. Creativity Support Tools: Report From a U.S. National Science Foundation Sponsored Workshop. International Journal of Human-Computer Interaction, Vol. 20, 2 (2006), 61--77. https://doi.org/10.1207/s15327590ijhc2002_1Google ScholarGoogle Scholar
  52. Stefan Thomke and Eric von Hippel. 2002. Customers as Innovators: A New Way to Create Value. Harvard Business Review, Vol. 80, 4 (2002), 51--61.Google ScholarGoogle Scholar
  53. Eric von Hippel. 2005. Democratizing Innovation .MIT Press, Cambridge, MA, USA.Google ScholarGoogle Scholar
  54. Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference. J. Amer. Statist. Assoc., Vol. 22, 158 (1927), 209--212. https://doi.org/10.1080/01621459.1927.10502953Google ScholarGoogle Scholar
  55. Jane Wood and Denise Silver. 1989. Joint Application Design .Wiley, New York, NY, USA. lc89032006Google ScholarGoogle Scholar
  56. Anbang Xu, Shih-Wen Huang, and Brian Bailey. 2014. Voyant: Generating Structured Feedback on Visual Designs Using a Crowd of Non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1433--1444. https://doi.org/10.1145/2531602.2531604Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Anbang Xu, Huaming Rao, Steven P. Dow, and Brian P. Bailey. 2015. A Classroom Study of Using Crowd Feedback in the Iterative Design Process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1637--1648. https://doi.org/10.1145/2675133.2675140Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Lixiu Yu, Aniket Kittur, and Robert E. Kraut. 2016. Encouraging “Outside-The-Box” Thinking in Crowd Innovation Through Identifying Domains of Expertise. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1214--1222. https://doi.org/10.1145/2818048.2820025Google ScholarGoogle Scholar
  59. Lixiu Yu and Jeffrey V. Nickerson. 2011. Cooks or Cobblers?: Crowd Creativity Through Combination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1393--1402. https://doi.org/10.1145/1978942.1979147Google ScholarGoogle Scholar

Index Terms

  1. CrowdUI: Supporting Web Design with the Crowd

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!