Abstract

Web design is a complex and challenging task. It involves making many design decisions that materialise preconceived notions of user needs that may or may not be true. In this paper, we investigate supporting the co-design of a website with visual feedback elicited from the website's community of users. Website users can express their needs by re-arranging and modifying the website's layout and design. To explore and validate this idea, we present CrowdUI, a web-based tool that enables members of the community of a website to visually express their design improvement ideas, frustrations and needs, and to send this feedback to the person in charge of designing or maintaining the website.
CrowdUI is validated in a study with 45 users of a popular social media and networking website. Second, our qualitative evaluation with 60 experienced web developers shows that CrowdUI is able to elicit diverse and meaningful feedback. Put together, our results suggest that CrowdUI's approach constitutes a productive setting for eliciting visual feedback from the user community as a complement to traditional ways of eliciting feedback and participatory design. Finally, based on our experiences, we discuss a design space for crowdsourced web design and provide design recommendations for similar future tools.
- Chadia Abras, Diane Maloney-Krichmar, and Jenny Preece. 2004. User-Centered Design. In Berkshire Encyclopedia of Human-Computer Interaction 2nd ed.), William S. Bainbridge (Ed.). Berkshire Publishing Company, Ltd., Great Barrington, MA, USA, 763--768.Google Scholar
- Pierre Akiki, Arosha Bandara, and Yijun Yu. 2013. Crowdsourcing User Interface Adaptations for Minimizing the Bloat in Enterprise Applications. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 121--126. https://doi.org/10.1145/2494603.2480319Google Scholar
Digital Library
- Salvatore Andolina, Hendrik Schneider, Joel Chan, Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017. Crowdboard: Augmenting In-Person Idea Generation with Real-Time Crowds. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17). ACM, New York, NY, USA, 106--118. https://doi.org/10.1145/3059454.3059477Google Scholar
Digital Library
- Goetz Botterweck. 2011. Multi Front-End Engineering. In Model-Driven Development of Advanced User Interfaces, Heinrich Hussmann, Gerrit Meixner, and Detlef Zuehlke (Eds.). Springer, Berlin/Heidelberg, Germany, 27--42. https://doi.org/10.1007/978--3--642--14562--9_2Google Scholar
- Margaret M. Bradley and Peter J. Lang. 2007. The International Affective Picture System (IAPS) in the Study of Emotion and Attention. In Series in Affective Science. Handbook of Emotion Elicitation and Assessment, James A. Coan and John J. B. Allen (Eds.). Oxford University Press, New York, NY, USA, 29--46.Google Scholar
- Marco Brambilla, Stefano Ceri, Andrea Mauri, and Riccardo Volonterio. 2014. Community-based Crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web (WWW '14 Companion). ACM, New York, NY, USA, 891--896. https://doi.org/10.1145/2567948.2578835Google Scholar
- Virginia Braun, Victoria Clarke, Nikki Hayfield, and Gareth Terry. 2019. Thematic Analysis. In Handbook of Research Methods in Health Social Sciences, Pranee Liamputtong (Ed.). Springer, Singapore, 843--860. https://doi.org/10.1007/978--981--10--5251--4_103Google Scholar
- Andrea Bunt, Cristina Conati, and Joanna McGrenere. 2007. Supporting Interface Customization Using a Mixed-initiative Approach. In Proceedings of the 12th International Conference on Intelligent User Interfaces (IUI '07). ACM, New York, NY, USA, 92--101. https://doi.org/10.1145/1216295.1216317Google Scholar
Digital Library
- Javier Caminero, Mari Carmen Rodríguez-Gancedo, Jean Vanderdonckt, Fabio Paternò, Joerg Rett, Dave Raggett, Jean-Loup Comeliau, and Ignacio Marín. 2012. Multidimensional Context-Aware Adaptation of Service Front-Ends. In Context-Aware Adaptation of Service Front-Ends 2012. Proceedings of the Workshop on Context-Aware Adaptation of Service Front-Ends (CASFE 2012), Francisco Javier Caminero Gil, Fabio Paternò, and Jean Vanderdonckt (Eds.). CEUR-WS, 5 pages.Google Scholar
- Biplab Deka, Zifeng Huang, Chad Franzen, Jeffrey Nichols, Yang Li, and Ranjitha Kumar. 2017. ZIPT: Zero-Integration Performance Testing of Mobile App Designs. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 727--736. https://doi.org/10.1145/3126594.3126647Google Scholar
Digital Library
- Stephen W. Draper and Don A. Norman. 1986. User Centered System Design .Lawrence Erlbaum Associates, Hillsdale, NJ, USA. 85025207Google Scholar
- Facebook, Inc. 2019. Facebook Reports Second Quarter 2019 Results. Press release, 24 July, 2019.Google Scholar
- Eureka Foong, Steven P. Dow, Brian P. Bailey, and Elizabeth M. Gerber. 2017a. Online Feedback Exchange: A Framework for Understanding the Socio-Psychological Factors. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4454--4467. https://doi.org/10.1145/3025453.3025791Google Scholar
- Eureka Foong, Darren Gergle, and Elizabeth M. Gerber. 2017b. Novice and Expert Sensemaking of Crowdsourced Design Feedback. Proceedings of the ACM on Human-Computer Interaction, Vol. 1, CSCW (Dec. 2017), 45:1--45:18. https://doi.org/10.1145/3134680Google Scholar
- Vivian Genaro Motti, Dave Raggett, Sascha Van Cauwelaert, and Jean Vanderdonckt. 2013. Simplifying the Development of Cross-Platform Web User Interfaces by Collaborative Model-Based Design. In Proceedings of the 31st ACM International Conference on Design of Communication (SIGDOC '13). ACM, New York, NY, USA, 55--64. https://doi.org/10.1145/2507065.2507067Google Scholar
- Michael D. Greenberg, Matthew W. Easterday, and Elizabeth M. Gerber. 2015. Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (C&C '15). ACM, New York, NY, USA, 235--244. https://doi.org/10.1145/2757226.2757249Google Scholar
- Eszter Hargittai and Gina Walejko. 2008. The Participation Divide: Content Creation and Sharing in the Digital Age. Information, Communication & Society, Vol. 11, 2 (2008), 239--256. https://doi.org/10.1080/13691180801946150Google Scholar
Cross Ref
- Marc Hassenzahl. 2004. The Thing and I: Understanding the Relationship Between User and Product. In Funology, Mark A. Blythe, Kees Overbeeke, Andrew F. Monk, and Peter C. Wright (Eds.). Kluwer Academic Publishers, Norwell, MA, USA, 31--42. https://doi.org/10.1007/1--4020--2967--5_4Google Scholar
- Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 203--212. https://doi.org/10.1145/1753326.1753357Google Scholar
Digital Library
- Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh, and Björn Hartmann. 2012. CommunitySourcing: Engaging Local Crowds to Perform Expert Work via Physical Kiosks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1539--1548. https://doi.org/10.1145/2207676.2208619Google Scholar
Digital Library
- Hyeonsu B. Kang, Gabriel Amoako, Neil Sengupta, and Steven P. Dow. 2018. Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, 606:1--606:13. https://doi.org/10.1145/3173574.3174180Google Scholar
- Louise H. Kidder and Michelle Fine. 1987. Qualitative and quantitative methods: When stories converge. New Directions for Program Evaluation, Vol. 1987, 35 (1987), 57--75. https://doi.org/10.1002/ev.1459Google Scholar
- Nigel King, Joanna Brooks, and Saloomeh Tabari. 2018. Template Analysis in Business and Management Research. In Qualitative Methodologies in Organization Studies: Volume II: Methods and Possibilities, Malgorzata Ciesielska and Dariusz Jemielniak (Eds.). Springer, Cham, Switzerland, 179--206. https://doi.org/10.1007/978--3--319--65442--3_8Google Scholar
- Yuki Koyama. 2016. Computational Design Driven by Aesthetic Preference. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct). ACM, New York, NY, USA, 1--4. https://doi.org/10.1145/2984751.2984781Google Scholar
Digital Library
- Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3083--3092. https://doi.org/10.1145/2470654.2466420Google Scholar
Digital Library
- Benjamin Lafreniere, Andrea Bunt, Matthew Lount, Filip Krynicki, and Michael A. Terry. 2011. AdaptableGIMP: Designing a Socially-adaptable Interface. In Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology (UIST '11 Adjunct). ACM, New York, NY, USA, 89--90. https://doi.org/10.1145/2046396.2046437Google Scholar
- Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. 2015. Apparition: Crowdsourced User Interfaces That Come to Life As You Sketch Them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1925--1934. https://doi.org/10.1145/2702123.2702565Google Scholar
- Talia Lavie and Noam Tractinsky. 2004. Assessing Dimensions of Perceived Visual Aesthetics of Web Sites. International Journal of Human-Computer Studies, Vol. 60, 3 (March 2004), 269--298. https://doi.org/10.1016/j.ijhcs.2003.09.002Google Scholar
Digital Library
- Sang Won Lee, Rebecca Krosnick, Sun Young Park, Brandon Keelean, Sach Vaidya, Stephanie D. O'Keefe, and Walter S. Lasecki. 2018. Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (Nov. 2018), 104:1--104:23. https://doi.org/10.1145/3274373Google Scholar
- Sang Won Lee, Yujin Zhang, Isabelle Wong, Yiwei Yang, Stephanie D. O'Keefe, and Walter S. Lasecki. 2017. SketchExpress: Remixing Animations for More Effective Crowd-Powered Prototyping of Interactive Interfaces. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 817--828. https://doi.org/10.1145/3126594.3126595Google Scholar
- Friedrich Leisch. 2008. Visualizing Cluster Analysis and Finite Mixture Models. In Handbook of Data Visualization, Chun-houh Chen, Wolfgang Karl Hardle, and Antony Unwin (Eds.). Springer, Berlin/Heidelberg, Germany, 561--587.Google Scholar
- Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P. Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P. Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 473--485. https://doi.org/10.1145/2675133.2675283Google Scholar
- Paul A. Mabe and Stephen West. 1982. Validity of Self-evaluation of Ability: A Review and Meta-analysis. Journal of Applied Psychology, Vol. 67, 3 (6 1982), 280--296. https://doi.org/10.1037/0021--9010.67.3.280Google Scholar
Cross Ref
- Wendy E. Mackay. 2004. The Interactive Thread: Exploring Methods for Multi-disciplinary Design. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS '04). ACM, New York, NY, USA, 103--112. https://doi.org/10.1145/1013115.1013131Google Scholar
Digital Library
- Joanna McGrenere and Gale Moore. 2000. Are We All in the Same "Bloat"?. In Proceedings of the Graphics Interface 2000 Conference. 187--196.Google Scholar
- John Michael Moore. 2007. Design Exploration: Engaging A Larger User Population .Texas A&M University, College Station, TX, USA. Doctoral Dissertation.Google Scholar
- Michael J. Muller and Sarah Kuhn. 1993. Taxonomy Of PD Practices: A Brief Practitioner's Guide. Commun. ACM, Vol. 36, 6 (1993), 24--28. https://doi.org/10.1145/153571.255960Google Scholar
Digital Library
- Michael Nebeling, Stefania Leone, and Moira C. Norrie. 2012. Crowdsourced Web Engineering and Design. In Web Engineering, Marco Brambilla, Takehiro Tokuda, and Robert Tolksdorf (Eds.). Vol. 7387. Springer, Berlin, Germany, 31--45. https://doi.org/10.1007/978--3--642--31753--8_3Google Scholar
- Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. 2013a. CrowdAdapt: Enabling Crowdsourced Web Page Adaptation for Individual Viewing Conditions and Preferences. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 23--32. https://doi.org/10.1145/2494603.2480304Google Scholar
- Michael Nebeling, Maximilian Speicher, and Moira C. Norrie. 2013b. CrowdStudy: General Toolkit for Crowdsourced Evaluation of Web Interfaces. In Proceedings of the 5th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '13). ACM, New York, NY, USA, 255--264. https://doi.org/10.1145/2494603.2480303Google Scholar
- Jakob Nielsen. 1993. Usability Engineering. Academic Press, Cambridge, MA, USA.Google Scholar
Digital Library
- Jakob Nielsen. 2000. Why You Only Need to Test with 5 Users. Jakob Nielsen's Alertbox. http://www.useit.com/alertbox/20000319.htmlGoogle Scholar
- Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020. Creativity on Paid Crowdsourcing Platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). ACM, New York, NY, USA, 1--14. https://doi.org/10.1145/3313831.3376677Google Scholar
Digital Library
- Judith Alice Redi, Tobias Hoßfeld, Pavel Korshunov, Filippo Mazza, Isabel Povoa, and Christian Keimel. 2013. Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal. In Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia (CrowdMM '13). ACM, New York, NY, USA, 29--34. https://doi.org/10.1145/2506364.2506368Google Scholar
Digital Library
- Christian Rohrer and John Boyd. 2004. The Rise of Intrusive Online Advertising and the Response of User Experience Research at Yahoo!. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). ACM, New York, NY, USA, 1085--1086. https://doi.org/10.1145/985921.985992Google Scholar
Digital Library
- Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the New Landscapes of Design. CoDesign, Vol. 4, 1 (2008), 5--18. https://doi.org/10.1080/15710880701875068Google Scholar
Cross Ref
- Ugo Braga Sangiorgi, Franccois Beuvens, and Jean Vanderdonckt. 2012. User Interface Design by Collaborative Sketching. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 378--387. https://doi.org/10.1145/2317956.2318013Google Scholar
Digital Library
- Martin Schmettow. 2012. Sample Size in Usability Studies. Commun. ACM, Vol. 55, 4 (April 2012), 64--70. https://doi.org/10.1145/2133806.2133824Google Scholar
Digital Library
- Ben Shneiderman. 2000. Universal Usability. Commun. ACM, Vol. 43, 5 (May 2000), 84--91. https://doi.org/10.1145/332833.332843Google Scholar
Digital Library
- Ben Shneiderman. 2009. Creativity Support Tools: A Grand Challenge for HCI Researchers. In Engineering the User Interface: From Research to Practice, Miguel Redondo, Crescencio Bravo, and Manuel Ortega (Eds.). Springer, London, UK, 1--9. https://doi.org/10.1007/978--1--84800--136--7_1Google Scholar
- Ben Shneiderman, Gerhard Fischer, Mary Czerwinski, Mitch Resnick, Brad Myers, Linda Candy, Ernest Edmonds, Mike Eisenberg, Elisa Giaccardi, Tom Hewett, Pamela Jennings, Bill Kules, Kumiyo Nakakoji, Jay Nunamaker, Randy Pausch, Ted Selker, Elisabeth Sylvan, and Michael Terry. 2006. Creativity Support Tools: Report From a U.S. National Science Foundation Sponsored Workshop. International Journal of Human-Computer Interaction, Vol. 20, 2 (2006), 61--77. https://doi.org/10.1207/s15327590ijhc2002_1Google Scholar
- Stefan Thomke and Eric von Hippel. 2002. Customers as Innovators: A New Way to Create Value. Harvard Business Review, Vol. 80, 4 (2002), 51--61.Google Scholar
- Eric von Hippel. 2005. Democratizing Innovation .MIT Press, Cambridge, MA, USA.Google Scholar
- Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference. J. Amer. Statist. Assoc., Vol. 22, 158 (1927), 209--212. https://doi.org/10.1080/01621459.1927.10502953Google Scholar
- Jane Wood and Denise Silver. 1989. Joint Application Design .Wiley, New York, NY, USA. lc89032006Google Scholar
- Anbang Xu, Shih-Wen Huang, and Brian Bailey. 2014. Voyant: Generating Structured Feedback on Visual Designs Using a Crowd of Non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1433--1444. https://doi.org/10.1145/2531602.2531604Google Scholar
Digital Library
- Anbang Xu, Huaming Rao, Steven P. Dow, and Brian P. Bailey. 2015. A Classroom Study of Using Crowd Feedback in the Iterative Design Process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1637--1648. https://doi.org/10.1145/2675133.2675140Google Scholar
Digital Library
- Lixiu Yu, Aniket Kittur, and Robert E. Kraut. 2016. Encouraging “Outside-The-Box” Thinking in Crowd Innovation Through Identifying Domains of Expertise. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1214--1222. https://doi.org/10.1145/2818048.2820025Google Scholar
- Lixiu Yu and Jeffrey V. Nickerson. 2011. Cooks or Cobblers?: Crowd Creativity Through Combination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1393--1402. https://doi.org/10.1145/1978942.1979147Google Scholar
Index Terms
CrowdUI: Supporting Web Design with the Crowd
Recommendations
Crowd-powered Interfaces for Creative Design Thinking
C&C '19: Proceedings of the 2019 Conference on Creativity and CognitionCrowdsourcing is a powerful approach for tapping into the collective insights of diverse crowds. Thus, crowdsourcing has potential to support designers in making sense of a design space. In this hands-on workshop, we will brainstorm and conceptualise ...
Engaging the Crowd: Lessons for Outreach and Tool Design From a Creative Online Participatory Study
In this article, the challenges of realising e-participatory projects in urban planning are described. A participatory case study in Singapore serves as the basis for their presented conclusions. The researchers used a map-based e-participation tool ...
Leveraging the Crowd to Support the Conversation Design Process
CUI '20: Proceedings of the 2nd Conference on Conversational User InterfacesBuilding a chatbot with human-like conversation capabilities is essential for users to feel more natural in task completion. Many designers try to collect human conversation data and apply them into a chatbot conversation, aiming that it could work like ...






Comments