skip to main content
research-article
Public Access

Novice and Expert Sensemaking of Crowdsourced Design Feedback

Published:06 December 2017Publication History
Skip Abstract Section

Abstract

Online feedback exchange (OFE) systems are an increasingly popular way to test concepts with millions of target users before going to market. Yet, we know little about how designers make sense of this abundant feedback. This empirical study investigates how expert and novice designers make sense of feedback in OFE systems. We observed that when feedback conflicted with frames originating from the participant's design knowledge, experts were more likely than novices to question the inconsistency, seeking critical information to expand their understanding of the design goals. Our results suggest that in order for OFE systems to be truly effective, they must be able to support nuances in sensemaking activities of novice and expert users.

References

  1. {n. d.}. BetaFamily. ({n. d.}). Retrieved July 7, 2017 from www.betafamily.comGoogle ScholarGoogle Scholar
  2. {n. d.}. Dribbble. ({n. d.}). Retrieved July 7, 2017 from www.dribbble.comGoogle ScholarGoogle Scholar
  3. {n. d.}. MURAL. ({n. d.}). Retrieved July 7, 2017 from www.mural.lyGoogle ScholarGoogle Scholar
  4. {n. d.}. UserTesting. ({n. d.}). Retrieved July 7, 2017 from www.usertesting.comGoogle ScholarGoogle Scholar
  5. Cynthia J Atman, Justin R Chimka, Karen M Bursic, and Heather L Nachtmann. 1999. A comparison of freshman and senior engineering design processes. Design Studies 20, 2 (1999), 131--152.Google ScholarGoogle ScholarCross RefCross Ref
  6. Sara L Beckman and Michael Barry. 2007. Innovation as a learning process: Embedding design thinking. California management review 50, 1 (Oct. 2007), 25--56.Google ScholarGoogle Scholar
  7. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213--220.Google ScholarGoogle Scholar
  8. Nigel Cross. 2004. Expertise in design: An overview. Design Studies 25, 5 (2004), 427--441.Google ScholarGoogle ScholarCross RefCross Ref
  9. Deanna P Dannels and Kelly Norris Martin. 2008. Critiquing critiques: A genre analysis of feedback across novice to expert design studios. Journal of Business and Technical Communication 22, 2 (April 2008), 135--159.Google ScholarGoogle Scholar
  10. Eureka Foong, Steven P Dow, Brian P Bailey, and Elizabeth M Gerber. 2017. Online feedback exchange: A framework for understanding the socio-psychological factors. In Proceedings of the ACM Conference on Human Factors in Computing Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. In the 2015 ACM SIGCHI Conference on Creativity and Cognition. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Jonathan Grudin. 1988. Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work & Social Computing. 85--93. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. 2016. Framing feedback: Choosing review environment features that support high quality peer assessment. In the 2016 ACM Conference on Human Factors in Computing Systems. 458--469. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Julie Hui, Amos Glenn, Rachel Jue, Elizabeth Gerber, and Steven Dow. 2015. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing.Google ScholarGoogle ScholarCross RefCross Ref
  15. Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP '10. ACM Press, New York, New York, USA, 64. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the 26th Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, New York, USA, 453--456. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Gary Klein, B Moon, and R R Ho man. 2006. Making sense of sensemaking 2: A macrocognitive model. IEEE intelligent systems 21, 5 (2006), 88--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Gary Klein, Jennifer K Phillips, Erica L Rall, and Deborah Peluso. 2007. A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.Google ScholarGoogle Scholar
  19. Gary Klein, W Seick, D Peluso, and J Smith. 2007. FOCUS: A model of sensemaking. Technical Report.Google ScholarGoogle Scholar
  20. Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique style guide: Improving crowdsourced design feedback with a natural Language Model. In Proceedings of the ACM Conference on Human Factors in Computing Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Bryan Lawson. 2005. How Designers Think: The Design Process Demysti ed. Elsevier, Oxford, England.Google ScholarGoogle Scholar
  22. Kurt Luther, Jari-Lee Tolentino,Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In the 18th ACM Conference. ACM Press, New York, New York, USA, 473--485. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. John Neuhart, Charles Eames, Ray Eames, and Marilyn Neuhart. 1989. Eames Design: The Work of the Office of Charles and Ray Eames. Harry N. Abrams.Google ScholarGoogle Scholar
  24. Duyen T Nguyen, Thomas R Garncarz, Felicia Ng, Laura Dabbish, and Steven Dow. 2016. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.Google ScholarGoogle Scholar
  25. Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Victor Papanek and R Buckminster Fuller. 1972. Design for the Real World. Thames and Hudson, London.Google ScholarGoogle Scholar
  27. Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on.Google ScholarGoogle Scholar
  28. Daniel M Russell, Mark J Stefik, Peter Pirolli, and Stuart K Card. 1993. The cost structure of sensemaking. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems - CHI '93. ACM Press, New York, New York, USA, 269--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Herbert A Simon. 1988. The science of design: Creating the artificial. Design Issues 4, 1/2 (1988), 67.Google ScholarGoogle ScholarCross RefCross Ref
  30. James P Spradley. 1980. Participant Observation. Wadsworth Publishing Company.Google ScholarGoogle Scholar
  31. Jennifer Thom-Santelli, Dan Cosley, and Geri Gay. 2010. What do you know? Experts, novices and territoriality in collaborative systems. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 1685--1694. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Karl E Weick, Kathleen M Sutcliffe, and David Obstfeld. 2005. Organizing and the Process of Sensemaking. Organization Science 16, 4 (Aug. 2005), 409--421. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Anbang Xu and Brian P Bailey. 2012. What do you think?: A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the 15th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, New York, New York, USA, 295--304. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Anbang Xu, Shih-Wen Huang, and Brian P Bailey. 2014. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing. 1433--1444. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Anbang Xu, Huaming Rao, Steven P Dow, and Brian P Bailey. 2015. A classroom study of using crowd feedback in the iterative design process. In the 18th ACM Conference. ACM Press, New York, New York, USA, 1637--1648. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Yu-Chun Grace Yen, Steven Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social network, web forum, or task market? Comparing different crowd genres for design feedback exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM Press, 773--784. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Alvin Yuan, Kurt Luther, Markus Krause, Sophie Vennix, Steven P Dow, and Björn Hartmann. 2016. Almost an expert: The effects of rubrics and expertise on the perceived value of crowdsourced design critique. In the 19th ACM Conference. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Novice and Expert Sensemaking of Crowdsourced Design Feedback

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!