Abstract
Online feedback exchange (OFE) systems are an increasingly popular way to test concepts with millions of target users before going to market. Yet, we know little about how designers make sense of this abundant feedback. This empirical study investigates how expert and novice designers make sense of feedback in OFE systems. We observed that when feedback conflicted with frames originating from the participant's design knowledge, experts were more likely than novices to question the inconsistency, seeking critical information to expand their understanding of the design goals. Our results suggest that in order for OFE systems to be truly effective, they must be able to support nuances in sensemaking activities of novice and expert users.
- {n. d.}. BetaFamily. ({n. d.}). Retrieved July 7, 2017 from www.betafamily.comGoogle Scholar
- {n. d.}. Dribbble. ({n. d.}). Retrieved July 7, 2017 from www.dribbble.comGoogle Scholar
- {n. d.}. MURAL. ({n. d.}). Retrieved July 7, 2017 from www.mural.lyGoogle Scholar
- {n. d.}. UserTesting. ({n. d.}). Retrieved July 7, 2017 from www.usertesting.comGoogle Scholar
- Cynthia J Atman, Justin R Chimka, Karen M Bursic, and Heather L Nachtmann. 1999. A comparison of freshman and senior engineering design processes. Design Studies 20, 2 (1999), 131--152.Google Scholar
Cross Ref
- Sara L Beckman and Michael Barry. 2007. Innovation as a learning process: Embedding design thinking. California management review 50, 1 (Oct. 2007), 25--56.Google Scholar
- Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213--220.Google Scholar
- Nigel Cross. 2004. Expertise in design: An overview. Design Studies 25, 5 (2004), 427--441.Google Scholar
Cross Ref
- Deanna P Dannels and Kelly Norris Martin. 2008. Critiquing critiques: A genre analysis of feedback across novice to expert design studios. Journal of Business and Technical Communication 22, 2 (April 2008), 135--159.Google Scholar
- Eureka Foong, Steven P Dow, Brian P Bailey, and Elizabeth M Gerber. 2017. Online feedback exchange: A framework for understanding the socio-psychological factors. In Proceedings of the ACM Conference on Human Factors in Computing Systems. Google Scholar
Digital Library
- Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. In the 2015 ACM SIGCHI Conference on Creativity and Cognition. Google Scholar
Digital Library
- Jonathan Grudin. 1988. Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. In Proceedings of the ACM Conference on Computer-Supported Cooperative Work & Social Computing. 85--93. Google Scholar
Digital Library
- Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. 2016. Framing feedback: Choosing review environment features that support high quality peer assessment. In the 2016 ACM Conference on Human Factors in Computing Systems. 458--469. Google Scholar
Digital Library
- Julie Hui, Amos Glenn, Rachel Jue, Elizabeth Gerber, and Steven Dow. 2015. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
Cross Ref
- Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP '10. ACM Press, New York, New York, USA, 64. Google Scholar
Digital Library
- Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the 26th Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, New York, USA, 453--456. Google Scholar
Digital Library
- Gary Klein, B Moon, and R R Ho man. 2006. Making sense of sensemaking 2: A macrocognitive model. IEEE intelligent systems 21, 5 (2006), 88--92. Google Scholar
Digital Library
- Gary Klein, Jennifer K Phillips, Erica L Rall, and Deborah Peluso. 2007. A data-frame theory of sensemaking. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.Google Scholar
- Gary Klein, W Seick, D Peluso, and J Smith. 2007. FOCUS: A model of sensemaking. Technical Report.Google Scholar
- Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique style guide: Improving crowdsourced design feedback with a natural Language Model. In Proceedings of the ACM Conference on Human Factors in Computing Systems. Google Scholar
Digital Library
- Bryan Lawson. 2005. How Designers Think: The Design Process Demysti ed. Elsevier, Oxford, England.Google Scholar
- Kurt Luther, Jari-Lee Tolentino,Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In the 18th ACM Conference. ACM Press, New York, New York, USA, 473--485. Google Scholar
Digital Library
- John Neuhart, Charles Eames, Ray Eames, and Marilyn Neuhart. 1989. Eames Design: The Work of the Office of Charles and Ray Eames. Harry N. Abrams.Google Scholar
- Duyen T Nguyen, Thomas R Garncarz, Felicia Ng, Laura Dabbish, and Steven Dow. 2016. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making.Google Scholar
- Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books.Google Scholar
Digital Library
- Victor Papanek and R Buckminster Fuller. 1972. Design for the Real World. Thames and Hudson, London.Google Scholar
- Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on.Google Scholar
- Daniel M Russell, Mark J Stefik, Peter Pirolli, and Stuart K Card. 1993. The cost structure of sensemaking. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems - CHI '93. ACM Press, New York, New York, USA, 269--276. Google Scholar
Digital Library
- Herbert A Simon. 1988. The science of design: Creating the artificial. Design Issues 4, 1/2 (1988), 67.Google Scholar
Cross Ref
- James P Spradley. 1980. Participant Observation. Wadsworth Publishing Company.Google Scholar
- Jennifer Thom-Santelli, Dan Cosley, and Geri Gay. 2010. What do you know? Experts, novices and territoriality in collaborative systems. In Proceedings of the ACM Conference on Human Factors in Computing Systems. 1685--1694. Google Scholar
Digital Library
- Karl E Weick, Kathleen M Sutcliffe, and David Obstfeld. 2005. Organizing and the Process of Sensemaking. Organization Science 16, 4 (Aug. 2005), 409--421. Google Scholar
Digital Library
- Anbang Xu and Brian P Bailey. 2012. What do you think?: A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the 15th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, New York, New York, USA, 295--304. Google Scholar
Digital Library
- Anbang Xu, Shih-Wen Huang, and Brian P Bailey. 2014. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing. 1433--1444. Google Scholar
Digital Library
- Anbang Xu, Huaming Rao, Steven P Dow, and Brian P Bailey. 2015. A classroom study of using crowd feedback in the iterative design process. In the 18th ACM Conference. ACM Press, New York, New York, USA, 1637--1648. Google Scholar
Digital Library
- Yu-Chun Grace Yen, Steven Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social network, web forum, or task market? Comparing different crowd genres for design feedback exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM Press, 773--784. Google Scholar
Digital Library
- Alvin Yuan, Kurt Luther, Markus Krause, Sophie Vennix, Steven P Dow, and Björn Hartmann. 2016. Almost an expert: The effects of rubrics and expertise on the perceived value of crowdsourced design critique. In the 19th ACM Conference. Google Scholar
Digital Library
Index Terms
Novice and Expert Sensemaking of Crowdsourced Design Feedback
Recommendations
Online Feedback Exchange: A Framework for Understanding the Socio-Psychological Factors
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsTo meet the demand for authentic, timely, and affordable feedback, researchers have explored technologies to connect designers with feedback providers online. While researchers have implemented mechanisms to improve the content of feedback, most systems ...
Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques
CSCW '16: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social ComputingExpert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide ...
Knowing That You Know What I Know Helps?: Understanding the Effects of Knowledge Transparency in Online Knowledge Transfer
To accomplish collaborative work, collaborators need to know one another's states of work for coordination. Similarly, in situations of knowledge transfer where experts need to instruct novices by passing implicit knowledge and personal experience about ...






Comments