skip to main content
research-article

Score-Group Framing Negatively Impacts Peer Evaluations

Authors Info & Claims
Published:01 November 2018Publication History
Skip Abstract Section

Abstract

How does group membership framing affect the feedback students provide learners? This paper presents two between-subjects experiments investigating the effect of Ingroup/Outgroup membership on effort spent in peer evaluations, and whether the group membership criterion type affects quality and stringency of evaluation. Two peer-review assignments were implemented in two separate classes. In the first study, students were nominally grouped by location they sat in class and non-nominally grouped by current class score; each was asked to review an Ingroup and Outgroup peer assignment. A second study randomly assigned students to one of four group types (random, score, motivation, and location); student reviewed two Ingroup assignments. In both studies, score-grouped students graded their peers more stringently than students grouped by location. These studies illustrate for system designers the impacts of group framing - and the disclosure of that-in peer review tasks.

References

  1. Daniel Balliet, Junhui Wu, and Carsten K. W. De Dreu. 2014. Ingroup favoritism in cooperation: A meta-analysis. Psychol. Bull. 140, 6 (2014), 1556--1581.Google ScholarGoogle ScholarCross RefCross Ref
  2. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. J. Stat. Softw. Vol 1 Issue 1 2015 (2015).Google ScholarGoogle Scholar
  3. Erin Bradner and Gloria Mark. 2002. Why Distance Matters: Effects on Cooperation, Persuasion and Deception. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (CSCW '02), 226--235. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Marilynn B. Brewer. 1979. In-group bias in the minimal intergroup situation: A cognitive-motivational analysis. Psychol. Bull. 86, 2 (1979), 307--324.Google ScholarGoogle ScholarCross RefCross Ref
  5. Priyanka B. Carr and Gregory M. Walton. 2014. Cues of working together fuel intrinsic motivation. J. Exp. Soc. Psychol. 53, (2014), 169--184.Google ScholarGoogle Scholar
  6. Justin Cheng, Jaime Teevan, and Michael S. Bernstein. 2015. Measuring Crowdsourcing Effort with Error-Time Curves. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1365--1374. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Winnie Cheng and Martin Warren. 1997. Having second thoughts: Student perceptions before and after a peer assessment exercise. Stud. High. Educ. 22, 2 (January 1997), 233--239.Google ScholarGoogle ScholarCross RefCross Ref
  8. Kwangsu Cho and Christian D. Schunn. 2007. Scaffolded Writing and Rewriting in the Discipline: A Web-based Reciprocal Peer Review System. Comput Educ 48, 3 (April 2007), 409--426. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Phil Davies. 2000. Computerized Peer Assessment. Innov. Educ. Train. Int. 37, 4 (January 2000), 346--355.Google ScholarGoogle Scholar
  10. Carol S. Dweck. 2008. Mindset: The New Psychology of Success. Ballantine Books, New York.Google ScholarGoogle Scholar
  11. Naomi Ellemers, S. Alexander Haslam, and Dick de Gilder. 2008. Motivating Individuals and Groups at Work in the 21st Century. In 21st Century Management: A Reference Handbook. SAGE Publications, Inc., Thousand Oaks, II--182--II--192. Retrieved from http://sk.sagepub.com/reference/management/n68.xmlGoogle ScholarGoogle Scholar
  12. Rosta Farzan, Laura A. Dabbish, Robert E. Kraut, and Tom Postmes. 2011. Increasing Commitment to Online Communities by Designing for Social Presence. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (CSCW '11), 321--330. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Dilrukshi Gamage, Mark E. Whiting, Thejan Rajapakshe, Haritha Thilakarathne, Indika Perera, and Shantha Fernando. 2017. Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale ([email protected] '17), 315--318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Andrew Gelman and Jennifer Hill. 2006. Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, Cambridge. Retrieved from https://www.cambridge.org/core/books/data-analysis-using-regression-and-multilevelhierarchical-models/32A29531C7FD730C3A68951A17C9D983Google ScholarGoogle Scholar
  15. Andrew Gelman, Jennifer Hill, and Masanao Yajima. 2012. Why We (Usually) Don't Have to Worry About Multiple Comparisons. J. Res. Educ. Eff. 5, 2 (April 2012), 189--211.Google ScholarGoogle ScholarCross RefCross Ref
  16. Stephanie J. Hanrahan and Geoff Isaacs. 2001. Assessing Self- and Peer-assessment: The students' views. High. Educ. Res. Dev. 20, 1 (May 2001), 53--70.Google ScholarGoogle ScholarCross RefCross Ref
  17. Catherine M. Hicks, Vineet Pandey, C. Ailie Fraser, and Scott Klemmer. 2016. Framing Feedback: Choosing Review Environment Features That Support High Quality Peer Assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 458--469. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Julia H. Kaufman and Christian D. Schunn. 2011. Students' perceptions about peer assessment for writing: their origin and impact on revision work. Instr. Sci. 39, 3 (2011), 387--406.Google ScholarGoogle ScholarCross RefCross Ref
  19. Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale ([email protected] '15), 75--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and Self Assessment in Massive Online Classes. ACM Trans Comput-Hum Interact 20, 6 (December 2013), 33:1--33:31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Kimberly Ling, Gerard Beenen, Pamela Ludford, Xiaoqing Wang, Klarissa Chang, Xin Li, Dan Cosley, Dan Frankowski, Loren Terveen, Al Mamunur Rashid, Paul Resnick, and Robert Kraut. 2005. Using social psychology to motivate contributions to online communities. J. Comput.-Mediat. Commun. 10, 4 (January 2005).Google ScholarGoogle ScholarCross RefCross Ref
  22. Steson Lo and Sally Andrews. 2015. To transform or not to transform: using generalized linear mixed models to analyse reaction time data. Front. Psychol. 6, (2015), 1171.Google ScholarGoogle Scholar
  23. Yanxin Lu, Joe Warren, Christopher Jermaine, Swarat Chaudhuri, and Scott Rixner. 2015. Grading the Graders: Motivating Peer Graders in a MOOC. In Proceedings of the 24th International Conference on World Wide Web (WWW '15), 680--690. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Jennifer Marlow and Laura Dabbish. 2012. Designing Interventions to Reduce Psychological Distance in Globally Distributed Teams. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work Companion (CSCW '12), 163--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. M. M. Nelson and C. D. Schunn. 2009. The nature of feedback: How different types of peer feedback affect writing performance. Instr. Sci. 37, (July 2009), 375--401.Google ScholarGoogle Scholar
  26. Melissa M. Patchan, Brandi Hawk, Christopher A. Stevens, and Christian D. Schunn. 2013. The effects of skill diversity on commenting and revisions. Instr. Sci. 41, 2 (2013), 381--405.Google ScholarGoogle ScholarCross RefCross Ref
  27. Maria Perez-Ortiz and Rafal K. Mantiuk. 2017. A practical guide and software for analysing pairwise comparison experiments. ArXiv171203686 Cs Stat (December 2017). Retrieved from http://arxiv.org/abs/1712.03686Google ScholarGoogle Scholar
  28. Ken Reily, Pam Ludford Finnerty, and Loren Terveen. 2009. Two Peers Are Better Than One: Aggregating Peer Reviews for Computing Assignments is Surprisingly Accurate. In Proceedings of the ACM 2009 International Conference on Supporting Group Work (GROUP '09), 115--124. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Yuqing Ren, F. Maxwell Harper, Sara Drenner, Loren Terveen, Sara Kiesler, John Riedl, and Robert E. Kraut. 2012. Building Member Attachment in Online Communities: Applying Theories of Group Identity and Interpersonal Bonds. MIS Q 36, 3 (September 2012), 841--864. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Todd Rogers and Avi Feller. 2016. Discouraged by Peer Excellence: Exposure to Exemplary Peer Performance Causes Quitting. Psychol. Sci. 27, 3 (March 2016), 365--374.Google ScholarGoogle Scholar
  31. J. Sanchez-Burks, R. E. Nisbett, and O. Ybarra. 2000. Cultural styles, relational schemas, and prejudice against out-groups. J. Pers. Soc. Psychol. 79, 2 (August 2000), 174--189.Google ScholarGoogle ScholarCross RefCross Ref
  32. Stanley Schacter. 1959. The Psychology of Affiliation; Experimental Studies of the Sources of Gregariousness. Stanford University Press, Stanford, California.Google ScholarGoogle Scholar
  33. Natalie Sebanz, Harold Bekkering, and Günther Knoblich. 2006. Joint action: bodies and minds moving together. Trends Cogn. Sci. 10, 2 (February 2006), 70--76.Google ScholarGoogle ScholarCross RefCross Ref
  34. Garriy Shteynberg and Adam D. Galinsky. 2011. Implicit coordination: Sharing goals with similar others intensifies goal pursuit. J. Exp. Soc. Psychol. 47, 6 (November 2011), 1291--1294.Google ScholarGoogle ScholarCross RefCross Ref
  35. Walter W. Stroup. 2013. Generalized linear mixed models: modern concepts, methods and applications. CRC Press, Boca Raton, Fl.Google ScholarGoogle Scholar
  36. Henri Tajfel. 1974. Social identity and intergroup behaviour. Inf. Int. Soc. Sci. Counc. 13, 2 (April 1974), 65--93.Google ScholarGoogle ScholarCross RefCross Ref
  37. Yla R. Tausczik, Laura A. Dabbish, and Robert E. Kraut. 2014. Building Loyalty to Online Communities Through Bond and Identity-based Attachment to Sub-groups. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14), 146--157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. UCLA: Statistical Consulting Group. Introduction to SAS. Retrieved from https://stats.idre.ucla.edu/sas/faq/how-can-i-interpret-log-transformed-variables-in-terms-of-percent-change-in-linear-regression/Google ScholarGoogle Scholar
  39. Siri Vinther, Ole Haagen Nielsen, Jacob Rosenberg, Niels Keiding, and Torben V. Schroeder. 2012. Same review quality in open versus blinded peer review in "Ugeskrift for Læger." Dan. Med. J. 59, 8 (August 2012), A4479.Google ScholarGoogle Scholar
  40. Gregory M. Walton and Shannon T. Brady. The Many Questions of Belonging. In Handbook of Competence and Motivation: Theory and Application (2nd ed.), Andrew J. Elliot, Carol S. Dweck and Yeager (eds.). Guilford Press, New York.Google ScholarGoogle Scholar
  41. Gregory M. Walton and Geoffrey L. Cohen. 2011. Sharing motivation. In Social motivation. Psychology Press, New York, NY, US, 79--101.Google ScholarGoogle Scholar
  42. Gregory M. Walton, Geoffrey L. Cohen, David Cwir, and Steven J. Spencer. 2012. Mere belonging: the power of social connections. J. Pers. Soc. Psychol. 102, 3 (March 2012), 513--532.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Score-Group Framing Negatively Impacts Peer Evaluations

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Article Metrics

          • Downloads (Last 12 months)11
          • Downloads (Last 6 weeks)4

          Other Metrics

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!