skip to main content
research-article
Public Access

Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

Published:15 October 2020Publication History
Skip Abstract Section

Abstract

Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.

References

  1. William RL Anderegg, James W Prall, Jacob Harold, and Stephen H Schneider. 2010. Expert credibility in climate change. Proceedings of the National Academy of Sciences, Vol. 107, 27 (2010), 12107--12109.Google ScholarGoogle ScholarCross RefCross Ref
  2. Lora Aroyo and Chris Welty. 2015. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, Vol. 36, 1 (2015), 15--24.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Mahmoudreza Babaei, Abhijnan Chakraborty, Juhi Kulshrestha, Elissa M Redmiles, Meeyoung Cha, and Krishna P Gummadi. 2019. Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking.. In FAT. 139.Google ScholarGoogle Scholar
  4. Mevan Babakar. 2018. Crowdsourced Factchecking.Google ScholarGoogle Scholar
  5. Betsy Jane Becker. 1994. Combining significance levels. The handbook of research synthesis (1994), 215--230.Google ScholarGoogle Scholar
  6. Joshua Becker, Ethan Porter, and Damon Centola. 2019. The wisdom of partisan crowds. Proceedings of the National Academy of Sciences of the United States of America, Vol. 166, 22 (2019), 10717--10722. https://doi.org/10.1073/pnas.1817195116Google ScholarGoogle ScholarCross RefCross Ref
  7. Brooke Borel. 2015. The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec 2015). https://www.theguardian.com/media/2015/dec/30/problem-with-science-journalism-2015-reality-kevin-foltaGoogle ScholarGoogle Scholar
  8. Alexandre Bovet and Hernán A Makse. 2019. Influence of fake news in Twitter during the 2016 US presidential election. Nature communications, Vol. 10, 1 (2019), 7.Google ScholarGoogle Scholar
  9. Mohamad Adam Bujang and Nurakmal Baharum. 2016. Sample size guideline for correlation analysis. World, Vol. 3, 1 (2016).Google ScholarGoogle Scholar
  10. Cody Buntain and Jennifer Golbeck. 2017. Automatically Identifying Fake News in Popular Twitter Threads. Proceedings - 2nd IEEE International Conference on Smart Cloud, SmartCloud 2017 (2017), 208--215. https://doi.org/10.1109/SmartCloud.2017.40 arxiv: 1705.01613Google ScholarGoogle ScholarCross RefCross Ref
  11. Davide Ceolin. 2019. Conference Presentation: On the Quality of Crowdsourced Information Quality Assessments. https://drive.google.com/a/hackshackers.com/file/d/1AJmFmRqEhdhSIZLwhXT_1bzStXfV-hVf/view?usp=drive_open&usp=embed_facebookGoogle ScholarGoogle Scholar
  12. Steven H Chaffee. 1982. Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter/media: Interpersonal communication in a media world, Vol. 57 (1982), 77.Google ScholarGoogle Scholar
  13. Shelly Chaiken. 1987. The heuristic model of persuasion. In Social influence: the ontario symposium, Vol. 5. Hillsdale, NJ: Lawrence Erlbaum, 3--39.Google ScholarGoogle Scholar
  14. Roy De Maesschalck, Delphine Jouan-Rimbaud, and Désiré L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems, Vol. 50, 1 (2000), 1--18.Google ScholarGoogle Scholar
  15. Jaap J Dijkstra, Wim BG Liebrand, and Ellen Timminga. 1998. Persuasiveness of expert systems. Behaviour & Information Technology, Vol. 17, 3 (1998), 155--163.Google ScholarGoogle ScholarCross RefCross Ref
  16. Ziv Epstein, Gordon Pennycook, and David Rand. 2020. Will the Crowd Game the Algorithm? Using Layperson Judgments to Combat Misinformation on Social Media by Downranking Distrusted Sources. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, 1--11. https://doi.org/10.1145/3313831.3376232Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Jonathan St BT Evans. 2008. Dual-processing accounts of reasoning, judgment, and social cognition. Annu. Rev. Psychol., Vol. 59 (2008), 255--278.Google ScholarGoogle ScholarCross RefCross Ref
  18. Facebook. 2020. Fact-Checking on Facebook: What Publishers Should Know. https://www.facebook.com/help/publisher/182222309230722. (Accessed on 01/14/2020).Google ScholarGoogle Scholar
  19. FactCheckEU. [n.d.]. FactCheckEU - 19 European media outlets are fact-checking the May 2019 European elections. https://www.factcheckeu.info/en/. (Accessed on 05/14/2020).Google ScholarGoogle Scholar
  20. Andrew J Flanagin and Miriam J Metzger. 2008. Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility (2008), 5--27.Google ScholarGoogle Scholar
  21. Fabrice Florin. 2010. Crowdsourced Fact-Checking? What We Learned from Truthsquad. Mediashift (2010).Google ScholarGoogle Scholar
  22. Brian J Fogg. 2003. Prominence-interpretation theory: Explaining how people assess credibility online. In CHI'03 extended abstracts on human factors in computing systems. Citeseer, 722--723.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Brian J Fogg and Hsiang Tseng. 1999. The elements of computer credibility. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 80--87.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. American Press Institute & The AP-NORC Center for Public Affairs Research. 2018. Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project (2018).Google ScholarGoogle Scholar
  25. Cary Funk, Meg Hefferon, Brian Kennedy, and Courtney Johnson. 2019. Trust and Mistrust in Americans? Views of Scientific Experts. Pew Research Center. https://www. pewresearch. org/science/2019/08/02/trust-and-mistrust-inamericans-views-of-scientific-experts (2019).Google ScholarGoogle Scholar
  26. Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly, Vol. 63, 3 (1986), 451--462.Google ScholarGoogle ScholarCross RefCross Ref
  27. Emma Grillo. 2020. What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr 2020). https://www.nytimes.com/2020/04/05/reader-center/coronavirus-sports-reporting.htmlGoogle ScholarGoogle Scholar
  28. Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on Twitter during the 2016 U.S. presidential election. Science, Vol. 363, 6425 (Jan 2019), 374--378.Google ScholarGoogle Scholar
  29. Naeemul Hassan, Mohammad Yousuf, Mahfuzul Haque, Javier A Suarez Rivas, and Md Khadimul Islam. 2017. Towards A Sustainable Model for Fact-checking Platforms: Examining the Roles of Automation, Crowds and Professionals. https://doi.org/10.1145/3308560.3316734Google ScholarGoogle Scholar
  30. Brian Hilligoss and Soo Young Rieh. 2008. Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, Vol. 44, 4 (2008), 1467--1484.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. (2017), 759--766. arxiv: 1703.09398 http://arxiv.org/abs/1703.09398Google ScholarGoogle Scholar
  32. Carl Iver Hovland, Irving Lester Janis, and Harold H Kelley. 1953. Communication and persuasion. (1953).Google ScholarGoogle Scholar
  33. Rebecca Iannucci and Bill Adair. 2017. Reporters? Lab Study Results: Effective News Labeling and Media Literacy.Google ScholarGoogle Scholar
  34. Jonathan Kennedy. 2019. Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data. European Journal of Public Health, Vol. 29, 3 (Jun 2019), 512--516. https://doi.org/10.1093/eurpub/ckz004Google ScholarGoogle ScholarCross RefCross Ref
  35. Gary King and Richard Nielsen. 2019. Why propensity scores should not be used for matching. Political Analysis, Vol. 27, 4 (2019), 435--454.Google ScholarGoogle ScholarCross RefCross Ref
  36. Spiro Kiousis. 2001. Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society, Vol. 4, 4 (2001), 381--403.Google ScholarGoogle Scholar
  37. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Michael Lucibella. 2009. Science Journalism Faces Perilous Times. American Physical Society (APS) News, Vol. 18, 4 (Apr 2009). http://www.aps.org/publications/apsnews/200904/journalism.cfmGoogle ScholarGoogle Scholar
  39. Albert Mannes, Jack Soll, and Richard Larrick. 2014. The Wisdom of Select Crowds. Journal of personality and social psychology (2014).Google ScholarGoogle Scholar
  40. Albert E. Mannes, Richard P. Larrick, and Jack B. Soll. 2012. The social psychology of the wisdom of crowds.Google ScholarGoogle Scholar
  41. Aaron M. McCright, Katherine Dentzman, Meghan Charters, and Thomas Dietz. 2013. The influence of political ideology on trust in science. Environmental Research Letters, Vol. 8, 4 (Nov 2013), 044029.Google ScholarGoogle ScholarCross RefCross Ref
  42. Miriam J Metzger. 2007. Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, Vol. 58, 13 (2007), 2078--2091.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Miriam J Metzger, Ethan H Hartsell, and Andrew J Flanagin. 2015. Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research (2015), 0093650215613136.Google ScholarGoogle Scholar
  44. Philip Meyer. 1988. Defining and measuring credibility of newspapers: Developing an index. Journalism quarterly, Vol. 65, 3 (1988), 567--574.Google ScholarGoogle ScholarCross RefCross Ref
  45. Amy Mitchell, Jeffrey Gottfried, Michael Barthel, and Nami Sumida. 2018. Can Americans Tell Factual From Opinion Statements in the News?Google ScholarGoogle Scholar
  46. Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations. In Proc. ICWSM'15.Google ScholarGoogle Scholar
  47. Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proc. CHI'15. ACM, 1345--1354.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Kevin Munger, Mario Luca, Jonathan Nagler, and Joshua Tucker. 2019. Age matters: Sampling strategies for studying digital media effects.Google ScholarGoogle Scholar
  49. American Society of Newspaper Editors. 1975. ASNE Statement of Principles. https://members.newsleaders.org/content.asp?pl=24&sl=171&contentid=171. (Accessed on 01/14/2020).Google ScholarGoogle Scholar
  50. Daniel J O'Keefe. 2008. Persuasion. The International Encyclopedia of Communication (2008).Google ScholarGoogle Scholar
  51. Sheila O'Riordan, Gaye Kiely, Bill Emerson, and Joseph Feller. 2019. Do you have a source for that? Understanding the Challenges of Collaborative Evidence-based Journalism. In Proceedings of the 15th International Symposium on Open Collaboration. 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Gordon Pennycook, Tyrone Cannon, and David G. Rand. 2018. Prior Exposure Increases Perceived Accuracy of Fake News. Number ID 2958246. https://papers.ssrn.com/abstract=2958246Google ScholarGoogle Scholar
  53. Gordon Pennycook and David G. Rand. 2019 a. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences, Vol. 116, 7 (Feb 2019), 2521--2526.Google ScholarGoogle ScholarCross RefCross Ref
  54. Gordon Pennycook and David G. Rand. 2019 b. Who Falls for Fake News? The Roles of Bullshit Receptivity, Overclaiming, Familiarity, and Analytic Thinking. Number ID 3023545. https://papers.ssrn.com/abstract=3023545Google ScholarGoogle Scholar
  55. Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion. Springer, 1--24.Google ScholarGoogle Scholar
  56. The Trust Project. 2017. Collaborator Materials. https://thetrustproject.org/collaborator-materials/. (Accessed on 01/14/2020).Google ScholarGoogle Scholar
  57. Soo Young Rieh and David R Danielson. 2007. Credibility: A multidisciplinary framework. Annual review of information science and technology, Vol. 41, 1 (2007), 307--364.Google ScholarGoogle Scholar
  58. Robert M. Ross, David G. Rand, and Gordon Pennycook. 2019. Beyond 'fake news': The role of analytic thinking in the detection of inaccuracy and partisan bias in news headlines. (2019), 1--22.Google ScholarGoogle Scholar
  59. Linda Schamber. 1991. Users' Criteria for Evaluation in a Multimedia Environment.. In Proceedings of the ASIS Annual Meeting, Vol. 28. ERIC, 126--33.Google ScholarGoogle Scholar
  60. Dietram A Scheufele and Nicole M Krause. 2019. Science audiences, misinformation, and fake news. Proceedings of the National Academy of Sciences, Vol. 116, 16 (2019), 7662--7669.Google ScholarGoogle Scholar
  61. Tracy Jia Shen, Robert Cowell, Aditi Gupta, Thai Le, Amulya Yadav, and Dongwon Lee. 2019. How Gullible Are You?: Predicting Susceptibility to Fake News. In Proceedings of the 10th ACM Conference on Web Science (WebSci '19). ACM, 287--288. https://doi.org/10.1145/3292522.3326055 event-place: Boston, Massachusetts, USA.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Art Silverblatt, Donald C. Miller, Julie Smith, and Nikole Brown. 2014. Media Literacy: Keys to Interpreting Media Messages, 4th Edition: Keys to Interpreting Media Messages .ABC-CLIO.Google ScholarGoogle Scholar
  63. Henry Silverman. 2019. Helping Fact-Checkers Identify False Claims Faster - About Facebook. https://about.fb.com/news/2019/12/helping-fact-checkers/. (Accessed on 01/10/2020).Google ScholarGoogle Scholar
  64. Julianne Stanford, Ellen R Tauber, BJ Fogg, and Leslie Marable. 2002. Experts vs. online consumers: A comparative credibility study of health and finance Web sites .Consumer Web Watch.Google ScholarGoogle Scholar
  65. Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research, Vol. 17 (1994), 273--85.Google ScholarGoogle Scholar
  66. S Shyam Sundar. 1999. Exploring receivers' criteria for perception of print and online news. Journalism & Mass Communication Quarterly, Vol. 76, 2 (1999), 373--386.Google ScholarGoogle ScholarCross RefCross Ref
  67. S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility, Vol. 73100 (2008).Google ScholarGoogle Scholar
  68. Cass R. Sunstein. 2006. When Crowds Aren't Wise. Harvard Business Review (Sep 2006). https://hbr.org/2006/09/when-crowds-arent-wiseGoogle ScholarGoogle Scholar
  69. James Surowiecki. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations .Doubleday.Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. Jan-Willem van Prooijen, André P. M. Krouwel, and Thomas V. Pollet. 2015. Political Extremism Predicts Belief in Conspiracy Theories. Social Psychological and Personality Science, Vol. 6, 5 (Jul 2015), 570--578. https://doi.org/10.1177/1948550614567356Google ScholarGoogle ScholarCross RefCross Ref
  71. Christian Wagner and Ayoung Suh. 2014. The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan 2014), 594--603. https://doi.org/10.1109/HICSS.2014.80Google ScholarGoogle Scholar
  72. Lorraine Whitmarsh. 2011. Scepticism and uncertainty about climate change: Dimensions, determinants and change over time. Global Environmental Change, Vol. 21, 2 (May 2011), 690--700.Google ScholarGoogle ScholarCross RefCross Ref
  73. Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science, Vol. 330, 6004 (Oct 2010), 686--688. https://doi.org/10.1126/science.1193147Google ScholarGoogle ScholarCross RefCross Ref
  74. Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Jennifer Lee, et almbox. 2018. A structured response to misinformation: Defining and annotating credibility indicators in news articles. In Companion Proceedings of The Web Conference 2018. 603--612.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image Proceedings of the ACM on Human-Computer Interaction
      Proceedings of the ACM on Human-Computer Interaction  Volume 4, Issue CSCW2
      CSCW
      October 2020
      2310 pages
      EISSN:2573-0142
      DOI:10.1145/3430143
      Issue’s Table of Contents

      Copyright © 2020 ACM

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 15 October 2020
      Published in pacmhci Volume 4, Issue CSCW2

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!