Abstract
Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.
- William RL Anderegg, James W Prall, Jacob Harold, and Stephen H Schneider. 2010. Expert credibility in climate change. Proceedings of the National Academy of Sciences, Vol. 107, 27 (2010), 12107--12109.Google Scholar
Cross Ref
- Lora Aroyo and Chris Welty. 2015. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, Vol. 36, 1 (2015), 15--24.Google Scholar
Digital Library
- Mahmoudreza Babaei, Abhijnan Chakraborty, Juhi Kulshrestha, Elissa M Redmiles, Meeyoung Cha, and Krishna P Gummadi. 2019. Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking.. In FAT. 139.Google Scholar
- Mevan Babakar. 2018. Crowdsourced Factchecking.Google Scholar
- Betsy Jane Becker. 1994. Combining significance levels. The handbook of research synthesis (1994), 215--230.Google Scholar
- Joshua Becker, Ethan Porter, and Damon Centola. 2019. The wisdom of partisan crowds. Proceedings of the National Academy of Sciences of the United States of America, Vol. 166, 22 (2019), 10717--10722. https://doi.org/10.1073/pnas.1817195116Google Scholar
Cross Ref
- Brooke Borel. 2015. The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec 2015). https://www.theguardian.com/media/2015/dec/30/problem-with-science-journalism-2015-reality-kevin-foltaGoogle Scholar
- Alexandre Bovet and Hernán A Makse. 2019. Influence of fake news in Twitter during the 2016 US presidential election. Nature communications, Vol. 10, 1 (2019), 7.Google Scholar
- Mohamad Adam Bujang and Nurakmal Baharum. 2016. Sample size guideline for correlation analysis. World, Vol. 3, 1 (2016).Google Scholar
- Cody Buntain and Jennifer Golbeck. 2017. Automatically Identifying Fake News in Popular Twitter Threads. Proceedings - 2nd IEEE International Conference on Smart Cloud, SmartCloud 2017 (2017), 208--215. https://doi.org/10.1109/SmartCloud.2017.40 arxiv: 1705.01613Google Scholar
Cross Ref
- Davide Ceolin. 2019. Conference Presentation: On the Quality of Crowdsourced Information Quality Assessments. https://drive.google.com/a/hackshackers.com/file/d/1AJmFmRqEhdhSIZLwhXT_1bzStXfV-hVf/view?usp=drive_open&usp=embed_facebookGoogle Scholar
- Steven H Chaffee. 1982. Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter/media: Interpersonal communication in a media world, Vol. 57 (1982), 77.Google Scholar
- Shelly Chaiken. 1987. The heuristic model of persuasion. In Social influence: the ontario symposium, Vol. 5. Hillsdale, NJ: Lawrence Erlbaum, 3--39.Google Scholar
- Roy De Maesschalck, Delphine Jouan-Rimbaud, and Désiré L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems, Vol. 50, 1 (2000), 1--18.Google Scholar
- Jaap J Dijkstra, Wim BG Liebrand, and Ellen Timminga. 1998. Persuasiveness of expert systems. Behaviour & Information Technology, Vol. 17, 3 (1998), 155--163.Google Scholar
Cross Ref
- Ziv Epstein, Gordon Pennycook, and David Rand. 2020. Will the Crowd Game the Algorithm? Using Layperson Judgments to Combat Misinformation on Social Media by Downranking Distrusted Sources. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, 1--11. https://doi.org/10.1145/3313831.3376232Google Scholar
Digital Library
- Jonathan St BT Evans. 2008. Dual-processing accounts of reasoning, judgment, and social cognition. Annu. Rev. Psychol., Vol. 59 (2008), 255--278.Google Scholar
Cross Ref
- Facebook. 2020. Fact-Checking on Facebook: What Publishers Should Know. https://www.facebook.com/help/publisher/182222309230722. (Accessed on 01/14/2020).Google Scholar
- FactCheckEU. [n.d.]. FactCheckEU - 19 European media outlets are fact-checking the May 2019 European elections. https://www.factcheckeu.info/en/. (Accessed on 05/14/2020).Google Scholar
- Andrew J Flanagin and Miriam J Metzger. 2008. Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility (2008), 5--27.Google Scholar
- Fabrice Florin. 2010. Crowdsourced Fact-Checking? What We Learned from Truthsquad. Mediashift (2010).Google Scholar
- Brian J Fogg. 2003. Prominence-interpretation theory: Explaining how people assess credibility online. In CHI'03 extended abstracts on human factors in computing systems. Citeseer, 722--723.Google Scholar
Digital Library
- Brian J Fogg and Hsiang Tseng. 1999. The elements of computer credibility. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 80--87.Google Scholar
Digital Library
- American Press Institute & The AP-NORC Center for Public Affairs Research. 2018. Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project (2018).Google Scholar
- Cary Funk, Meg Hefferon, Brian Kennedy, and Courtney Johnson. 2019. Trust and Mistrust in Americans? Views of Scientific Experts. Pew Research Center. https://www. pewresearch. org/science/2019/08/02/trust-and-mistrust-inamericans-views-of-scientific-experts (2019).Google Scholar
- Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly, Vol. 63, 3 (1986), 451--462.Google Scholar
Cross Ref
- Emma Grillo. 2020. What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr 2020). https://www.nytimes.com/2020/04/05/reader-center/coronavirus-sports-reporting.htmlGoogle Scholar
- Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on Twitter during the 2016 U.S. presidential election. Science, Vol. 363, 6425 (Jan 2019), 374--378.Google Scholar
- Naeemul Hassan, Mohammad Yousuf, Mahfuzul Haque, Javier A Suarez Rivas, and Md Khadimul Islam. 2017. Towards A Sustainable Model for Fact-checking Platforms: Examining the Roles of Automation, Crowds and Professionals. https://doi.org/10.1145/3308560.3316734Google Scholar
- Brian Hilligoss and Soo Young Rieh. 2008. Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, Vol. 44, 4 (2008), 1467--1484.Google Scholar
Digital Library
- Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. (2017), 759--766. arxiv: 1703.09398 http://arxiv.org/abs/1703.09398Google Scholar
- Carl Iver Hovland, Irving Lester Janis, and Harold H Kelley. 1953. Communication and persuasion. (1953).Google Scholar
- Rebecca Iannucci and Bill Adair. 2017. Reporters? Lab Study Results: Effective News Labeling and Media Literacy.Google Scholar
- Jonathan Kennedy. 2019. Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data. European Journal of Public Health, Vol. 29, 3 (Jun 2019), 512--516. https://doi.org/10.1093/eurpub/ckz004Google Scholar
Cross Ref
- Gary King and Richard Nielsen. 2019. Why propensity scores should not be used for matching. Political Analysis, Vol. 27, 4 (2019), 435--454.Google Scholar
Cross Ref
- Spiro Kiousis. 2001. Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society, Vol. 4, 4 (2001), 381--403.Google Scholar
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.Google Scholar
Digital Library
- Michael Lucibella. 2009. Science Journalism Faces Perilous Times. American Physical Society (APS) News, Vol. 18, 4 (Apr 2009). http://www.aps.org/publications/apsnews/200904/journalism.cfmGoogle Scholar
- Albert Mannes, Jack Soll, and Richard Larrick. 2014. The Wisdom of Select Crowds. Journal of personality and social psychology (2014).Google Scholar
- Albert E. Mannes, Richard P. Larrick, and Jack B. Soll. 2012. The social psychology of the wisdom of crowds.Google Scholar
- Aaron M. McCright, Katherine Dentzman, Meghan Charters, and Thomas Dietz. 2013. The influence of political ideology on trust in science. Environmental Research Letters, Vol. 8, 4 (Nov 2013), 044029.Google Scholar
Cross Ref
- Miriam J Metzger. 2007. Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, Vol. 58, 13 (2007), 2078--2091.Google Scholar
Digital Library
- Miriam J Metzger, Ethan H Hartsell, and Andrew J Flanagin. 2015. Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research (2015), 0093650215613136.Google Scholar
- Philip Meyer. 1988. Defining and measuring credibility of newspapers: Developing an index. Journalism quarterly, Vol. 65, 3 (1988), 567--574.Google Scholar
Cross Ref
- Amy Mitchell, Jeffrey Gottfried, Michael Barthel, and Nami Sumida. 2018. Can Americans Tell Factual From Opinion Statements in the News?Google Scholar
- Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations. In Proc. ICWSM'15.Google Scholar
- Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proc. CHI'15. ACM, 1345--1354.Google Scholar
Digital Library
- Kevin Munger, Mario Luca, Jonathan Nagler, and Joshua Tucker. 2019. Age matters: Sampling strategies for studying digital media effects.Google Scholar
- American Society of Newspaper Editors. 1975. ASNE Statement of Principles. https://members.newsleaders.org/content.asp?pl=24&sl=171&contentid=171. (Accessed on 01/14/2020).Google Scholar
- Daniel J O'Keefe. 2008. Persuasion. The International Encyclopedia of Communication (2008).Google Scholar
- Sheila O'Riordan, Gaye Kiely, Bill Emerson, and Joseph Feller. 2019. Do you have a source for that? Understanding the Challenges of Collaborative Evidence-based Journalism. In Proceedings of the 15th International Symposium on Open Collaboration. 1--10.Google Scholar
Digital Library
- Gordon Pennycook, Tyrone Cannon, and David G. Rand. 2018. Prior Exposure Increases Perceived Accuracy of Fake News. Number ID 2958246. https://papers.ssrn.com/abstract=2958246Google Scholar
- Gordon Pennycook and David G. Rand. 2019 a. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences, Vol. 116, 7 (Feb 2019), 2521--2526.Google Scholar
Cross Ref
- Gordon Pennycook and David G. Rand. 2019 b. Who Falls for Fake News? The Roles of Bullshit Receptivity, Overclaiming, Familiarity, and Analytic Thinking. Number ID 3023545. https://papers.ssrn.com/abstract=3023545Google Scholar
- Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion. Springer, 1--24.Google Scholar
- The Trust Project. 2017. Collaborator Materials. https://thetrustproject.org/collaborator-materials/. (Accessed on 01/14/2020).Google Scholar
- Soo Young Rieh and David R Danielson. 2007. Credibility: A multidisciplinary framework. Annual review of information science and technology, Vol. 41, 1 (2007), 307--364.Google Scholar
- Robert M. Ross, David G. Rand, and Gordon Pennycook. 2019. Beyond 'fake news': The role of analytic thinking in the detection of inaccuracy and partisan bias in news headlines. (2019), 1--22.Google Scholar
- Linda Schamber. 1991. Users' Criteria for Evaluation in a Multimedia Environment.. In Proceedings of the ASIS Annual Meeting, Vol. 28. ERIC, 126--33.Google Scholar
- Dietram A Scheufele and Nicole M Krause. 2019. Science audiences, misinformation, and fake news. Proceedings of the National Academy of Sciences, Vol. 116, 16 (2019), 7662--7669.Google Scholar
- Tracy Jia Shen, Robert Cowell, Aditi Gupta, Thai Le, Amulya Yadav, and Dongwon Lee. 2019. How Gullible Are You?: Predicting Susceptibility to Fake News. In Proceedings of the 10th ACM Conference on Web Science (WebSci '19). ACM, 287--288. https://doi.org/10.1145/3292522.3326055 event-place: Boston, Massachusetts, USA.Google Scholar
Digital Library
- Art Silverblatt, Donald C. Miller, Julie Smith, and Nikole Brown. 2014. Media Literacy: Keys to Interpreting Media Messages, 4th Edition: Keys to Interpreting Media Messages .ABC-CLIO.Google Scholar
- Henry Silverman. 2019. Helping Fact-Checkers Identify False Claims Faster - About Facebook. https://about.fb.com/news/2019/12/helping-fact-checkers/. (Accessed on 01/10/2020).Google Scholar
- Julianne Stanford, Ellen R Tauber, BJ Fogg, and Leslie Marable. 2002. Experts vs. online consumers: A comparative credibility study of health and finance Web sites .Consumer Web Watch.Google Scholar
- Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research, Vol. 17 (1994), 273--85.Google Scholar
- S Shyam Sundar. 1999. Exploring receivers' criteria for perception of print and online news. Journalism & Mass Communication Quarterly, Vol. 76, 2 (1999), 373--386.Google Scholar
Cross Ref
- S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility, Vol. 73100 (2008).Google Scholar
- Cass R. Sunstein. 2006. When Crowds Aren't Wise. Harvard Business Review (Sep 2006). https://hbr.org/2006/09/when-crowds-arent-wiseGoogle Scholar
- James Surowiecki. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations .Doubleday.Google Scholar
Digital Library
- Jan-Willem van Prooijen, André P. M. Krouwel, and Thomas V. Pollet. 2015. Political Extremism Predicts Belief in Conspiracy Theories. Social Psychological and Personality Science, Vol. 6, 5 (Jul 2015), 570--578. https://doi.org/10.1177/1948550614567356Google Scholar
Cross Ref
- Christian Wagner and Ayoung Suh. 2014. The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan 2014), 594--603. https://doi.org/10.1109/HICSS.2014.80Google Scholar
- Lorraine Whitmarsh. 2011. Scepticism and uncertainty about climate change: Dimensions, determinants and change over time. Global Environmental Change, Vol. 21, 2 (May 2011), 690--700.Google Scholar
Cross Ref
- Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science, Vol. 330, 6004 (Oct 2010), 686--688. https://doi.org/10.1126/science.1193147Google Scholar
Cross Ref
- Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Jennifer Lee, et almbox. 2018. A structured response to misinformation: Defining and annotating credibility indicators in news articles. In Companion Proceedings of The Web Conference 2018. 603--612.Google Scholar
Digital Library
Index Terms
Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
Recommendations
Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge ManagementFact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus does not scale well on social media because of the continuous flow of ...
GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations
Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we ...
Who's in the Crowd Matters: Cognitive Factors and Beliefs Predict Misinformation Assessment Accuracy
CSCWMisinformation runs rampant on social media and has been tied to adverse health behaviors such as vaccine hesitancy. Crowdsourcing can be a means to detect and impede the spread of misinformation online. However, past studies have not deeply examined ...






Comments