skip to main content
10.1145/3178876.3186143acmotherconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article
Public Access

Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages

Published: 23 April 2018 Publication History

Abstract

Search engines are a primary means through which people obtain information in today»s connected world. Yet, apart from the search engine companies themselves, little is known about how their algorithms filter, rank, and present the web to users. This question is especially pertinent with respect to political queries, given growing concerns about filter bubbles, and the recent finding that bias or favoritism in search rankings can influence voting behavior. In this study, we conduct a targeted algorithm audit of Google Search using a dynamic set of political queries. We designed a Chrome extension to survey participants and collect the Search Engine Results Pages (SERPs) and autocomplete suggestions that they would have been exposed to while searching our set of political queries during the month after Donald Trump»s Presidential inauguration. Using this data, we found significant differences in the composition and personalization of politically-related SERPs by query type, subjects» characteristics, and date.

References

[1]
Ahmed Allam, Peter Johannes Schulz, and Kent Nakamoto. 2014. The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: two experiments manipulating Google output. Journal of Medical Internet Research 16, 4 (2014), e100.
[2]
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives 31, 2 (May 2017), 211--236.
[3]
Solomon E Asch. 1946. Forming impressions of personality. The Journal of Abnormal and Social Psychology 41, 3 (1946), 258--290.
[4]
Eytan Bakshy, Solomon Messing, and Lada A Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science 348, 6239 (2015), 1130--1132.
[5]
Andrea Ballatore, Mark Graham, and Shilad Sen. 2017. Digital hegemonies: the localness of search engine results. Annals of the American Association of Geographers (2017), 1--22.
[6]
Adam J Berinsky, Gregory A Huber, and Gabriel S Lenz. 2012. Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis 20, 3 (2012), 351--368.
[7]
Edelman Berland. 2017. 2017 Edelman Trust Barometer. http://www.edelman. com/trust2017/. (2017). Accessed: 2017-03-07.
[8]
SEO Blog. 2014. How to Turn Off Google Personalized Search Results. http: //www.seoblog.com/2014/09/turn-google-personalized-search-results/. (2014). Accessed: 2017--10-01.
[9]
Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 1 (2011), 3--5.
[10]
Georg Buscher, Edward Cutrell, and Meredith Ringel Morris. 2009. What do you see when you're surfing?: using eye tracking to predict salient regions of web pages. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 21--30.
[11]
Traffic Generation Cafe. 2017. Google Incognito: How to Search Google Without Being Tracked. https://trafficgenerationcafe.com/ search-engine-ranking-tip-incognito-search/. (2017). Accessed: 2017--10-01.
[12]
Le Chen, Alan Mislove, and Christo Wilson. 2015. Peeking Beneath the Hood of Uber. In Proceedings of the 2015 ACM Conference on Internet Measurement.
[13]
Le Chen, Alan Mislove, and Christo Wilson. 2016. An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace. In Proceedings of the 25th International World Wide Web Conference.
[14]
Chitika. 2012. Ask.com Has The Most Long-Winded Searchers, Report Says. Technical Report. Chitika. http://searchengineland.com/ ask-com-has-the-most-long-winded-searchers-report-says-109202
[15]
Inc comScore. 2017. comScore Explicit Core Search Query Report (Desktop Only). https://www.comscore.com/Insights/Rankings. (2017). Accessed: 2017-02--12.
[16]
Nick Diakopoulos. 2013. Algorithmic defamation: The case of the shameless autocomplete. http://www.nickdiakopoulos.com/2013/08/06/ algorithmic-defamation-the-case-of-the-shameless-autocomplete/. (2013).
[17]
Nick Diakopoulos. 2013. Sex, Violence, and Autocomplete Algorithms: Methods and Context. http://www.nickdiakopoulos.com/2013/08/01/ sex-violence-and-autocomplete-algorithms-methods-and-context/. (2013).
[18]
Nicholas Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. Tow Center for Digital Journalism, Columbia University (2014).
[19]
Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. 2016. Visual positions of links and clicks on wikipedia. In Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 27--28.
[20]
Hermann Ebbinghaus. 1913. Memory: A contribution to experimental psychology. Number 3. University Microfilms.
[21]
Robert Epstein and Ronald E Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112, 33 (2015), E4512--E4521.
[22]
Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson. 2017. Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human-Computer Interaction 1, 42 (2017). Issue 2.
[23]
Eyal Ert and Aliza Fleischer. 2016. Mere Position Effect in Booking Hotels Online. Journal of Travel Research 55, 3 (2016), 311--321.
[24]
Motahhare Eslami, Amirhossein Aleyasen, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. Feedvis: A path for exploring news feed curation algorithms. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing.
[25]
Gallup. 2017. Presidential Approval Ratings -- Donald Trump. http://news.gallup. com/poll/203198/presidential-approval-ratings-donald-trump.aspx. (2017). Accessed: 2017--10-08.
[26]
Florian Geigl, Kristina Lerman, Simon Walk, Markus Strohmaier, and Denis Helic. 2016. Assessing the Navigational Effects of Click Biases and Link Insertion on the Web. In Proceedings of the 27th ACM Conference on Hypertext and Social Media. ACM, 37--47.
[27]
David F Gleich, Paul G Constantine, Abraham D Flaxman, and Asela Gunawardana. 2010. Tracking the random surfer: empirically measured teleportation parameters in PageRank. In Proceedings of the 19th International Conference on World Wide Web. ACM, 381--390.
[28]
Google. 2017. Google Chrome Privacy Notice. https://www.google.com/chrome/ browser/privacy/#browser-modes. (2017). Accessed: 2017-04-01.
[29]
Google. 2017. Search using autocomplete. https://support.google.com/websearch/ answer/106230. (2017). Accessed: 2017-04-01.
[30]
Andrew Guess, Brendan Nyhan, and Jason Reifler. 2018. Selective Exposure to Misinformation: Evidence from the consumption of fake news during the 2016 US presidential campaign. (2018).
[31]
Xunhua Guo, Mingyue Zhang, Chenyue Yang, et al. 2016. Order Effects in Online Product Recommendation: A Scenario-based Analysis. In Proceedings of the 22nd Americas Conference on Information Systems.
[32]
Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In Proceedings of the 22nd International World Wide Web Conference.
[33]
Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove, and Christo Wilson. 2014. Measuring Price Discrimination and Steering on E-commerce Web Sites. In Proceedings of the 2014 ACM Conference on Internet Measurement.
[34]
Robin M Hogarth and Hillel J Einhorn. 1992. Order effects in belief updating: The belief-adjustment model. Cognitive Psychology 24, 1 (1992), 1--55.
[35]
Adrianne Jeffries. 2017. Google's featured snippets are worse than fake news. https://theoutline.com/post/1192/ google-s-featured-snippets-are-worse-than-fake-news. (2017). Accessed: 2017--10-08.
[36]
Chloe Kliman-Silver, Aniko Hannak, David Lazer, Christo Wilson, and Alan Mislove. 2015. Location, Location, Location: The Impact of Geolocation on Web Search Personalization. In Proceedings of the 2015 ACM Conference on Internet Measurement.
[37]
Certified Knowledge. 2012. Ask.com Has The Most Long-Winded Searchers, Report Says. Technical Report. Certified Knowledge. http://certifiedknowledge.org/ blog/are-search-queries-becoming-even-more-unique-statistics-from-google/
[38]
Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P Gummadi, and Karrie Karahalios. 2017. Quantifying search bias: Investigating sources of bias for political searches in social media. In 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017).
[39]
David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The Science of Fake News. Science 359, 6380 (March 2018), 1094--1096.
[40]
Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PloS one 9, 6 (2014), e98914.
[41]
Xiaolu Lu, Alistair Moffat, and J Shane Culpepper. 2016. The effect of pooling and evaluation depth on IR metrics. Information Retrieval Journal 19, 4 (2016), 416--445.
[42]
Ramona Ludolph, Ahmed Allam, and Peter J Schulz. 2016. Manipulating Google's Knowledge Graph box to counter biased information processing during an online search on vaccination: application of a technological debiasing strategy. Journal of Medical Internet research 18, 6 (2016).
[43]
Connor McMahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. (2017).
[44]
Nicco Mele, David Lazer, Matthew Baum, Nir Grinberg, Lisa Friedland, Kenneth Joseph, Will Hobbs, and Carolina Mattsson. 2017. Combating fake news: An agenda for research and action. (2017).
[45]
P Takis Metaxas and Yada Pruksachatkun. 2017. Manipulation of Search Engine Results during the 2016 US Congressional Elections. In ICIW.
[46]
Peter J. Meyers. 2013. 101 Google Answer Boxes: A Journey into the Knowledge Graph. https://moz.com/blog/ 101-google-answer-boxes-a-journey-into-the-knowledge-graph. (2013). Accessed: 2017--10-08.
[47]
Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2012. Detecting price and search discrimination on the Internet. In Proceedings of the 11th ACM Workshop on Hot Topics in Networks.
[48]
Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2013. Crowd-assisted search for price discrimination in e-commerce: First results. In Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies.
[49]
Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.
[50]
Amy Mitchell, Jeffrey Gottfried, Elisa Shearer, and Kristine Lu. 2017. How Americans Encounter, Recall and Act Upon Digital News. Technical Report. Pew Research Center.
[51]
Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS) 27, 1 (2008), 2.
[52]
Motherboard. 2017. Go Ahead, Ask Google ?What Happened to the Dinosaurs?. https://motherboard.vice.com/en_us/article/pga4wg/ go-ahead-ask-google-what-happened-to-the-dinosaurs. (2017). Accessed: 2017--10-08.
[53]
Moz. 2012. Face-off - 4 Ways to De-personalize Google. https://moz.com/blog/ face-off-4-ways-to-de-personalize-google. (2012). Accessed: 2017--10-01.
[54]
Sean A Munson, Stephanie Y Lee, and Paul Resnick. 2013. Encouraging Reading of Diverse Political Viewpoints with a Browser Widget. In ICWSM.
[55]
Bennet B Murdock. 1962. The serial position effect of free recall. Journal of Experimental Psychology 64, 5 (1962), 482--488.
[56]
Jamie Murphy, Charles Hofacker, and Richard Mizerski. 2006. Primacy and recency effects on clicking behavior. Journal of Computer-Mediated Communication 11, 2 (2006), 522--535.
[57]
Nic Newman, Richard Fletcher, Antonis Kalogeropoulos, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. Technical Report. Reuters Institute for the Study of Journalism.
[58]
Helen Nissenbaum. 1996. Accountability in a computerized society. Science and Engineering Ethics 2, 1 (1996), 25--42.
[59]
Alamir Novin and Eric Meyers. 2017. Making sense of conflicting science information: Exploring bias in the search engine result page. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval. ACM, 175--184.
[60]
Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In google we trust: Users' decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12, 3 (2007), 801--823.
[61]
Eli Pariser. 2011. The filter bubble: What the Internet is hiding from you. Penguin UK.
[62]
Parse.ly. 2017. The authority report: How audiences find articles, by topic. Technical Report. Parse.ly. http://learn.parsely.com/rs/314-EBB-255/images/ authority-report-13.pdf
[63]
Frank Pasquale. 2015. The black box society: The secret algorithms that control money and information. Harvard University Press, Cambridge, MA.
[64]
Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153--163.
[65]
Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior research methods 46, 4 (2014), 1023--1031.
[66]
Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.
[67]
Advanced Web Ranking. 2017. CTR Study February 2017. https://www. advancedwebranking.com/cloud/ctrstudy. (2017). Accessed: 2017-04-01.
[68]
Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of 'Data and Discrimination: Converting Critical Concerns into Productive Inquiry?, a Productivereconference at the 64th Annual Meeting of the International Communication Association.
[69]
Gary Soeller, Karrie Karahalios, Christian Sandvig, and Christo Wilson. 2016. MapWatch: Detecting and Monitoring International Border Personalization on Online Maps. In Proceedings of the 25th International World Wide Web Conference.
[70]
Lauren Starling. 2013. How to remove a word from Google autocomplete. (2013). http://www.laurenstarling.org/ how-to-remove-a-word-from-google-autocomplete/
[71]
Danny Sullivan. 2017. Google's 'One True Answer' problem when featured snippets go bad. https://searchengineland.com/ googles-one-true-answer-problem-featured-snippets-270549. (2017). Accessed: 2017--10-08.
[72]
Sander van der Linden, Anthony Leiserowitz, Seth Rosenthal, and Edward Maibach. 2017. Inoculating the public against misinformation about climate change. Global Challenges 1, 2 (2017).
[73]
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The Spread of True and False News Online. Science 359, 6380 (March 2018), 1146--1151.
[74]
William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS) 28, 4 (2010), 20.
[75]
Wiideman. 2010. Beat the autocomplete - A study of Google auto-suggest. (2010). https://www.wiideman.com/research/google-autocomplete/study-results

Cited By

View all
  • (2024)Black-Box Access is Insufficient for Rigorous AI AuditsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3659037(2254-2272)Online publication date: 3-Jun-2024
  • (2024)Algorithmic Misjudgement in Google Search Results: Evidence from Auditing the US Online Electoral Information EnvironmentProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658916(433-443)Online publication date: 3-Jun-2024
  • (2024)Dynamics in Search Engine Query Suggestions for European PoliticiansProceedings of the 16th ACM Web Science Conference10.1145/3614419.3644006(279-289)Online publication date: 21-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
WWW '18: Proceedings of the 2018 World Wide Web Conference
April 2018
2000 pages
ISBN:9781450356398
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

  • IW3C2: International World Wide Web Conference Committee

In-Cooperation

Publisher

International World Wide Web Conferences Steering Committee

Republic and Canton of Geneva, Switzerland

Publication History

Published: 23 April 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. autocomplete search suggestions
  2. filter bubble
  3. political personalization
  4. search engine manipulation
  5. search engine results
  6. search ranking bias

Qualifiers

  • Research-article

Funding Sources

Conference

WWW '18
Sponsor:
  • IW3C2
WWW '18: The Web Conference 2018
April 23 - 27, 2018
Lyon, France

Acceptance Rates

WWW '18 Paper Acceptance Rate 170 of 1,155 submissions, 15%;
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)379
  • Downloads (Last 6 weeks)43
Reflects downloads up to 23 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Black-Box Access is Insufficient for Rigorous AI AuditsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3659037(2254-2272)Online publication date: 3-Jun-2024
  • (2024)Algorithmic Misjudgement in Google Search Results: Evidence from Auditing the US Online Electoral Information EnvironmentProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658916(433-443)Online publication date: 3-Jun-2024
  • (2024)Dynamics in Search Engine Query Suggestions for European PoliticiansProceedings of the 16th ACM Web Science Conference10.1145/3614419.3644006(279-289)Online publication date: 21-May-2024
  • (2024)Digital Democracy at Crossroads: A Meta-Analysis of Web and AI Influence on Global ElectionsCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3652003(1126-1129)Online publication date: 13-May-2024
  • (2024)Googling Politics? Comparing Five Computational Methods to Identify Political and News-related Searches from Web Browser HistoriesCommunication Methods and Measures10.1080/19312458.2024.2363776(1-27)Online publication date: 18-Jun-2024
  • (2024)Beyond phase-in: assessing impacts on disinformation of the EU Digital Services ActAI and Ethics10.1007/s43681-024-00467-wOnline publication date: 11-Apr-2024
  • (2023)Auditing YouTube’s Recommendation Algorithm for Misinformation Filter BubblesACM Transactions on Recommender Systems10.1145/35683921:1(1-33)Online publication date: 27-Jan-2023
  • (2023)Assessing enactment of content regulation policies: A post hoc crowd-sourced audit of election misinformation on YouTubeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580846(1-22)Online publication date: 19-Apr-2023
  • (2023)Online searches to evaluate misinformation can increase its perceived veracityNature10.1038/s41586-023-06883-y625:7995(548-556)Online publication date: 20-Dec-2023
  • (2023)Testing the Effect of Information on Discerning the Veracity of News in Real TimeJournal of Experimental Political Science10.1017/XPS.2023.20(1-15)Online publication date: 8-Nov-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media