research-article

Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems

Authors:

Rishabh Mehrotra,

James McInerney,

Hugues Bouchard,

Fernando DiazAuthors Info & Claims

CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

Pages 2243 - 2251

https://doi.org/10.1145/3269206.3272027

Published: 17 October 2018 Publication History

Abstract

Two-sided marketplaces are platforms that have customers not only on the demand side (e.g. users), but also on the supply side (e.g. retailer, artists). While traditional recommender systems focused specifically towards increasing consumer satisfaction by providing relevant content to consumers, two-sided marketplaces face the problem of additionally optimizing for supplier preferences, and visibility. Indeed, the suppliers would want afair opportunity to be presented to users. Blindly optimizing for consumer relevance may have a detrimental impact on supplier fairness. Motivated by this problem, we focus on the trade-off between objectives of consumers and suppliers in the case of music streaming services, and consider the trade-off betweenrelevance of recommendations to the consumer (i.e. user) andfairness of representation of suppliers (i.e. artists) and measure their impact on consumersatisfaction.

We propose a conceptual and computational framework using counterfactual estimation techniques to understand, and evaluate different recommendation policies, specifically around the trade-off between relevance and fairness, without the need for running many costly A/B tests. We propose a number of recommendation policies which jointly optimize relevance and fairness, thereby achieving substantial improvement in supplier fairness without noticeable decline in user satisfaction. Additionally, we consider user disposition towards fair content, and propose a personalized recommendation policy which takes into account consumer's tolerance towards fair content. Our findings could guide the design of algorithms powering two-sided marketplaces, as well as guide future research on sophisticated algorithms for joint optimization of user relevance, satisfaction and fairness.

References

[1]

H. Abdollahpouri, R. Burke, and B. Mobasher. {n. d.}. Recommender Systems as Multistakeholder Environments. In Proceedings of UMAP 2017.

Digital Library

[2]

Rediet Abebe, Jon Kleinberg, and David C Parkes. {n. d.}. Fair division via social comparison. In Proceedings of AAMAS 2017 .

Digital Library

[3]

Mark Armstrong. 2006. Competition in two-sided markets. The RAND Journal of Economics, Vol. 37, 3 (2006), 668--691.

[4]

Asia J Biega, Krishna P Gummadi, and Gerhard Weikum. 2018. Equity of Attention: Amortizing Individual Fairness in Rankings. arXiv:1805.01788 (2018).

Digital Library

[5]

Robin Burke. 2017. Multisided Fairness for Recommendation. arXiv preprint arXiv:1707.00093 (2017).

[6]

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. {n. d.}. Fairness through awareness. In Proceedings ITCS 2012 .

Digital Library

[7]

Thomas Eisenmann, Geoffrey Parker, and Marshall W Van Alstyne. 2006. Strategies for two-sided markets. Harvard business review, Vol. 84, 10 (2006), 92.

[8]

Henry A Feild, James Allan, and Rosie Jones. {n. d.}. Predicting searcher frustration. In SIGIR 2010 .

Digital Library

[9]

Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2018. Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning. (2018).

[10]

Qi Guo, Dmitry Lagun, and Eugene Agichtein. {n. d.}. Predicting web search success with fine-grained interaction data. In CIKM 2012 .

Digital Library

[11]

Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey. {n. d.}. Beyond clicks: query reformulation as a predictor of search satisfaction. In CIKM 2013 .

Digital Library

[12]

Daniel G Horvitz and Donovan J Thompson. 1952. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, Vol. 47, 260 (1952), 663--685.

[13]

Diane Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval (2009).

Digital Library

[14]

Lihong Li, Shunbao Chen, Jim Kleban, and Ankur Gupta. {n. d.} a. Counterfactual estimation and optimization of click metrics in search engines: A case study. In Proceedings WWW 2015 .

Digital Library

[15]

Lihong Li, Wei Chu, Langford, and Schapire. {n. d.} b. A contextual-bandit approach to personalized news article recommendation. In WWW 2010 .

Digital Library

[16]

Bertin Martens. 2016. An economic policy perspective on online platforms. (2016).

[17]

Rishabh Mehrotra, Ashton Anderson, Fernando Diaz, Amit Sharma, Hanna Wallach, and Emine Yilmaz. 2017a. Auditing search engines for differential satisfaction across demographics. In Proceedings of the 26th International Conference on World Wide Web Companion . International World Wide Web Conferences Steering Committee, 626--633.

Digital Library

[18]

Rishabh Mehrotra, Imed Zitouni, Ahmed Hassan Awadallah, Ahmed El Kholy, and Madian Khabsa. 2017b. User Interaction Sequences for Search Satisfaction Prediction. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 165--174.

Digital Library

[19]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. {n. d.}. Distributed representations of words and phrases and their compositionality. In NIPS 2013 .

Digital Library

[20]

Thomas Nedelec, Nicolas Le Roux, and Vianney Perchet. 2017. A comparative study of counterfactual estimators. arXiv preprint arXiv:1704.00773 (2017).

[21]

Sherwin Rosen. 1981. The economics of superstars. The American economic review, Vol. 71, 5 (1981), 845--858.

[22]

Donald B Rubin. 1978. Bayesian inference for causal effects: The role of randomization. The Annals of statistics (1978), 34--58.

[23]

Marc Rysman. 2009. The economics of two-sided markets. Journal of Economic Perspectives, Vol. 23, 3 (2009), 125--43.

[24]

Parikshit Shah, Akshay Soni, and Troy Chevalier. 2017. Online Ranking with Constraints: A Primal-Dual Algorithm and Applications to Web Traffic-Shaping. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 405--414.

Digital Library

[25]

Ashudeep Singh and Thorsten Joachims. 2018. Fairness of Exposure in Rankings. arXiv preprint arXiv:1802.07281 (2018).

Digital Library

[26]

Srinivasaraghavan Sriram, Puneet Manchanda, and Bravo. 2015. Platforms: a multiplicity of research opportunities. Marketing Letters (2015).

[27]

Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. {n. d.}. Off-policy evaluation for slate recommendation. In NIPS 2017 .

[28]

Xing Yi, Liangjie Hong, Erheng Zhong, Nanthan Nan Liu, and Suju Rajan. {n. d.}. Beyond clicks: dwell time for personalization. In ProceedingsRecSys 2014 .

Digital Library

[29]

Shuai Yuan, Ahmad Zainal Abidin, Marc Sloan, and Jun Wang. 2012. Internet advertising: An interplay among advertisers, online publishers, ad exchanges and web users. arXiv preprint arXiv:1206.1754 (2012).

[30]

Muhammad Zafar, Valera, Gomez Rodriguez, and Gummadi. {n. d.}. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of WWW 2017 .

Digital Library

[31]

Li Zhou and Emma Brunskill. 2016. Latent contextual bandits and their application to personalized recommendations for new users. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. AAAI Press, 3646--3653.

Digital Library

Cited By

Canbay YUtku A(2024)A Comparative Study for Privacy-Aware Recommendation SystemsGazi University Journal of Science Part A: Engineering and Innovation10.54287/gujsa.139369211:1(68-79)Online publication date: 28-Mar-2024
https://doi.org/10.54287/gujsa.1393692
Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Yang HWu XQiu ZZheng YChen X(2024)Distributional Fairness-aware RecommendationACM Transactions on Information Systems10.1145/365285442:5(1-28)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3652854
Show More Cited By

Index Terms

Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Introduction of the marketplace channel under logistics service sharing in an e-commerce platform
Highlights
- The interaction between the channel structure and logistics mode is studied.
- ...
Abstract
In online retailing, in addition to the reseller channel, the marketplace channel has been favored by many e-commerce platforms and suppliers. Different from the reseller channel, the novel channel enables suppliers to obtain the ...
Platform sharing: From reseller to marketplace
Highlights
- We examine the motivations for an online retailer that exclusively resells goods to open its platform to competing sellers.
Abstract
We examine the motivations for an online retailer that exclusively resells goods to open its platform to competing sellers. We further examine competing seller’s incentives to join such a platform. The retailer can receive rent and ...
Marketplace or Reseller?

Intermediaries can choose between functioning as a marketplace in which suppliers sell their products directly to buyers or as a reseller by purchasing products from suppliers and selling them to buyers. We model this as a decision between whether ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

October 2018

2362 pages

ISBN:9781450360142

DOI:10.1145/3269206

General Chair:
Alfredo Cuzzocrea
University of Trieste, Italy
,
Program Chairs:
James Allan
University of Massachusetts, USA
,
Norman Paton
University of Manchester, United Kingdom
,
Divesh Srivastava
AT&T Labs Research, USA
,
Rakesh Agrawal
Data Insights Lab, USA
,
Andrei Broder
Google Research, USA
,
Mohammed Zaki
Rensselaer Polytechnic Institute, USA
,
Selcuk Candan
Arizona State University, USA
,
Alexandros Labrinidis
University of Pittsburgh, USA
,
Assaf Schuster
Technion, Israel
,
Haixun Wang
Google Research, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '18

Sponsor:

CIKM '18: The 27th ACM International Conference on Information and Knowledge Management

October 22 - 26, 2018

Torino, Italy

Acceptance Rates

CIKM '18 Paper Acceptance Rate 147 of 826 submissions, 18%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

159
Total Citations
View Citations
1,914
Total Downloads

Downloads (Last 12 months)294
Downloads (Last 6 weeks)18

Reflects downloads up to 24 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Canbay YUtku A(2024)A Comparative Study for Privacy-Aware Recommendation SystemsGazi University Journal of Science Part A: Engineering and Innovation10.54287/gujsa.139369211:1(68-79)Online publication date: 28-Mar-2024
https://doi.org/10.54287/gujsa.1393692
Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Yang HWu XQiu ZZheng YChen X(2024)Distributional Fairness-aware RecommendationACM Transactions on Information Systems10.1145/365285442:5(1-28)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3652854
Ferraro AFerreira GDiaz FBorn G(2024)Measuring Commonality in Recommendation of Cultural Content to Strengthen Cultural CitizenshipACM Transactions on Recommender Systems10.1145/36431382:1(1-32)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3643138
Jeunen OPotapov IUstimenko ABaeza-Yates RBonchi F(2024)On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671687(1222-1233)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671687
Kruse JMichiels LStarke ATintarev NVrijenhoek S(2024)NORMalize: A Tutorial on the Normative Design and Evaluation of Information Access SystemsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638319(422-424)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638319
Gómez EContreras DSalamo MBoratto L(2024)Bringing Equity to Coarse and Fine-Grained Provider Groups in Recommender SystemsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659552(18-23)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659552
Wang YSun PMa WZhang MZhang YJiang PMa SChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Intersectional Two-sided Fairness in RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645518(3609-3620)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645518
Čutura LVladimir KDelač GŠilić M(2024)Fairness in Graph-based Recommendation: Methods Overview2024 47th MIPRO ICT and Electronics Convention (MIPRO)10.1109/MIPRO60963.2024.10569148(850-855)Online publication date: 20-May-2024
https://doi.org/10.1109/MIPRO60963.2024.10569148
Zhang HWu BYuan XPan STong HPei J(2024)Trustworthy Graph Neural Networks: Aspects, Methods, and TrendsProceedings of the IEEE10.1109/JPROC.2024.3369017112:2(97-139)Online publication date: Mar-2024
https://doi.org/10.1109/JPROC.2024.3369017
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents