Abstract
The emerging trend of social information processing has resulted in Web users’ increased reliance on user-generated content contributed by others for information searching and decision making. Rating scores, a form of user-generated content contributed by reviewers in online rating systems, allow users to leverage others’ opinions in the evaluation of objects. In this article, we focus on the problem of summarizing the rating scores given to an object into an overall score that reflects the object’s quality. We observe that the existing approaches for summarizing scores largely ignores the effect of reviewers exercising different standards in assigning scores. Instead of treating all reviewers as equals, our approach models the leniency of reviewers, which refers to the tendency of a reviewer to assign higher scores than other coreviewers. Our approach is underlined by two insights: (1) The leniency of a reviewer depends not only on how the reviewer rates objects, but also on how other reviewers rate those objects and (2) The leniency of a reviewer and the quality of rated objects are mutually dependent. We develop the leniency-aware quality, or LQ model, which solves leniency and quality simultaneously. We introduce both an exact and a ranked solution to the model. Experiments on real-life and synthetic datasets show that LQ is more effective than comparable approaches. LQ is also shown to perform consistently better under different parameter settings.
- Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6, 734--749. Google Scholar
Digital Library
- Anton, H. and Rorres, C. 1987. Elementary Linear Algebra with Applications. John Wiley & Sons, Hoboken, NJ.Google Scholar
- Arampatzis, A. and Kamps, J. 2009. A signal-to-noise approach to score normalization. In Proceeding of the 18th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 797--806. Google Scholar
Digital Library
- Arkes, H. R. 2003. The nonuse of psychological research at two federal agencies. Psychol. Science 14, 1, 1--6.Google Scholar
Cross Ref
- Backstrom, L., Dwork, C., and Kleinberg, J. 2007. Wherefore art thou r3579x?: Anonymized social networks, hidden patterns, and structural steganography. In Proceedings of the 16th International World Wide Web Conference. ACM, New York, 181--190. Google Scholar
Digital Library
- Bazerman, M. H. 1990. Judgment in Managerial Decision-Making 2nd Ed. Wiley, Hoboken, NJ.Google Scholar
- Bell, R. M. and Koren, Y. 2007. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining. 43--52. Google Scholar
Digital Library
- Bhattacharjee, R. and Goel, A. 2005. Avoiding ballot stuffing in Ebay-like reputation systems. In Proceeding of the ACM SIGCOMM Workshop on Economics of Peer-to-Peer Systems. ACM, New York, 133--137. Google Scholar
Digital Library
- Blackburn, J. L. and Hakel, M. D. 2006. An examination of sources of peer-review bias. Psychol. Science 17, 5, 378--382.Google Scholar
Cross Ref
- Borgatti, S. P. and Everett, M. G. 1997. Network analysis of 2-mode data. Social Netw. 19, 3, 243--269.Google Scholar
Cross Ref
- Borodin, A., Roberts, G. O., Rosenthal, J. S., and Tsaparas, P. 2005. Link analysis ranking: Algorithms, theory, and experiments. ACM Trans. Internet Technol. 5, 1, 231--297. Google Scholar
Digital Library
- Busenitz, L. and Lau, C. 1996. A cross-cultural cognitive model of new venture creation. Entrepreneurship: Theory Pract. 20, 4, 25--39.Google Scholar
Cross Ref
- Chen, M. and Singh, J. P. 2001. Computing and using reputations for internet ratings. In Proceedings of the 3rd ACM Conference on Electronic Commerce. ACM, New York, 154--162. Google Scholar
Digital Library
- Dellarocas, C. 2000. Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior. In Proceedings of the 2nd ACM Conference on Electronic Commerce. ACM, New York, 150--157. Google Scholar
Digital Library
- Dumais, S. and Nielsen, J. 1992. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 233--244. Google Scholar
Digital Library
- Dwork, C., Kumar, R., Naor, M., and Sivakumar, D. 2001. Rank aggregation methods for the Web. In Proceedings of the 10th International World Wide Web Conference. 613--622. Google Scholar
Digital Library
- Fagin, R., Kumar, R., and Sivakumar, D. 2003. Comparing top k lists. SIAM J. Discrete Math. 17, 1, 134--160. Google Scholar
Digital Library
- Faust, K. 1997. Centrality in affiliation networks. Social Netw. 19, 2, 157--191.Google Scholar
Cross Ref
- Fernandez, M., Vallet, D., and Castells, P. 2006. Probabilistic score normalization for rank aggregation. In Proceedings of the European Conference on Information Retrieval. 553--556. Google Scholar
Digital Library
- Figueira, J., Greco, S., and Ehrgott, M., Eds. 2005. Multiple Criteria Decision Analysis: State of the Art Surveys. Springer Science and Business Media, Inc.Google Scholar
- Geller, J. and Scherl, R. 1997. Challenge: Technology for automated reviewer selection. In Proceedings of the International Joint Conferences on Artificial Intelligence. 55--61.Google Scholar
- Golbeck, J. and Hendler, J. 2006. Inferring binary trust relationships in Web-based social networks. ACM Trans. Internet Technol. 6, 4, 497--529. Google Scholar
Digital Library
- Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations 3rd Ed. Johns Hopkins University Press. Google Scholar
Digital Library
- Grimmett, G. R. and Stirzaker, D. R. 1982. Probability and Random Processes. Oxford University Press, Oxford, UK.Google Scholar
- Guha, R., Kumar, R., Raghavan, P., and Tomkins, A. 2004. Propagation of trust and distrust. In Proceedings of the International World Wide Web Conference. ACM, New York. Google Scholar
Digital Library
- Haveliwala, T. H. 2003. Topic-sensitive PageRank: A context-sensitive ranking algorithm for Web search. IEEE Trans. Knowl. Data Engin. 15, 4, 784--796. Google Scholar
Digital Library
- Herlocker, J. L., Konstan, J. A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY, 241--250. Google Scholar
Digital Library
- Herlocker, J., Konstan, J. A., and Riedl, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Information Retrieval 5, 287--310. Google Scholar
Digital Library
- Hettich, S. and Pazzani, M. J. 2006. Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 862--871. Google Scholar
Digital Library
- Jin, R. and Si, L. 2004. A study of methods for normalizing user ratings in collaborative filtering. In Proceedings of the 27th ACM SIGIR Conference on Research and Development in Information Retrieval. 568--569. Google Scholar
Digital Library
- Jin, R., Si, L., Zhai, C.-X., and Callan, J. 2003. Collaborative filtering with decoupled models for preferences and ratings. In Proceedings of the International Conference on Information and Knowledge Management. ACM, New York, NY, 309--316. Google Scholar
Digital Library
- Jindal, N. and Liu, B. 2007. Analyzing and detecting review spam. In Proceedings of the IEEE International Conference on Data Mining. IEEE Computer Society, Los, Alamitos, CA. Google Scholar
Digital Library
- Kleinberg, J. M. 1999. Authoritative sources in a hyperlinked environment. J. ACM 46, 5, 604--632. Google Scholar
Digital Library
- Korhonen, P., Moskowitz, H., and Wallenius, J. 1992. Multiple criteria decision support---a review. Euro. J. Operational Res. 63, 3, 361--375.Google Scholar
Cross Ref
- Lam, S. K. and Riedl, J. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th International World Wide Web Conference. ACM, New York, 393--402. Google Scholar
Digital Library
- Lauw, H. W., Lim, E.-P., and Wang, K. 2007. Summarizing review scores of “unequal” reviewers. In Proceedings of the 2007 SIAM International Conference on Data Mining.Google Scholar
- Lemire, D. 2005. Scale and translation invariant collaborative filtering systems. Info. Retrieval 8, 1, 129--150. Google Scholar
Digital Library
- Ma, H., Yang, H., Lyu, M., and King, I. 2008. Sorec: social recommendation using probabilistic matrix factorization. In CIKM. ACM, New York, NY, 931--940. Google Scholar
Digital Library
- Ma, H., King, I., and Lyu, M. R. 2009. Learning to recommend with social trust ensemble. In SIGIR. ACM, New York, NY, 203--210. Google Scholar
Digital Library
- Manmatha, R. and Sever, H. 2002. A formal approach to score normalization for meta-search. In Proceedings of the 2nd International Conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., San Francisco, CA, 98--103. Google Scholar
Digital Library
- Massa, P. and Avesani, P. 2005. Controversial users demand local trust metrics: An experimental study on epinions.com community. In AAAI. AAAI Press, 121--126. Google Scholar
Digital Library
- Mobasher, B., Burke, R., and Sandvig, J. 2006. Model-based collaborative filtering as a defense against profile injection attacks. In Proceedings of the National Conference on Artificial Intelligence. Google Scholar
Digital Library
- Page, L., Brin, S., Motwani, R., and Winograd, T. 1998. The PageRank citation ranking: Bringing order to the Web. Stanford Digital Library Technologies Project.Google Scholar
- Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 175--186. Google Scholar
Digital Library
- Riggs, T. and Wilensky, R. 2001. An algorithm for automated rating of reviewers. In Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries. ACM, New York, 381--387. Google Scholar
Digital Library
- Sarwar, B. M., Karypis, G., Konstan, J. A., and Riedl, J. 2000. Application of dimensionality reduction in recommender system -- a case study. In Proceedings of the ACM WebKDD Web Mining for E-Commerce Workshop.Google Scholar
- Shen, J., Lin, Y., Xue, G.-R., Zhu, F.-D., and Yao, A.-G. 2006. IRFCF: Iterative rating filling collaborative filtering algorithm. In Proceedings of the 8th Asia-Pacific Web Conference. 862--867. Google Scholar
Digital Library
- Simon, M., Houghton, S. M., and Aquino, K. 1999. Cognitive biases, risk perception, and venture formation: How individuals decide to start companies. J. Bus. Venturing 15, 113--134.Google Scholar
Cross Ref
- Tantipathananandh, C., Berger-Wolf, T., and Kempe, D. 2007. A framework for community identification in dynamic social networks. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 717--726. Google Scholar
Digital Library
- Traupman, J. and Wilensky, R. 2004a. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Google Scholar
Digital Library
- Traupman, J. and Wilensky, R. 2004b. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 73--86. Google Scholar
Digital Library
- Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. 2002. Probability & Statistics for Engineers & Scientists 7th Ed. Prentice Hall, NJ.Google Scholar
- Wasserman, S. and Faust, K. 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge, UK.Google Scholar
- Yang, B., Cheung, W., and Liu, J. 2007. Community mining from signed social networks. IEEE Trans. Knowl. Data Engin. 19, 10, 1333--1348. Google Scholar
Digital Library
- Zhang, J. and Cohen, R. 2006. Trusting advice from other buyers in e-marketplaces: The problem of unfair ratings. In Proceedings of the 8th International Conference on Electronic Commerce: The new e-Commerce: Innovations for Conquering Current Barriers, Obstacles and Limitations to Conducting Successful Business on the Internet. ACM, New York, 225--234. Google Scholar
Digital Library
- Zhou, D., Manavoglu, E., Li, J., Giles, C. L., and Zha, H. 2006. Probabilistic models for discovering e-communities. In Proceedings of the 15th International World Wide Web Conference. ACM, New York, 173--182. Google Scholar
Digital Library
Index Terms
Quality and Leniency in Online Collaborative Rating Systems
Recommendations
A Neural Network Go Rating Model Considering Winning Rate
CSAI '19: Proceedings of the 2019 3rd International Conference on Computer Science and Artificial IntelligenceThe existing rating system, as the foundation of estimating results of competitive sports and games, has problems of insufficient use of competing information and the unilateral considering of competing process modeling. In this paper, the Artificial ...
A document rating system for preference judgements
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrievalHigh quality relevance judgments are essential for the evaluation of information retrieval systems. Traditional methods of collecting relevance judgments are based on collecting binary or graded nominal judgments, but such judgments are limited by ...
The Persistence of Lenient Market Categories
Research across disciplines presumes that market categories will have strong boundaries. Categories without well-defined boundaries typically are not useful and so are expected to fade away. We suggest many contexts contain lenient market categories, or ...






Comments