skip to main content
research-article

Quality and Leniency in Online Collaborative Rating Systems

Published:01 March 2012Publication History
Skip Abstract Section

Abstract

The emerging trend of social information processing has resulted in Web users’ increased reliance on user-generated content contributed by others for information searching and decision making. Rating scores, a form of user-generated content contributed by reviewers in online rating systems, allow users to leverage others’ opinions in the evaluation of objects. In this article, we focus on the problem of summarizing the rating scores given to an object into an overall score that reflects the object’s quality. We observe that the existing approaches for summarizing scores largely ignores the effect of reviewers exercising different standards in assigning scores. Instead of treating all reviewers as equals, our approach models the leniency of reviewers, which refers to the tendency of a reviewer to assign higher scores than other coreviewers. Our approach is underlined by two insights: (1) The leniency of a reviewer depends not only on how the reviewer rates objects, but also on how other reviewers rate those objects and (2) The leniency of a reviewer and the quality of rated objects are mutually dependent. We develop the leniency-aware quality, or LQ model, which solves leniency and quality simultaneously. We introduce both an exact and a ranked solution to the model. Experiments on real-life and synthetic datasets show that LQ is more effective than comparable approaches. LQ is also shown to perform consistently better under different parameter settings.

References

  1. Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6, 734--749. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Anton, H. and Rorres, C. 1987. Elementary Linear Algebra with Applications. John Wiley & Sons, Hoboken, NJ.Google ScholarGoogle Scholar
  3. Arampatzis, A. and Kamps, J. 2009. A signal-to-noise approach to score normalization. In Proceeding of the 18th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 797--806. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Arkes, H. R. 2003. The nonuse of psychological research at two federal agencies. Psychol. Science 14, 1, 1--6.Google ScholarGoogle ScholarCross RefCross Ref
  5. Backstrom, L., Dwork, C., and Kleinberg, J. 2007. Wherefore art thou r3579x?: Anonymized social networks, hidden patterns, and structural steganography. In Proceedings of the 16th International World Wide Web Conference. ACM, New York, 181--190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Bazerman, M. H. 1990. Judgment in Managerial Decision-Making 2nd Ed. Wiley, Hoboken, NJ.Google ScholarGoogle Scholar
  7. Bell, R. M. and Koren, Y. 2007. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining. 43--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Bhattacharjee, R. and Goel, A. 2005. Avoiding ballot stuffing in Ebay-like reputation systems. In Proceeding of the ACM SIGCOMM Workshop on Economics of Peer-to-Peer Systems. ACM, New York, 133--137. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Blackburn, J. L. and Hakel, M. D. 2006. An examination of sources of peer-review bias. Psychol. Science 17, 5, 378--382.Google ScholarGoogle ScholarCross RefCross Ref
  10. Borgatti, S. P. and Everett, M. G. 1997. Network analysis of 2-mode data. Social Netw. 19, 3, 243--269.Google ScholarGoogle ScholarCross RefCross Ref
  11. Borodin, A., Roberts, G. O., Rosenthal, J. S., and Tsaparas, P. 2005. Link analysis ranking: Algorithms, theory, and experiments. ACM Trans. Internet Technol. 5, 1, 231--297. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Busenitz, L. and Lau, C. 1996. A cross-cultural cognitive model of new venture creation. Entrepreneurship: Theory Pract. 20, 4, 25--39.Google ScholarGoogle ScholarCross RefCross Ref
  13. Chen, M. and Singh, J. P. 2001. Computing and using reputations for internet ratings. In Proceedings of the 3rd ACM Conference on Electronic Commerce. ACM, New York, 154--162. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Dellarocas, C. 2000. Immunizing online reputation reporting systems against unfair ratings and discriminatory behavior. In Proceedings of the 2nd ACM Conference on Electronic Commerce. ACM, New York, 150--157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Dumais, S. and Nielsen, J. 1992. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 233--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Dwork, C., Kumar, R., Naor, M., and Sivakumar, D. 2001. Rank aggregation methods for the Web. In Proceedings of the 10th International World Wide Web Conference. 613--622. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Fagin, R., Kumar, R., and Sivakumar, D. 2003. Comparing top k lists. SIAM J. Discrete Math. 17, 1, 134--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Faust, K. 1997. Centrality in affiliation networks. Social Netw. 19, 2, 157--191.Google ScholarGoogle ScholarCross RefCross Ref
  19. Fernandez, M., Vallet, D., and Castells, P. 2006. Probabilistic score normalization for rank aggregation. In Proceedings of the European Conference on Information Retrieval. 553--556. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Figueira, J., Greco, S., and Ehrgott, M., Eds. 2005. Multiple Criteria Decision Analysis: State of the Art Surveys. Springer Science and Business Media, Inc.Google ScholarGoogle Scholar
  21. Geller, J. and Scherl, R. 1997. Challenge: Technology for automated reviewer selection. In Proceedings of the International Joint Conferences on Artificial Intelligence. 55--61.Google ScholarGoogle Scholar
  22. Golbeck, J. and Hendler, J. 2006. Inferring binary trust relationships in Web-based social networks. ACM Trans. Internet Technol. 6, 4, 497--529. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Golub, G. H. and Van Loan, C. F. 1996. Matrix Computations 3rd Ed. Johns Hopkins University Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Grimmett, G. R. and Stirzaker, D. R. 1982. Probability and Random Processes. Oxford University Press, Oxford, UK.Google ScholarGoogle Scholar
  25. Guha, R., Kumar, R., Raghavan, P., and Tomkins, A. 2004. Propagation of trust and distrust. In Proceedings of the International World Wide Web Conference. ACM, New York. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Haveliwala, T. H. 2003. Topic-sensitive PageRank: A context-sensitive ranking algorithm for Web search. IEEE Trans. Knowl. Data Engin. 15, 4, 784--796. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Herlocker, J. L., Konstan, J. A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY, 241--250. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Herlocker, J., Konstan, J. A., and Riedl, J. 2002. An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Information Retrieval 5, 287--310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Hettich, S. and Pazzani, M. J. 2006. Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 862--871. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Jin, R. and Si, L. 2004. A study of methods for normalizing user ratings in collaborative filtering. In Proceedings of the 27th ACM SIGIR Conference on Research and Development in Information Retrieval. 568--569. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Jin, R., Si, L., Zhai, C.-X., and Callan, J. 2003. Collaborative filtering with decoupled models for preferences and ratings. In Proceedings of the International Conference on Information and Knowledge Management. ACM, New York, NY, 309--316. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Jindal, N. and Liu, B. 2007. Analyzing and detecting review spam. In Proceedings of the IEEE International Conference on Data Mining. IEEE Computer Society, Los, Alamitos, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Kleinberg, J. M. 1999. Authoritative sources in a hyperlinked environment. J. ACM 46, 5, 604--632. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Korhonen, P., Moskowitz, H., and Wallenius, J. 1992. Multiple criteria decision support---a review. Euro. J. Operational Res. 63, 3, 361--375.Google ScholarGoogle ScholarCross RefCross Ref
  35. Lam, S. K. and Riedl, J. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th International World Wide Web Conference. ACM, New York, 393--402. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Lauw, H. W., Lim, E.-P., and Wang, K. 2007. Summarizing review scores of “unequal” reviewers. In Proceedings of the 2007 SIAM International Conference on Data Mining.Google ScholarGoogle Scholar
  37. Lemire, D. 2005. Scale and translation invariant collaborative filtering systems. Info. Retrieval 8, 1, 129--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Ma, H., Yang, H., Lyu, M., and King, I. 2008. Sorec: social recommendation using probabilistic matrix factorization. In CIKM. ACM, New York, NY, 931--940. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Ma, H., King, I., and Lyu, M. R. 2009. Learning to recommend with social trust ensemble. In SIGIR. ACM, New York, NY, 203--210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Manmatha, R. and Sever, H. 2002. A formal approach to score normalization for meta-search. In Proceedings of the 2nd International Conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., San Francisco, CA, 98--103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Massa, P. and Avesani, P. 2005. Controversial users demand local trust metrics: An experimental study on epinions.com community. In AAAI. AAAI Press, 121--126. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Mobasher, B., Burke, R., and Sandvig, J. 2006. Model-based collaborative filtering as a defense against profile injection attacks. In Proceedings of the National Conference on Artificial Intelligence. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Page, L., Brin, S., Motwani, R., and Winograd, T. 1998. The PageRank citation ranking: Bringing order to the Web. Stanford Digital Library Technologies Project.Google ScholarGoogle Scholar
  44. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 175--186. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Riggs, T. and Wilensky, R. 2001. An algorithm for automated rating of reviewers. In Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries. ACM, New York, 381--387. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Sarwar, B. M., Karypis, G., Konstan, J. A., and Riedl, J. 2000. Application of dimensionality reduction in recommender system -- a case study. In Proceedings of the ACM WebKDD Web Mining for E-Commerce Workshop.Google ScholarGoogle Scholar
  47. Shen, J., Lin, Y., Xue, G.-R., Zhu, F.-D., and Yao, A.-G. 2006. IRFCF: Iterative rating filling collaborative filtering algorithm. In Proceedings of the 8th Asia-Pacific Web Conference. 862--867. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Simon, M., Houghton, S. M., and Aquino, K. 1999. Cognitive biases, risk perception, and venture formation: How individuals decide to start companies. J. Bus. Venturing 15, 113--134.Google ScholarGoogle ScholarCross RefCross Ref
  49. Tantipathananandh, C., Berger-Wolf, T., and Kempe, D. 2007. A framework for community identification in dynamic social networks. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, 717--726. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Traupman, J. and Wilensky, R. 2004a. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Traupman, J. and Wilensky, R. 2004b. Collaborative quality filtering: Establishing consensus or recovering ground truth? In Proceedings of the KDD Workshop on Web Mining and Web Usage Analysis, in conjunction with the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 73--86. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. 2002. Probability & Statistics for Engineers & Scientists 7th Ed. Prentice Hall, NJ.Google ScholarGoogle Scholar
  53. Wasserman, S. and Faust, K. 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, Cambridge, UK.Google ScholarGoogle Scholar
  54. Yang, B., Cheung, W., and Liu, J. 2007. Community mining from signed social networks. IEEE Trans. Knowl. Data Engin. 19, 10, 1333--1348. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Zhang, J. and Cohen, R. 2006. Trusting advice from other buyers in e-marketplaces: The problem of unfair ratings. In Proceedings of the 8th International Conference on Electronic Commerce: The new e-Commerce: Innovations for Conquering Current Barriers, Obstacles and Limitations to Conducting Successful Business on the Internet. ACM, New York, 225--234. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Zhou, D., Manavoglu, E., Li, J., Giles, C. L., and Zha, H. 2006. Probabilistic models for discovering e-communities. In Proceedings of the 15th International World Wide Web Conference. ACM, New York, 173--182. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Quality and Leniency in Online Collaborative Rating Systems

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on the Web
        ACM Transactions on the Web  Volume 6, Issue 1
        March 2012
        109 pages
        ISSN:1559-1131
        EISSN:1559-114X
        DOI:10.1145/2109205
        Issue’s Table of Contents

        Copyright © 2012 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 1 March 2012
        • Accepted: 1 October 2011
        • Revised: 1 March 2011
        • Received: 1 March 2009
        Published in tweb Volume 6, Issue 1

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!