10.1145/2532508.2532512acmotherconferencesArticle/Chapter ViewAbstractPublication PagesrepsysConference Proceedingsconference-collections
research-article

Research paper recommender system evaluation: a quantitative literature survey

Online:12 October 2013Publication History

ABSTRACT

Over 80 approaches for academic literature recommendation exist today. The approaches were introduced and evaluated in more than 170 research articles, as well as patents, presentations and blogs. We reviewed these approaches and found most evaluations to contain major shortcomings. Of the approaches proposed, 21% were not evaluated. Among the evaluated approaches, 19% were not evaluated against a baseline. Of the user studies performed, 60% had 15 or fewer participants or did not report on the number of participants. Information on runtime and coverage was rarely provided. Due to these and several other shortcomings described in this paper, we conclude that it is currently not possible to determine which recommendation approaches for academic literature are the most promising. However, there is little value in the existence of more than 80 approaches if the best performing approaches are unknown.

References

  1. G. Shani and A. Gunawardana, "Evaluating recommendation systems," Recommender systems handbook, Springer, 2011, pp. 257--297.Google ScholarGoogle ScholarCross RefCross Ref
  2. D. N. Chin, "Empirical evaluation of user models and user-adapted systems," User modeling and user-adapted interaction, vol. 11, 2001, pp. 181--194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. P. Brusilovsky and E. Millán, "User models for adaptive hypermedia and adaptive educational systems," The adaptive web, 2007, pp. 3--53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Lam, D. Frankowski, and J. Riedl, "Do you trust your recommendations? An exploration of security and privacy issues in recommender systems," Emerging Trends in Information and Communication Security, 2006, pp. 14--29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. N. Good, J. B. Schafer, J. A. Konstan, A. Borchers, B. Sarwar, J. Herlocker, and J. Riedl, "Combining collaborative filtering with personal agents for better recommendations," Proceedings of the National Conference on Artificial Intelligence, JOHN WILEY & SONS LTD, 1999, pp. 439--446. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. D. Manning, P. Raghavan, and H. Schütze, An Introduction to Information Retrieval, Cambridge University Press, Cambridge, England, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. J. Beel, S. Langer, and M. Genzmehr, "Sponsored vs. Organic (Research Paper) Recommendations and the Impact of Labeling," Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), T. Aalberg, M. Dobreva, C. Papatheodorou, G. Tsakonas, and C. Farrugia, eds., Valletta, Malta: 2013, pp. 395--399.Google ScholarGoogle Scholar
  8. J. Beel, S. Langer, A. Nürnberger, and M. Genzmehr, "The Impact of Demographics (Age and Gender) and Other User Characteristics on Evaluating Recommender Systems," Proceedings of the 17th International Conference on Theory and Practice of Digital Libraries (TPDL 2013), T. Aalberg, M. Dobreva, C. Papatheodorou, G. Tsakonas, and C. Farrugia, eds., Valletta, Malta: Springer, 2013, pp. 400--404.Google ScholarGoogle Scholar
  9. M. Ge, C. Delgado-Battenfeld, and D. Jannach, "Beyond accuracy: evaluating recommender systems by coverage and serendipity," Proceedings of the fourth ACM conference on Recommender systems, ACM, 2010, pp. 257--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. R. Burke, "Hybrid recommender systems with case-based components," Advances in Case-Based Reasoning, Springer, 2004, pp. 91--105.Google ScholarGoogle ScholarCross RefCross Ref
  11. F. Ricci, L. Rokach, B. Shapira, and K. B. P., "Recommender systems handbook," Recommender Systems Handbook, 2011, pp. 1--35.Google ScholarGoogle Scholar
  12. A. Naak, "Papyres: un système de gestion et de recommandation d'articles de recherche," Université de Montréal, 2009.Google ScholarGoogle Scholar
  13. A. Naak, H. Hage, and E. Aımeur, "Papyres: A Research Paper Management System," Proceedings of the 10th E-Commerce Technology Conference on Enterprise Computing, E-Commerce and E-Services, IEEE, 2008, pp. 201--208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. C. Bancu, M. Dagadita, M. Dascalu, C. Dobre, S. Trausan-Matu, and A. M. F. Florea, "ARSYS-Article Recommender System," Proceedings of the 14th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, 2012, pp. 349--355. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. K. D. Bollacker, S. Lawrence, and C. L. Giles, "CiteSeer: An autonomous web agent for automatic retrieval and identification of interesting publications," Proceedings of the 2nd international conference on Autonomous agents, ACM, 1998, pp. 116--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. S. C. Cazella and L. O. C. Alvares, "Combining Data Mining Technique and Users' Relevance Opinion to Build an Efficient Recommender System," Revista Tecnologia da Informação, UCB, vol. 4, 2005.Google ScholarGoogle Scholar
  17. P. Chirawatkul, "Structured Peer-to-Peer Search to build a Bibliographic Paper Recommendation System," Saarland University, 2006.Google ScholarGoogle Scholar
  18. L. Fernández, J. A. Sánchez, and A. Garcıa, "Mibiblio: personal spaces in a digital library universe," Proceedings of the fifth ACM conference on Digital libraries, ACM, 2000, pp. 232--233. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. G. Geisler, D. McArthur, and S. Giersch, "Developing recommendation services for a digital library with uncertain and changing data," Proceedings of the 1st ACM/IEEE-CS joint conference on Digital libraries, ACM, 2001, pp. 199--200. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. B. Gipp, J. Beel, and C. Hentschel, "Scienstein: A Research Paper Recommender System," Proceedings of the International Conference on Emerging Trends in Computing (ICETiC'09), Virudhunagar (India): IEEE, 2009, pp. 309--315.Google ScholarGoogle Scholar
  21. A. Nakagawa and T. Ito, "An implementation of a knowledge recommendation system based on similarity among users' profiles," Proceedings of the 41st SICE Annual Conference, IEEE, 2002, pp. 326--327.Google ScholarGoogle Scholar
  22. H.-E. Popa, V. Negru, D. Pop, and I. Muscalagiu, "DL-AgentRecom-A multi-agent based recommendation system for scientific documents," Proceedings of the 10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, IEEE, 2008, pp. 320--324. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. L. M. Rocha, "TalkMine: a soft computing approach to adaptive knowledge recommendation," Studies in fuzziness and soft computing, vol. 75, 2001, pp. 89--116. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. J. Vassileva, "Supporting peer-to-peer user communities," Proceedings of the Conference On the Move to Meaningful Internet Systems, Springer, 2002, pp. 230--247. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. A. S. Vivacqua, J. Oliveira, and J. M. de Souza, "i-ProSE: inferring user profiles in a scientific context," The Computer Journal, vol. 52, 2009, pp. 789--798. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Q. Yang, S. Zhang, and B. Feng, "Research on Personalized Recommendation System of Scientific and Technological Periodical Based on Automatic Summarization," Proceedings of the 1st International Symposium on Information Technologies and Applications in Education, IEEE, 2007, pp. 34--39.Google ScholarGoogle Scholar
  27. C. Hess, Trust-based recommendations in multi-layer networks, IOS Press, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. N. F. Matsatsinis, K. Lakiotaki, and P. Delia, "A system based on multiple criteria analysis for scientific paper recommendation," Proceedings of the 11th Panhellenic Conference on Informatics, 2007, pp. 135--149.Google ScholarGoogle Scholar
  29. S. Watanabe, T. Ito, T. Ozono, and T. Shintani, "A paper recommendation mechanism for the research support system papits," Proceedings of the International Workshop on Data Engineering Issues in E-Commerce, IEEE, 2005, pp. 71--80. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. S.-S. Weng and H.-L. Chang, "Using ontology network analysis for research document recommendation," Expert Systems with Applications, vol. 34, 2008, pp. 1857--1869. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. S.-Y. Yang and C.-L. Hsu, "A New Ontology-Supported and Hybrid Recommending Information System for Scholars," Proceedings of the 13th International Conference on Network-Based Information Systems (NBiS), IEEE, 2010, pp. 379--384. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. H. Avancini, L. Candela, and U. Straccia, "Recommenders in a personalized, collaborative digital library environment," Journal of Intelligent Information Systems, vol. 28, 2007, pp. 253--283. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. N. Agarwal, E. Haque, H. Liu, and L. Parsons, "A subspace clustering framework for research group collaboration," International Journal of Information Technology and Web Engineering, vol. 1, 2006, pp. 35--58.Google ScholarGoogle ScholarCross RefCross Ref
  34. A. Arnold and W. W. Cohen, "Information extraction as link prediction: Using curated citation networks to improve gene detection," Proceedings of the 4th International Conference on Wireless Algorithms, Systems, and Applications, Springer, 2009, pp. 541--550. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. M. Baez, D. Mirylenka, and C. Parra, "Understanding and supporting search for scholarly knowledge," Proceeding of the 7th European Computer Science Summit, 2011, pp. 1--8.Google ScholarGoogle Scholar
  36. S. Bethard and D. Jurafsky, "Who should I cite: learning literature search models from citation behavior," Proceedings of the 19th ACM international conference on Information and knowledge management, ACM, 2010, pp. 609--618. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. F. Ferrara, N. Pudota, and C. Tasso, "A Keyphrase-Based Paper Recommender System," Proceedins of the IRCDL'11, Springer, 2011, pp. 14--25.Google ScholarGoogle Scholar
  38. M. Gori and A. Pucci, "Research paper recommender systems: A random-walk based approach," Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence, IEEE Computer Society Washington, DC, USA, 2006, pp. 778--781. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. S.-Y. Hwang, W.-C. Hsiung, and W.-S. Yang, "A prototype WWW literature recommendation system for digital libraries," Online Information Review, vol. 27, 2003, pp. 169--182.Google ScholarGoogle ScholarCross RefCross Ref
  40. Y. Huang, "Combining Social Networks and Content for Recommendation in a Literature Digital Library," National Sun Yat-Sen University, Taiwan, 2007.Google ScholarGoogle Scholar
  41. K. Jack, "Mahout Becomes a Researcher: Large Scale Recommendations at Mendeley," Presentation at Big Data Week Conferences, 2012.Google ScholarGoogle Scholar
  42. O. Küçüktunç, E. Saule, K. Kaya, and Ü.V. Çatalyürek, "Recommendation on Academic Networks using Direction Aware Citation Analysis," arXiv preprint arXiv:1205.1143, 2012, pp. 1--10.Google ScholarGoogle Scholar
  43. J. Lin and W. J. Wilbur, "PubMed Related Articles: a Probabilistic Topic-based Model for Content Similarity," BMC Bioinformatics, vol. 8, 2007, pp. 423--436.Google ScholarGoogle ScholarCross RefCross Ref
  44. G. H. Martın, S. Schockaert, C. Cornelis, and H. Naessens, "Metadata impact on research paper similarity," 14th European Conference on Digital Libraries, Springer, 2010, pp. 457--460. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. S. Pohl, F. Radlinski, and T. Joachims, "Recommending Related Papers Based on Digital Library Access Records," Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, Vancouver, BC, Canada: ACM, 2007, pp. 417--418. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. T. Strohman, W. B. Croft, and D. Jensen, "Recommending citations for academic papers," Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, 2007, pp. 705--706. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. K. Sugiyama and M.-Y. Kan, "Scholarly paper recommendation via user's recent research interests," Proceedings of the 10th annual joint conference on Digital libraries, ACM, 2010, pp. 29--38. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. H. Wu, Y. Hua, B. Li, and Y. Pei, "Enhancing citation recommendation with various evidences," Proceedings of the 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), IEEE, 2012, pp. 1160--1165.Google ScholarGoogle Scholar
  49. H. Xia, J. Li, J. Tang, and M.-F. Moens, "Plink-LDA: using link as prior information in topic modeling," Proceedings of the Conference on Database Systems for Advanced Applications (DASFAA), Springer, 2012, pp. 213--227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. P. Yin, M. Zhang, and X. Li, "Recommending scientific literatures in a collaborative tagging environment," Proceedings of the 10th international conference on Asian digital libraries, Springer, 2007, pp. 478--481. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. F. Zarrinkalam and M. Kahani, "SemCiR" Program: electronic library and information systems, vol. 47, 2013, pp. 92--112.Google ScholarGoogle Scholar
  52. F. Zarrinkalam and M. Kahani, "A multi-criteria hybrid citation recommendation system based on linked data," Proceedings of the 2nd International eConference on Computer and Knowledge Engineering, IEEE, 2012, pp. 283--288.Google ScholarGoogle ScholarCross RefCross Ref
  53. J. Beel, S. Langer, M. Genzmehr, and A. Nürnberger, "Introducing Docear's Research Paper Recommender System," Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL'13), ACM, 2013, pp. 459--460. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. T. Bogers and A. van den Bosch, "Recommending scientific articles using citeulike," Proceedings of the 2008 ACM conference on Recommender systems, ACM New York, NY, USA 2008, pp. 287--290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. M. Mönnich and M. Spiering, "Adding value to the library catalog by implementing a recommendation system," D-Lib Magazine, vol. 14, 2008, pp. 4--11.Google ScholarGoogle ScholarCross RefCross Ref
  56. S. E. Middleton, N. R. Shadbolt, and D. C. De Roure, "Ontological user profiling in recommender systems," ACM Transactions on Information Systems (TOIS), vol. 22, 2004, pp. 54--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Q. He, D. Kifer, J. Pei, P. Mitra, and C. L. Giles, "Citation recommendation without author supervision," Proceedings of the fourth ACM international conference on Web search and data mining, ACM, 2011, pp. 755--764. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Q. He, J. Pei, D. Kifer, P. Mitra, and L. Giles, "Context-aware citation recommendation," Proceedings of the 19th international conference on World wide web, ACM, 2010, pp. 421--430. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. L. Rokach, P. Mitra, S. Kataria, W. Huang, and L. Giles, "A Supervised Learning Method for Context-Aware Citation Recommendation in a Large Corpus," Proceedings of the Large-Scale and Disributed Systems for Information Retrieval Workshop (LSDS-IR), 2013, pp. 17--22.Google ScholarGoogle Scholar
  60. G. Adomavicius and A. Tuzhilin, "Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions," Knowledge and Data Engineering, IEEE Transactions on, vol. 17, 2005, pp. 734--749. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. A. Gunawardana and G. Shani, "A survey of accuracy evaluation metrics of recommendation tasks," The Journal of Machine Learning Research, vol. 10, 2009, pp. 2935--2962. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. J. L. Herlocker, J. A. Konstan, L. G Terveen, and J. T. Riedl, "Evaluating collaborative filtering recommender systems," ACM Transactions on Information Systems (TOIS), vol. 22, 2004, pp. 5--53. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. W. Hersh, A. Turpin, S. Price, B. Chan, D. Kramer, L. Sacherek and D. Olson, "Do batch and user evaluations give the same results?," Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, ACM, 2000, pp. 17--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. D. Jannach, L. Lerche, F. Gedikli, and G. Bonnin, "What Recommenders Recommend--An Analysis of Accuracy, Popularity, and Sales Diversity Effects," User Modeling, Adaptation, and Personalization, Springer, 2013, pp. 25--37.Google ScholarGoogle ScholarCross RefCross Ref
  65. A. H. Turpin and W. Hersh, "Why batch and user evaluations do not give the same results," Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, 2001, pp. 225--231. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. P. Jomsri, S. Sanguansintukul, and W. Choochaiwattana, "A framework for tag-based research paper recommender system: an IR approach," Proceedings of the 24th International Conference on Advanced Information Networking and Applications (WAINA), IEEE, 2010, pp. 103--108. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. A. Woodruff, R. Gossweiler, J. Pitkow, E. H. Chi, and S. K Card, "Enhancing a digital book with a reading recommender," Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, 2000, pp. 153--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. J. Bollen and H. Van de Sompel, "An architecture for the aggregation and analysis of scholarly usage data," Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, ACM, 2006, pp. 298--307. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. B. Gipp and J. Beel, "Citation Proximity Analysis (CPA) - A new approach for identifying related work based on Co-Citation Analysis," Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI'09), B. Larsen and J. Leta, eds., Rio de Janeiro (Brazil): International Society for Scientometrics and Informetrics, 2009, pp. 571--575.Google ScholarGoogle Scholar
  70. K. Uchiyama, H. Nanba, A. Aizawa, and T. Sagara, "OSUSUME: cross-lingual recommender system for research papers," Proceedings of the 2011 Workshop on Context-awareness in Retrieval and Recommendation, ACM, 2011, pp. 39--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. Y. Wang, E. Zhai, J. Hu, and Z. Chen, "Claper: Recommend classical papers to beginners," Seventh International Conference on Fuzzy Systems and Knowledge Discovery, IEEE, 2010, pp. 2777--2781.Google ScholarGoogle Scholar
  72. U. Farooq, C. H. Ganoe, J. M. Carroll, I. G. Councill, and C. Lee Giles, "Design and evaluation of awareness mechanisms in CiteSeer," Information Processing & Management, vol. 44, 2008, pp. 596--612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. S. Kang and Y. Cho, "A novel personalized paper search system," Proceedings of the International Conference on Intelligent Computing, Springer, 2006, pp. 1257--1262. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. A. Naak, H. Hage, and E. Aımeur, "A multi-criteria collaborative filtering approach for research paper recommendation in papyres," Proceedings of the 4th International Conference MCETECH, Springer, 2009, pp. 25--39.Google ScholarGoogle Scholar
  75. S. Huang, G. R. Xue, B. Y. Zhang, Z. Chen, Y. Yu, and W. Y. Ma, "Tssp: A reinforcement algorithm to find related papers," Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (WI), IEEE, 2004, pp. 117--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Y. Jiang, A. Jia, Y. Feng, and D. Zhao, "Recommending academic papers via users' reading purposes," Proceedings of the sixth ACM conference on Recommender systems, ACM, 2012, pp. 241--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  77. K. Hong, H. Jeon, and C. Jeon, "UserProfile-based personalized research paper recommendation system," Proceedings of the 8th International Conference on Computing and Networking Technology, IEEE, 2012, pp. 134--138.Google ScholarGoogle Scholar
  78. J. He, J.-Y. Nie, Y. Lu, and W. X. Zhao, "Position-Aligned translation model for citation recommendation," Proceedings of the 19th international conference on String Processing and Information Retrieval, Springer, 2012, pp. 251--263. Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. A. Kodakateri Pudhiyaveetil, S. Gauch, H. Luong, and J. Eno, "Conceptual recommender system for CiteSeerX," Proceedings of the third ACM conference on Recommender systems, ACM, 2009, pp. 241--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  80. Y. Lu, J. He, D. Shan, and H. Yan, "Recommending citations with translation model," Proceedings of the 20th ACM international conference on Information and knowledge management, ACM, 2011, pp. 2017--2020. Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. J. M. Morales-del-Castillo, E. Peis, and E. Herrera-Viedma, "A filtering and recommender system prototype for scholarly users of digital libraries," Proceedings of the Second World Summit on the Knowledge Society, Springer, 2009, pp. 108--117. Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. G. Mishra, "Optimised Research Paper Recommender System Using Social Tagging," International Journal of Engineering Research and Applications, vol. 2, 2012, pp. 1503--1507.Google ScholarGoogle Scholar
  83. C. Pan and W. Li, "Research paper recommendation with topic analysis," Proceedings of the International Conference on Computer Design and Applications (ICCDA), IEEE, 2010, pp. 264--268.Google ScholarGoogle Scholar
  84. K. Stock, A. Robertson, F. Reitsma, T. Stojanovic, M. Bishr, D. Medyckyj-Scott, and J. Ortmann, "eScience for Sea Science: A Semantic Scientific Knowledge Infrastructure for Marine Scientists," Proceedings of the 5th IEEE International Conference on e-Science, IEEE, 2009, pp. 110--117. Google ScholarGoogle ScholarDigital LibraryDigital Library
  85. K. Stock, V. Karasova, A. Robertson, G. Roger, M. Small, M. Bishr, J. Ortmann, T. Stojanovic, F. Reitsma, L. Korczynski, B. Brodaric, and Z. Gardner, "Finding Science with Science: Evaluating a Domain and Scientific Ontology User Interface for the Discovery of Scientific Resources," Transactions in GIS, vol. 1, 2013, pp. 1--28.Google ScholarGoogle Scholar
  86. M. Zhang, W. Wang, and X. Li, "A Paper Recommender for Scientific Literatures Based on Semantic Concept Similarity," Proceedings of the International Conference on Asian Digital Libraries, 2008, pp. 359--362. Google ScholarGoogle ScholarDigital LibraryDigital Library
  87. Z. Zhang and L. Li, "A research paper recommender system based on spreading activation model," Proceedings of the 2nd International Conference on Information Science and Engineering (ICISE), IEEE, 2010, pp. 928--931.Google ScholarGoogle Scholar
  88. E. Erosheva, S. Fienberg, and J. Lafferty, "Mixed-membership models of scientific publications," Proceedings of the National Academy of Sciences of the United States of America, National Acad Sciences, 2004, pp. 5220--5227.Google ScholarGoogle ScholarCross RefCross Ref
  89. N. Ratprasartporn and G. Ozsoyoglu, "Finding related papers in literature digital libraries," Proceedings of the 11th European Conference on Digital Libraries, Springer, 2007, pp. 271--284. Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. A. Vellino, "A comparison between usage-based and citation-based methods for recommending scholarly research articles," Proceedings of the American Society for Information Science and Technology, Wiley Online Library, 2010, pp. 1--2. Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. C. Yang, B. Wei, J. Wu, Y. Zhang, and L. Zhang, "CARES: a ranking-oriented CADAL recommender system," Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, ACM, 2009, pp. 203--212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  92. R. Dong, L. Tokarchuk, and A. Ma, "Digging Friendship: Paper Recommendation in Social Network," Proceedings of Networking & Electronic Commerce Research Conference (NAEC 2009), 2009, pp. 21--28.Google ScholarGoogle Scholar
  93. A. Daud and A. H. Muhammad Akramand Rajpar Shaikh, "Scientific Reference Mining using Semantic Information through Topic Modeling," Research Journal of Engineering & Technology, vol. 28, 2009, pp. 253--262.Google ScholarGoogle Scholar
  94. C. Caragea, A. Silvescu, P. Mitra, and C. L. Giles, "Can't See the Forest for the Trees? A Citation Recommendation System," iConference 2013 Proceedings, 2013, pp. 849--851.Google ScholarGoogle Scholar
  95. N. Lao and W. W. Cohen, "Relational retrieval using a combination of path-constrained random walks," Machine learning, vol. 81, 2010, pp. 53--67. Google ScholarGoogle ScholarDigital LibraryDigital Library
  96. D. Zhou, S. Zhu, K. Yu, X. Song, B. L. Tseng, H. Zha, and C. L. Giles, "Learning multiple graphs for document recommendations," Proceedings of the 17th international conference on World Wide Web, ACM, 2008, pp. 141--150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  97. R. M. Nallapati, A. Ahmed, E. P. Xing, and W. W. Cohen, "Joint latent topic models for text and citations," Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, 2008, pp. 542--550. Google ScholarGoogle ScholarDigital LibraryDigital Library
  98. S. Kataria, P. Mitra, and S. Bhatia, "Utilizing context in generative bayesian models for linked corpus," Proceedings of the 24th AAAI Conference on Artificial Intelligence, 2010, pp. 1340--1345.Google ScholarGoogle Scholar
  99. D. M. Pennock, E. Horvitz, S. Lawrence, and C. L. Giles, "Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach," Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence, Morgan Kaufmann Publishers Inc., 2000, pp. 473--480. Google ScholarGoogle ScholarDigital LibraryDigital Library
  100. W. Huang, S. Kataria, C. Caragea, P. Mitra, C. L. Giles, and L. Rokach, "Recommending citations: translating papers into references," Proceedings of the 21st ACM international conference on Information and knowledge management, ACM, 2012, pp. 1910--1914. Google ScholarGoogle ScholarDigital LibraryDigital Library
  101. G. Karypis, "Evaluation of item-based top-n recommendation algorithms," Proceedings of the tenth international conference on Information and knowledge management, ACM, 2001, pp. 247--254. Google ScholarGoogle ScholarDigital LibraryDigital Library
  102. X. Tang and Q. Zeng, "Keyword clustering for user interest profiling refinement within paper recommender systems," Journal of Systems and Software, vol. 85, 2012, pp. 87--101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  103. Y. Liang, Q. Li, and T. Qian, "Finding relevant papers based on citation relations," Proceedings of the 12th international conference on Web-age information management, Springer, 2011, pp. 403--414. Google ScholarGoogle ScholarDigital LibraryDigital Library
  104. Z. Huang, W. Chung, T. H. Ong, and H. Chen, "A graph-based recommender system for digital library," Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, ACM, 2002, pp. 65--73. Google ScholarGoogle ScholarDigital LibraryDigital Library
  105. M. D. Ekstrand, P. Kannan, J. A. Stemper, J. T. Butler, J. A. Konstan, and J. T. Riedl, "Automatically building research reading lists," Proceedings of the fourth ACM conference on Recommender systems, ACM, 2010, pp. 159--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  106. S. M. McNee, I. Albert, D. Cosley, P. Gopalkrishnan, S. K. Lam, A. M. Rashid, J. A. Konstan, and J. Riedl, "On the Recommending of Citations for Research Papers," Proceedings of the ACM Conference on Computer Supported Cooperative Work, New Orleans, Louisiana, USA: ACM, 2002, pp. 116--125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  107. R. Torres, S. M. McNee, M. Abel, J. A. Konstan, and J. Riedl, "Enhancing digital libraries with TechLens," Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, ACM New York, NY, USA, 2004, pp. 228--236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  108. S. M. McNee, N. Kapoor, and J. A. Konstan, "Don't look stupid: avoiding pitfalls when recommending research papers," Proceedings of the 20th anniversary conference on Computer supported cooperative work, ProQuest, 2006, pp. 171--180. Google ScholarGoogle ScholarDigital LibraryDigital Library
  109. K. Jack, "Mendeley: Recommendation Systems for Academic Literature," Presentation at Technical University of Graz (TUG), 2012.Google ScholarGoogle Scholar
  110. C. Nascimento, A. H. Laender, A. S. da Silva, and M. A. Gonçalves, "A source independent framework for research paper recommendation," Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, ACM, 2011, pp. 297--306. Google ScholarGoogle ScholarDigital LibraryDigital Library
  111. J. Beel, S. Langer, M. Genzmehr, B. Gipp, and A. Nürnberger, "A Comparative Analysis of Offline and Online Evaluations and Discussion of Research Paper Recommender System Evaluation," Proceedings of the Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys) at the ACM Recommender System Conference (RecSys), 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Research paper recommender system evaluation: a quantitative literature survey

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!