A new rank correlation coefficient for information retrieval

ABSTRACT

In the field of information retrieval, one is often faced with the problem of computing the correlation between two ranked lists. The most commonly used statistic that quantifies this correlation is Kendall's Τ. Often times, in the information retrieval community, discrepancies among those items having high rankings are more important than those among items having low rankings. The Kendall's Τ statistic, however, does not make such distinctions and equally penalizes errors both at high and low rankings.

In this paper, we propose a new rank correlation coefficient, AP correlation (Τap), that is based on average precision and has a probabilistic interpretation. We show that the proposed statistic gives more weight to the errors at high rankings and has nice mathematical properties which make it easy to interpret. We further validate the applicability of the statistic using experimental data.

References

1. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In O. Frieder, J. Hammer, S. Quershi, and L. Seligman, editors, Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 484--491. ACM Press, November 2003.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25--32, New York, NY, USA, 2004. ACM Press.
3. B. Carterette and J. Allan. Incremental test collections. In CIKM '05: Proceedings of the 14th ACM international conference on Information and knowledge management, pages 680--687, New York, NY, USA, 2005. ACM Press.
4. T. Cover and J. Thomas. Elements of Information Theory. Wiley, 1991.
5. R. Fagin, R. Kumar, and D. Sivakumar. Comparing top k lists. In SODA '03: Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, pages 28--36, Philadelphia, PA, USA, 2003. Society for Industrial and Applied Mathematics.
6. T. H. Haveliwala, A. Gionis, D. Klein, and P. Indyk. Evaluating strategies for similarity search on the web. In WWW '02: Proceedings of the 11th international conference on World Wide Web, pages 432--442, New York, NY, USA, 2002. ACM.
7. M. Kendall. A new measure of rank correlation. Biometrica, 30(1-2):81--89, 1938.
8. M. Melucci. On rank correlation in information retrieval evaluation. SIGIR Forum, 41(1):18--33, 2007.
9. M. Sanderson and H. Joho. Forming test collections with no system pooling. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40, New York, NY, USA, 2004. ACM.
10. M. Sanderson and I. Soboroff. Problems with kendall's tau. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 839--840, New York, NY, USA, 2007. ACM.
11. G. S. Shieh. A weighted kendall's tau statistic. Statistics & Probability Letters, 39:17--24, 1998.
12. I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66--73, New Orleans, Louisiana, USA, Sept. 2001. ACM Press, New York.
13. E. M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74--82. ACM Press, 2001.
14. E. M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2004.
15. D. D. Wackerly, W. Mendenhall, and R. L. Scheaffer. Mathematical Statistics with Applications. Duxbury Advanced Series, 2002.
16. S. Wu and F. Crestani. Methods for ranking information retrieval systems without relevance judgments. In SAC '03: Proceedings of the 2003 ACM symposium on Applied computing, pages 811--816, 2003.
17. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proceedings of the Fifteenth ACM International Conference on Information and Knowledge Management. ACM Press, November 2006.

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results