The Treatment of Ties in AP Correlation

ABSTRACT

The Kendall tau and AP correlation coefficients are very commonly use to compare two rankings over the same set of items. Even though Kendall tau was originally defined assuming that there are no ties in the rankings, two alternative versions were soon developed to account for ties in two different scenarios: measure the accuracy of an observer with respect to a true and objective ranking, and measure the agreement between two observers in the absence of a true ranking. These two variants prove useful in cases where ties are possible in either ranking, and may indeed result in very different scores. AP correlation was devised to incorporate a top-heaviness component into Kendall tau, penalizing more heavily if differences occur between items at the top of the rankings, making it a very compelling coefficient in Information Retrieval settings. However, the treatment of ties in AP correlation remains an open problem. In this paper we fill this gap, providing closed analytical formulations of AP correlation under the two scenarios of ties contemplated in Kendall tau. In addition, we developed an R package that implements these coefficients.

References

1. Ricardo Baeza-Yates, Carlos Castillo, Mauricio Marin, and Andrea Rodriguez 2005. Crawling a country: better strategies than breadth-first for web page ordering ACM WWW. 864--872.
2. Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, and Emine Yilmaz. 2008. Relevance Assessment: Are Judges Exchangeable and Does it Matter? ACM SIGIR. 667--674.
3. Ben Carterette, Virgil Pavlu, Hui Fang, and Evangelos Kanoulas 2009. Million Query Track 2009 Overview. In TREC.
4. H. E. Daniels. 1944. The Relation between Measures of Correlation in the Universe of Sample Permutations. Biometrika, Vol. 33, 2 (1944), 129--135.
5. Maurice G. Kendall. 1938. A New Measure of Rank Correlation. Biometrika, Vol. 30, 1 (1938), 81--93.
6. Maurice G. Kendall. 1945. The Treatment of Ties in Ranking Problems. Biometrika, Vol. 33, 3 (1945), 239--251.
7. Maurice G. Kendall. 1948. Rank Correlation Methods (4th ed.). Charles Griffin & Company Limited.
8. Karl Pearson. 1907. On Further Methods of Determining Correlation. Technical Report.
9. Tetsuya Sakai. 2007. On the Reliability of Information Retrieval Metrics Based on Graded Relevance. Information Processing and Management Vol. 43, 2 (2007), 531--548.
10. Mark Sanderson, Monica Lestari Paramita, Paul Clough, and Evangelos Kanoulas 2010. Do User Preferences and Evaluation Measures Line Up? ACM SIGIR. 555--562.
11. Mark D. Smucker, Gabriella Kazai, and Matthew Lease. 2013. Overview of the TREC 2013 Crowdsourcing Track. TREC.
12. Student 1921. An Experimental Determination of the Probable Error of Dr. Spearman's Correlation Coefficients. Biometrika, Vol. 13, 2/3 (1921), 263--282.
13. Julián Urbano and Mónica Marrero 2016. Toward Estimating the Rank Correlation between the Test Collection Results and the True System Performance. In ACM SIGIR. 1033--1036.
14. Ellen M. Voorhees. 1998. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness ACM SIGIR. 315--323.
15. Ryen W White, Ian Ruthven, Joemon M Jose, and CJ Van Rijsbergen 2005. Evaluating implicit feedback models using searcher simulations. ACM TOIS, Vol. 23, 3 (2005), 325--361.
16. Max A. Woodbury. 1940. Rank Correlation When There are Equal Variates. Annals of Mathematical Statistics Vol. 11, 3 (1940), 358--362.
17. Emine Yilmaz, Javed A. Aslam, and Stephen Robertson. 2008. A New Rank Correlation Coefficient for Information Retrieval ACM SIGIR. 587--594.

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

    • Retrieval models and ranking