Precision-at-ten considered redundant

ABSTRACT

Information retrieval systems are compared using evaluation metrics, with researchers commonly reporting results for simple metrics such as precision-at-10 or reciprocal rank together with more complex ones such as average precision or discounted cumulative gain. In this paper, we demonstrate that complex metrics are as good as or better than simple metrics at predicting the performance of the simple metrics on other topics. Therefore, reporting of results from simple metrics alongside complex ones is redundant.

References

1. C. Buckley and E. Voorhees. Retrieval system evaluation. In E. Voorhees and D. Harman, editors, TREC: Experiment and Evaluation in Information Retrieval, chapter 3. The MIT Press, Cambridge, MA, 2005.
2. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In Proc. 23rd Ann. Int. ACM SIGIR Conf. on Res. and Dev. in Info. Retr., pages 33--40, Athens, Greece, 2000.
3. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20 (4): 422--446, 2002.
4. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst., to appear.
5. T. Sakai. The effect of topic sampling on sensitivity comparisons of information retrieval metrics. In Proc. 5th NTCIR Workshop Meeting on Information Access Technologies, pages 505--512, Tokyo, Japan, 2005.
6. M. Sanderson and J. Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proc. 28th Ann. Int. ACM SIGIR Conf. on Res. and Dev. in Info. Retr., pages 162--169, Salvador, Brazil, 2005.
7. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proc. 29th Ann. Int. ACM SIGIR Conf. on Res. and Dev. in Info. Retr., pages 11--18, Seattle, WA, 2006.
8. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In Proc. 25th Ann. Int. ACM SIGIR Conf. on Res. and Dev. in Info. Retr., pages 316--323, Tampere, Finland, 2002.

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results