skip to main content
research-article

ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval

Published:16 January 2023Publication History
Skip Abstract Section

Abstract

Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models, have shown the usefulness of expanding and reweighting the users’ initial queries using information occurring in an initial set of retrieved documents, known as the pseudo-relevant set. Recently, dense retrieval – through the use of neural contextual language models such as BERT for analysing the documents’ and queries’ contents and computing their relevance scores – has shown a promising performance on several information retrieval tasks still relying on the traditional inverted index for identifying documents relevant to a query. Two different dense retrieval families have emerged: the use of single embedded representations for each passage and query, e.g., using BERT’s [CLS] token, or via multiple representations, e.g., using an embedding for each token of the query and document (exemplified by ColBERT). In this work, we conduct the first study into the potential for multiple representation dense retrieval to be enhanced using pseudo-relevance feedback and present our proposed approach ColBERT-PRF. In particular, based on the pseudo-relevant set of documents identified using a first-pass dense retrieval, ColBERT-PRF extracts the representative feedback embeddings from the document embeddings of the pseudo-relevant set. Among the representative feedback embeddings, the embeddings that most highly discriminate among documents are employed as the expansion embeddings, which are then added to the original query representation. We show that these additional expansion embeddings both enhance the effectiveness of a reranking of the initial query results as well as an additional dense retrieval operation. Indeed, experiments on the MSMARCO passage ranking dataset show that MAP can be improved by up to 26% on the TREC 2019 query set and 10% on the TREC 2020 query set by the application of our proposed ColBERT-PRF method on a ColBERT dense retrieval approach.We further validate the effectiveness of our proposed pseudo-relevance feedback technique for a dense retrieval model on MSMARCO document ranking and TREC Robust04 document ranking tasks. For instance, ColBERT-PRF exhibits up to 21% and 14% improvement in MAP over the ColBERT E2E model on the MSMARCO document ranking TREC 2019 and TREC 2020 query sets, respectively. Additionally, we study the effectiveness of variants of the ColBERT-PRF model with different weighting methods. Finally, we show that ColBERT-PRF can be made more efficient, attaining up to 4.54× speedup over the default ColBERT-PRF model, and with little impact on effectiveness, through the application of approximate scoring and different clustering methods.

REFERENCES

  1. [1] Abdul-Jaleel Nasreen, Allan James, Croft W. Bruce, Diaz Fernando, Larkey Leah, Li Xiaoyan, Smucker Mark D., and Wade Courtney. 2004. UMass at TREC 2004: Novelty and HARD. In Proceedings of TREC.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Amati Giambattista. 2003. Probability Models for Information Retrieval Based on Divergence from Randomness Ph.D. thesis. University of Glasgow (2003).Google ScholarGoogle Scholar
  3. [3] Amati Giambattista, Carpineto Claudio, and Romano Giovanni. 2004. Query difficulty, robustness, and selective application of query expansion. In Proceedings of ECIR. 127137.Google ScholarGoogle ScholarCross RefCross Ref
  4. [4] Amati Gianni and Van Rijsbergen Cornelis Joost. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS) 20, 4 (2002), 357389.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Arthur David and Vassilvitskii Sergei. 2007. K-Means++: The advantages of careful seeding. In Proceedings of SODA. 10271035.Google ScholarGoogle Scholar
  6. [6] Cao Guihong, Nie Jian-Yun, Gao Jianfeng, and Robertson Stephen. 2008. Selecting good expansion terms for pseudo-relevance feedback. In Proceedings of SIGIR. 243250.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. [7] Craswell Nick, Mitra Bhaskar, Yilmaz Emine, and Campos Daniel. 2021. Overview of the TREC 2020 deep learning track. In Proceedings of TREC.Google ScholarGoogle Scholar
  8. [8] Craswell Nick, Mitra Bhaskar, Yilmaz Emine, Campos Daniel, and Voorhees Ellen M.. 2020. Overview of the TREC 2019 deep learning track. In Proceedings of TREC.Google ScholarGoogle Scholar
  9. [9] Dai Zhuyun and Callan Jamie. 2019. Deeper text understanding for IR with contextual neural language modeling. In Proceedings of SIGIR. 985988.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. [10] Dai Zhuyun and Callan Jamie. 2020. Context-aware document term weighting for ad-hoc search. In Proceedings of WWW. 18971907.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Devlin Jacob, Chang Ming-Wei, Lee Kenton, and Toutanova Kristina. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of ACL. 41714186.Google ScholarGoogle Scholar
  12. [12] Diaz Fernando, Mitra Bhaskar, and Craswell Nick. 2016. Query expansion with locally-trained word embeddings. In Proceedings of ACL. 367377.Google ScholarGoogle ScholarCross RefCross Ref
  13. [13] Formal Thibault, Piwowarski Benjamin, and Clinchant Stéphane. 2021. A white box analysis of ColBERT. In Proceedings of ECIR. 257263.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. [14] Guo Jiafeng, Fan Yixing, Ai Qingyao, and Croft W. Bruce. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of CIKM. 5564.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Johnson Jeff, Douze Matthijs, and Jégou Hervé. 2017. Billion-scale similarity search with GPUs. arXiv preprint arXiv:1702.08734 (2017).Google ScholarGoogle Scholar
  16. [16] Karpukhin Vladimir, Oguz Barlas, Min Sewon, Lewis Patrick, Wu Ledell, Edunov Sergey, Chen Danqi, and Yih Wen-tau. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP. 67696781.Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Khattab Omar and Zaharia Matei. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of SIGIR. 3948.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Khennak Ilyes, Drias Habiba, Kechid Amine, and Moulai Hadjer. 2019. Clustering algorithms for query expansion based information retrieval. In Proceedings of ICCI. 261272.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Kuzi Saar, Shtok Anna, and Kurland Oren. 2016. Query expansion using word embeddings. In Proceedings of CIKM. 19291932.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Li Canjia, Sun Yingfei, He Ben, Wang Le, Hui Kai, Yates Andrew, Sun Le, and Xu Jungang. 2018. NPRF: A neural pseudo relevance feedback framework for ad-hoc information retrieval. In Proceedings of EMNLP. 44824491.Google ScholarGoogle ScholarCross RefCross Ref
  21. [21] Li Canjia, Yates Andrew, MacAvaney Sean, He Ben, and Sun Yingfei. 2021. PARADE: Passage Representation Aggregation for Document Reranking. arXiv:2008.09093 [cs.IR].Google ScholarGoogle Scholar
  22. [22] Li Hang, Zhuang Shengyao, Mourad Ahmed, Ma Xueguang, Lin Jimmy, and Zuccon Guido. 2021. Improving query representations for dense retrieval with pseudo relevance feedback: A reproducibility study. In Proceedings of ECIR.Google ScholarGoogle Scholar
  23. [23] Luan Yi, Eisenstein Jacob, Toutanova Kristina, and Collins Michael. 2021. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics 9 (2021), 329345.Google ScholarGoogle Scholar
  24. [24] MacAvaney Sean. 2020. OpenNIR: A complete neural ad-hoc ranking pipeline. In Proceedings of WSDM. 845848.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. [25] Macdonald Craig and Tonellotto Nicola. 2020. Declarative experimentation in information retrieval using PyTerrier. In Proceedings of ICTIR. 161168.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Macdonald Craig and Tonellotto Nicola. 2021. On approximate nearest neighbour selection for multi-stage dense retrieval. In Proceedings of CIKM. 33183322.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Macdonald Craig, Tonellotto Nicola, MacAvaney Sean, and Ounis Iadh. 2021. PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval. In Proceedings of CIKM. 45264533.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. [28] Macdonald Craig, Tonellotto Nicola, and Ounis Iadh. 2021. On single and multiple representations in dense passage retrieval. IIR 2021 Workshop (2021).Google ScholarGoogle Scholar
  29. [29] Naseri Shahrzad, Dalton Jeffrey, Yates Andrew, and Allan James. 2021. CEQE: Contextualized embeddings for query expansion. Proceedings of ECIR (2021), 467482.Google ScholarGoogle Scholar
  30. [30] Nguyen Tri, Rosenberg Mir, Song Xia, Gao Jianfeng, Tiwary Saurabh, Majumder Rangan, and Deng Li. 2016. MS MARCO: A human generated machine reading comprehension dataset. In [email protected].Google ScholarGoogle Scholar
  31. [31] Nogueira Rodrigo, Jiang Zhiying, and Lin Jimmy. 2020. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020).Google ScholarGoogle Scholar
  32. [32] Nogueira Rodrigo, Lin Jimmy, and Epistemic AI. 2019. From doc2query to docTTTTTquery. Online preprint (2019).Google ScholarGoogle Scholar
  33. [33] Nogueira Rodrigo, Yang Wei, Lin Jimmy, and Cho Kyunghyun. 2019. Document expansion by query prediction. arXiv preprint arXiv:1904.08375 (2019).Google ScholarGoogle Scholar
  34. [34] Ounis Iadh, Amati Gianni, Plachouras Vassilis, He Ben, Macdonald Craig, and Johnson Douglas. 2005. Terrier information retrieval platform. In Proceedings of ECIR. 517519.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. [35] Padaki Ramith, Dai Zhuyun, and Callan Jamie. 2020. Rethinking query expansion for BERT reranking. In Proceedings of ECIR. 297304.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Peters Matthew E., Neumann Mark, Iyyer Mohit, Gardner Matt, Clark Christopher, Lee Kenton, and Zettlemoyer Luke. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT. 22272237.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Rocchio Joseph. 1971. Relevance feedback in information retrieval. The Smart Retrieval System-experiments in Automatic Document Processing (1971), 313323.Google ScholarGoogle Scholar
  38. [38] Roy Dwaipayan, Bhatia Sumit, and Mitra Mandar. 2019. Selecting discriminative terms for relevance model. In Proceedings of SIGIR. 12531256.Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. [39] Roy Dwaipayan, Ganguly Debasis, Bhatia Sumit, Bedathur Srikanta, and Mitra Mandar. 2018. Using word embeddings for information retrieval: How collection and term normalization choices affect performance. In Proceedings of CIKM. 18351838.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Roy Dwaipayan, Paul Debjyoti, Mitra Mandar, and Garain Utpal. 2016. Using word embeddings for automatic query expansion. In Proceedings of SIGIR Workshop on Neural Information Retrieval. arXiv:1606.07608.Google ScholarGoogle Scholar
  41. [41] Tonellotto Nicola and Macdonald Craig. 2021. Query embedding pruning for dense retrieval. In Proceedings of CIKM. 34533457.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Wang Junmei, Pan Min, He Tingting, Huang Xiang, Wang Xueyan, and Tu Xinhui. 2020. A pseudo-relevance feedback framework combining relevance matching and semantic matching for information retrieval. Information Processing & Management 57, 6 (2020), 102342.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Wang Xiao, Macdonald Craig, and Ounis Iadh. 2022. Improving zero-shot retrieval using dense external expansion. Information Processing & Management 59, 5 (2022), 103026.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. [44] Wang Xiao, Macdonald Craig, Tonellotto Nicola, and Ounis Iadh. 2021. Pseudo-relevance feedback for multiple representation dense retrieval. In Proceedings of ICTIR. 297306.Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. [45] Xiong Chenyan, Dai Zhuyun, Callan Jamie, Liu Zhiyuan, and Power Russell. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of SIGIR. 5564.Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Xiong Lee, Xiong Chenyan, Li Ye, Tang Kwok-Fung, Liu Jialin, Bennett Paul, Ahmed Junaid, and Overwijk Arnold. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of ICLR.Google ScholarGoogle Scholar
  47. [47] Yu HongChien, Dai Zhuyun, and Callan Jamie. 2021. PGT: Pseudo relevance feedback using a graph-based transformer. In Proceedings of ECIR. 440447.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Yu HongChien, Xiong Chenyan, and Callan Jamie. 2021. Improving query representations for dense retrieval with pseudo relevance feedback. In Proceedings of CIKM. 35923596.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. [49] Zamani Hamed and Croft W. Bruce. 2016. Embedding-based query language models. In Proceedings of ICTIR. 147156.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. [50] Zamani Hamed, Dehghani Mostafa, Croft W. Bruce, Learned-Miller Erik, and Kamps Jaap. 2018. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proceedings of CIKM. 497506.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Zheng Zhi, Hui Kai, He Ben, Han Xianpei, Sun Le, and Yates Andrew. 2020. BERT-QE: Contextualized query expansion for document re-ranking. In Proceedings of EMNLP: Findings. 47184728.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on the Web
      ACM Transactions on the Web  Volume 17, Issue 1
      February 2023
      189 pages
      ISSN:1559-1131
      EISSN:1559-114X
      DOI:10.1145/3575872
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 16 January 2023
      • Online AM: 22 November 2022
      • Accepted: 19 September 2022
      • Revised: 4 August 2022
      • Received: 22 February 2022
      Published in tweb Volume 17, Issue 1

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!