Abstract

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.

Keywords

Computer scienceParaphraseRanking (information retrieval)Leverage (statistics)Information retrievalPost hocTransformerQuestion answeringArtificial intelligenceNatural language processingMatching (statistics)

Related Publications

Publication Info

Year
2019
Type
preprint
Citations
145
Access
Closed

External Links

Social Impact

Social media, news, blog, policy document mentions

Citation Metrics

145
OpenAlex

Cite This

Yifan Qiao, Chenyan Xiong, Zhenghao Liu et al. (2019). Understanding the Behaviors of BERT in Ranking. arXiv (Cornell University) . https://doi.org/10.48550/arxiv.1904.07531

Identifiers

DOI
10.48550/arxiv.1904.07531