💬 NLP

PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking

This paper presents PASH's participation in the TREC 2021 Deep Learning Track, detailing a multi-stage ranking framework that integrates sparse and dense retrieval, continual pre-training for point-wise and pair-wise ranking, and the novel incorporation of the generative T5 model to improve performance over the previous year.

Original authors: Yixuan Qiao, Shanshan Zhao, Jun Wang, Hao Chen, Tuozhen Liu, Xianbin Ye, Xin Tang, Rui Fang, Peng Gao, Wenfeng Xie, Guotong Xie

Published 2026-02-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yixuan Qiao, Shanshan Zhao, Jun Wang, Hao Chen, Tuozhen Liu, Xianbin Ye, Xin Tang, Rui Fang, Peng Gao, Wenfeng Xie, Guotong Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive library with millions of books (documents) and billions of pages (passages). Every day, thousands of people walk in asking very specific questions. Your job is to find the exact page or book that answers their question and put it right at the top of the list.

This paper describes how a team called PASH entered a high-stakes competition (TREC 2021) to see who could build the best "librarian" for this job. Here is how they did it, explained simply:

1. The Two-Step Search Strategy (Recall Stage)

Before the librarian can decide which answer is the best, they first need to gather a big pile of possible answers. PASH used a "two-pronged" approach to gather these candidates:

The Keyword Hunter (Sparse Retrieval): Think of this as a librarian who only looks at the exact words on the page. If you ask for "apple," they find pages with the word "apple." To make this smarter, they used a robot (T5) that reads a page and writes down a list of questions that page might answer. This helps the keyword hunter find pages even if the user didn't use the exact right words.
The Concept Reader (Dense Retrieval): This is a librarian who understands the meaning behind the words. Even if you ask about "fruit" and the page says "red delicious," this librarian knows they match. PASH used a system called ColBERT for this, which is great at understanding context without getting bogged down by speed issues.

By combining these two, they cast a wide net to make sure they didn't miss any good answers.

2. The Sorting Lineup (Multi-Stage Ranking)

Once they had a big pile of potential answers, they needed to sort them from "best" to "worst." They did this in two rounds:

Round 1 (The Point-by-Point Check): They looked at each answer individually and asked, "How good is this answer on its own?" They used powerful AI models (like BERT and XLNet) that had been trained on general knowledge and then fine-tuned on the specific library data.
Round 2 (The Head-to-Head Match): This was a new trick for them. Instead of just grading each answer alone, they started comparing answers directly against each other. "Is Answer A better than Answer B?" This allowed them to use much more data to train their sorter, making it much sharper.

3. The New Super-Tool (The Generative Model)

The biggest change from last year's competition was introducing a new, massive AI model called T5.

The Analogy: Imagine the previous models were like expert students who memorized the library catalog. The new T5 model is like a genius who can actually write a summary of the answer from scratch.
The Result: This "generative" model was so powerful that it could often find the perfect answer in the very top 5 results (Top 5) without needing to look further. However, the paper notes that while it was great at finding the very best answer quickly, it wasn't as perfect at sorting the rest of the list (Top 10) compared to the older models.

4. The Teamwork (Ensemble)

Finally, instead of relying on just one librarian, PASH trained several different versions of their sorting system with slight variations (like giving them different random starting points). They then combined the opinions of all these librarians to make the final decision. This "team vote" usually leads to a more accurate result than any single librarian could achieve alone.

The Bottom Line

The team found that mixing the "keyword hunter" with the "concept reader" worked very well. They also discovered that using the new "generative" T5 model helped them find the absolute best answers very quickly, but they realized that for the future, they need to teach these generative models how to sort the entire list better, not just pick the top winner.

In short: They built a super-librarian that uses two different search methods to find candidates, then uses a team of AI models (including a new "writer" AI) to vote on the best answer, resulting in a very high score in the competition.

Technical Summary: PASH at TREC 2021 Deep Learning Track

Problem Statement
This paper details the participation of the PASH team in the TREC 2021 Deep Learning Track, which focuses on Information Retrieval (IR) tasks involving large-scale training datasets. The track presented two specific challenges: Document Ranking and Passage Ranking. The objective was to retrieve and rank relevant content from massive collections (138 million passages and ~12 million documents) using a set of 477 test queries. The primary evaluation metric was Normalized Discounted Cumulative Gain (NDCG). The team aimed to improve upon their TREC 2020 performance by integrating generative models and refining multi-stage ranking strategies.

Methodology
The PASH team employed a multi-stage pipeline comprising retrieval, multi-way matching, and a hierarchical ranking process.

Multi-way Matching (Retrieval Stage):
- Sparse Retrieval: The team utilized docTTTTTquery (also known as docT5query or doc2query-T5) to generate pseudo-queries for documents and passages.
  - For passages, T5-base was used to sample 40 queries per passage.
  - For documents, a sliding window approach (10 sentences, stride of 5) segmented documents into passages. Each segment was prepended with the document's URL and title. Ten queries were generated per passage.
  - These generated queries were indexed using BM25.
- Dense Retrieval: To complement sparse methods, the team adopted ColBERT as their dense retriever, selected for its balance between effectiveness and efficiency.
Multi-stage Ranking:
- Foundation Models: The ranking pipeline inherited models from TREC 2020, including BERT-Large, ALBERT-XXLarge, ELECTRA-Large, and XLNet-Large. These models underwent continual pre-training on general knowledge and document-level data, followed by fine-tuning on the current year's dataset.
- Strategy Shift: Unlike the previous year, the second stage of ranking utilized a pair-wise loss objective. This allowed the team to leverage a broader range of training data beyond the limited 4-class TREC labeled data.
- Generative Enhancement (T5): A key addition for 2021 was the introduction of the generative model T5 to enhance performance.
  - Implementation: Based on the Megatron-LM repository with training strategies aligned with prior work [10].
  - Scale: The team trained T5-3B (approx. 12 days on 32 V100 GPUs, ~12.8M examples) and T5-11B (approx. 30 days).
  - Performance Note: The T5-11B model demonstrated a zero-shot NDCG@5 of 0.3651.
Ensemble Learning:
- Benefiting from the multi-way matching approach, the team sequentially trained different re-rankers using various random seeds to create an ensemble for final predictions.

Results
The team submitted six official runs for each sub-task (passage and document ranking), categorized as re-ranking runs ( $r^*$ ) and full-ranking runs ( $f^*$ ).

Passage Ranking:
- In the re-ranking task, the best run (pash r2/r3) achieved an NDCG@5 of 0.7390 and NDCG@10 of 0.7076.
- In the full-ranking task, the best run (pash f3) achieved an NDCG@5 of 0.7596 and NDCG@10 of 0.7494, with a Recall@100 (R@100) of 0.5500.
Document Ranking:
- In the re-ranking task, the best run (pash doc r3) achieved an NDCG@5 of 0.7383 and NDCG@10 of 0.7164.
- In the full-ranking task, the best run (pash doc f5) achieved an NDCG@5 of 0.7508 and NDCG@10 of 0.7368, with an R@100 of 0.4419.

Key Contributions and Significance
The paper highlights several specific contributions and observations:

Integration of Generative Models: The introduction of T5 marked a significant step forward compared to TREC 2020. The authors observed that while T5-based models achieved higher NDCG@5 scores, they exhibited lower NDCG@10 scores compared to transformer-based models.
Interpretation of Results: The authors interpret the high NDCG@5 but lower NDCG@10 in generative models as an indication that relevant passages are successfully ranked within the top 5. However, this suggests a specific "thirst" for multi-stage ranking within generative frameworks. The authors posit that pair-wise or list-wise training strategies could further unlock the potential of these models.
Hybrid Retrieval Success: The experiments demonstrated that the expressiveness of ColBERT integrates well with the traditional BM25 + docTTTTTquery approach, validating the multi-way matching strategy.
Future Direction: The paper concludes that the "interweaving sequence of different methods" (combining sparse, dense, and generative approaches) will be a priority for future work in the field.

The authors maintain a modest tone, framing their work as a description of their participation and an exploration of how generative models fit into the existing multi-stage ranking infrastructure, rather than claiming a definitive solution to all IR challenges.