💬 NLP

DS@GT at TREC TOT 2025: Bridging Vague Recollection with Fusion Retrieval and Learned Reranking

The DS@GT system addresses the TREC Tip-of-the-Tongue task by employing a two-stage retrieval framework that fuses hybrid and topic-aware dense retrieval with a Gemini-2.5-flash reranker, achieving a recall of 0.66 on the test set.

Original authors: Wenxin Zhou, Ritesh Mehta, Anthony Miyaguchi

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenxin Zhou, Ritesh Mehta, Anthony Miyaguchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to remember a specific movie you saw years ago. You can't recall the title, the actors, or the director. But you remember the plot: "It was that sci-fi movie where the main character is a robot who falls in love with a human, and they live on a spaceship that looks like a giant donut."

This is the "Tip-of-the-Tongue" (ToT) problem. You have the feeling and the description, but not the name. Traditional search engines are like librarians who only look for exact words. If you ask for "robot love donut spaceship," they might get confused because the actual movie is called The Iron Giant or Star Trek.

The paper you shared describes a team from Georgia Tech (DS@GT) who built a super-smart search system to solve this exact problem for the TREC 2025 competition. Here is how they did it, explained simply with some analogies.

The Big Idea: A Two-Stage Detective Agency

Instead of just one search engine, the team built a two-stage detective agency.

Stage 1: The "Wide Net" (Retrieval)

First, the system casts a very wide net to catch as many possible candidates as possible. They don't rely on just one method; they use three different "detectives" working together:

The Keyword Detective (Sparse/BM25): This is the old-school librarian. It looks for exact word matches. If you say "robot," it finds documents with the word "robot."
The Semantic Detective (Dense/BGE-M3): This detective understands meaning, not just words. If you say "metal man," it knows that's similar to "robot," even if the word "robot" isn't there.
The Creative Detective (LLM Retrieval): This is an AI that reads your vague description and guesses the title directly. It's like asking a friend, "Hey, what movie fits this description?" and them shouting out, "Is it The Iron Giant?"

The "Round-Robin" Merge:
Instead of picking the winner from one detective, they use a round-robin strategy. Imagine three people taking turns picking the best card from a deck.

Detective A picks the #1 card.
Detective B picks the #1 card.
Detective C picks the #1 card.
Then they go back to A for the #2 card, and so on.
This ensures that if one detective misses the answer, the others might catch it. They also created a special "Topic Map" (dividing the library into 24 different rooms) so the search doesn't waste time looking in the "Cooking" section when you are looking for "Movies."

Stage 2: The "Expert Judge" (Reranking)

Now, the system has a huge pile of potential answers (maybe 1,000 candidates). It needs to find the one true answer at the very top. This is where the Reranker comes in. They tried two types of judges:

The Learned Judge (LambdaMART): This is a computer model trained on thousands of fake "Tip-of-the-Tongue" questions. It looks at clues like:
- How popular is this page? (PageRank)
- How many people have viewed it? (Pageviews)
- How well did the initial search match?
- Analogy: It's like a seasoned editor who knows which articles are usually the right answer based on patterns.
The AI Judge (LLM Reranker): This is a powerful AI (like Gemini) that reads the full text of the top candidates and your question, then decides, "Yes, this is definitely the one."
- Analogy: This is like hiring a movie critic who actually watches the clips and says, "This is the one you're thinking of!"

The Secret Sauce: Making Up Practice Questions

One of the biggest hurdles in training these systems is that there aren't enough real "Tip-of-the-Tongue" questions to learn from.

The Solution: The team used AI to invent 5,000 fake questions. They told the AI: "Pretend you forgot a famous person's name. Describe them vaguely, but make sure you're describing a real person."
This gave the system a massive gym to practice on, making it much sharper when it faced real human questions.

The Results: Who Won?

The team submitted several versions of their system. The winner was a combination of:

The Wide Net: Merging the Keyword, Semantic, and Creative detectives.
The AI Judge: Using the powerful Gemini model to do the final sorting.

The Outcome:

Their system found the correct answer in the top 1,000 results 66% of the time (Recall).
It was much better than using just one method. For example, the "Keyword Detective" alone would have missed most of them.

Why This Matters

This paper shows that when you are trying to find something you can't quite remember, combining different ways of searching is better than relying on just one.

Keywords catch the exact matches.
Meaning catches the concepts.
AI catches the intuition.
Reranking ensures the best match is at the very top.

It's like having a team of experts instead of a single person trying to solve a mystery. By fusing these methods, the system bridges the gap between your vague memory and the exact answer you need.

1. Problem Definition

The paper addresses the TREC Tip-of-the-Tongue (ToT) task, a specific known-item retrieval challenge.

The Challenge: Users possess a vague, verbose, natural language description of an item they have previously encountered (e.g., a movie, landmark, or celebrity) but cannot recall the specific title or name.
Complexity: These queries often contain multi-hop reasoning, expressions of uncertainty, false memories, and lack precise keywords, rendering traditional keyword-based search (like standard BM25) ineffective.
Goal: Develop a system capable of identifying the correct "known-item" from the Wikipedia corpus based on these imprecise descriptions.

2. Methodology

The authors propose a two-stage retrieval architecture combining hybrid retrieval, topic-aware indexing, and advanced reranking strategies.

A. Data Augmentation

To train the learned reranker, the team generated 5,000 synthetic ToT queries using Large Language Models (LLMs) including gemini-2.5-flash-lite, o4-mini, and gpt-4o.

Strategy: They sampled 5,000 diverse Wikipedia entities based on page view popularity and infobox availability.
Prompt Engineering: Prompts were designed to generate open-domain queries (removing specific domain keywords like "movie") and output results in JSON for easy parsing.
Validation: Synthetic queries showed high semantic correlation (Pearson > 0.93) with original queries, validating their use for training.

B. First-Stage Retrieval (Hybrid & Topic-Aware)

The system employs three distinct retrieval methods merged via a round-robin strategy:

Sparse Retrieval: Uses the official PyTerrier BM25 baseline.
Dense Retrieval: Uses the BGE-M3 model to generate embeddings for query-document similarity.
- Topic-Aware Multi-Index: The Wikipedia corpus is partitioned into 24 topical domains using ModernBERT-web-topics. Queries are classified, and retrieval is restricted to the relevant topic-specific FAISS index to improve efficiency.
LLM Retrieval: LLMs (e.g., Gemini-2.5-flash) are prompted to directly generate up to 20 matching Wikipedia titles with relevance scores.
- Matching Logic: Includes exact title matching, redirect/alias resolution, and parenthetical removal.

Fusion: Results from all three methods are merged using a round-robin approach, prioritizing the top candidate from each method in a cyclic manner to ensure diversity.

C. Network Construction (Graph Features)

A composite directed graph $G=(V, E)$ was constructed over Wikipedia links to generate features for reranking:

Edge Weights: A weighted sum of three signals:
1. Direct Citations ( $u \to v$ ).
2. Meta-Path Connections (via intermediate nodes like Categories).
3. Semantic Nearest Neighbors (based on BGE-M3 cosine similarity).
Features: PageRank, In-Degree, and Out-Degree were computed on this graph and used as features in the learned reranker.

D. Second-Stage Reranking

Two reranking approaches were evaluated:

Learned Reranker (LambdaMART):
- Trained using XGBoost with a pairwise loss function.
- Features: Dense score, Sparse (BM25) score, normalized pageview count, PageRank, and query word count.
- Training: Used official data plus the 5,000 synthetic queries (75% train, 15% dev, 10% test).
LLM Reranker:
- Implemented using the RankLLM library with a sliding window approach to fit documents into context windows.
- Tested various models (Gemma, Gemini) and context lengths (title, first paragraph, full text).

3. Key Contributions

Hybrid Fusion Strategy: Demonstrated that merging LLM-based, sparse, and dense retrieval via round-robin significantly outperforms any single retrieval method.
Topic-Aware Dense Retrieval: Introduced a multi-index approach that partitions the corpus into 24 domains, offering a trade-off between computational efficiency and recall.
Synthetic Data Generation: Validated a pipeline for generating high-quality synthetic ToT queries to augment training data for learned rerankers.
Comparative Reranking Analysis: Systematically compared learned (LambdaMART) vs. generative (LLM) reranking, highlighting the strengths and weaknesses of each.

4. Results

The team submitted 7 systems to the TREC TOT 2025 challenge. The best-performing system, gmn-rerank-500, achieved the following on the official test set:

Recall@1000: 0.6559
NDCG@1000: 0.4106

Key Findings:

Hybrid Retrieval: Improved Recall@1000 by 7.8% to 11.0% over the best individual retrieval method across development sets.
LLM Reranking: Outperformed baselines significantly. For sparse retrieval, Gemma-3-27B-QAT improved Recall@1000 by 68% over the baseline.
LambdaMART Performance:
- Successfully improved Recall (e.g., from 0.60 to 0.66 on Dev3 sparse).
- Precision Trade-off: While it boosted recall, it significantly lowered Reciprocal Rank (RR) and NDCG on the test set, indicating it pushed relevant documents down the list (good for recall, bad for precision).
Data Source Analysis: The system performed best on NIST-assessor queries (Recall@1000: 0.89) but struggled more with MS-ToT and synthetic datasets, suggesting a difficulty gap between human-elicited and generated queries.

5. Significance and Conclusion

This work demonstrates that fusion retrieval is critical for solving the Tip-of-the-Tongue problem, where no single retrieval paradigm (keyword, semantic, or generative) is sufficient alone.

Effectiveness: Combining diverse retrieval signals with strong LLM-based reranking yields the highest performance.
Efficiency vs. Accuracy: While LLM reranking is superior in accuracy, the paper shows that a learned LambdaMART reranker can serve as a computationally efficient filter to boost recall, which can then be refined by a final LLM pass.
Future Directions: The authors suggest future work in graph-based retrieval expansion (BFS) and using network nodes to guide synthetic dataset construction.

The implementation is open-source, providing a robust baseline for future research in vague recollection and known-item search.