Technical Summary: LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval

Problem Statement

Modern search systems are increasingly tasked with reasoning-intensive queries, where relevance depends on complex relations, arguments, or chains of inference between the query and document, rather than surface vocabulary or topical similarity. The standard retrieval paradigm—a cheap embedding-based retriever followed by an LLM reranker—fails in this setting. Recent benchmarks (e.g., BRIGHT) demonstrate that even state-of-the-art embedding models often fail to place relevant documents in the top-$k$ when relevance requires deep reasoning. Consequently, the reranker cannot recover documents the retriever missed.

Existing attempts to fix this by pushing the LLM "upstream" (via query rewriting or agentic loops) remain brittle. In these approaches, the LLM indirectly interacts with the corpus through the same noisy embedding retriever. If the retriever fails to surface genuinely new information, the LLM's agentic loops often reinforce its own parametric beliefs rather than correcting them. Furthermore, flat retrieve-then-rerank pipelines discard the inherent hierarchical structure of many corpora (e.g., Wikipedia sections, document passages), forcing the LLM to score passages out of context.

Methodology: LATTICE

The authors propose LATTICE (LLM-guided Hierarchical Search), a paradigm where an LLM interacts directly with the corpus via a navigable, hierarchical search index, eliminating the embedding model from the search loop. The system consists of two main phases:

1. Offline: Top-Down Hierarchical Index Construction

Unlike prior work (e.g., RAPTOR) that builds indices bottom-up using embedding-based clustering, LATTICE constructs the index top-down using LLM judgments.

• Hierarchical Summarization: Each document is preprocessed into $k$ levels of summaries with increasing specificity (from a 1-2 word topic to a full sentence). This creates a token-efficient, depth-adaptive interface.
• Recursive Partitioning: Starting with a root node containing all documents, the algorithm recursively partitions clusters. For a given node, the LLM is shown multi-level summaries of the leaves and asked to organize them into $M$ coherent topics.
• Dynamic Adaptation: The system selects the appropriate summary level based on the cluster size and the LLM's context budget, ensuring coarse distinctions at the root and fine-grained distinctions deeper in the tree.
• Hybrid Construction: For very large corpora or those with natural hierarchies (e.g., Wikipedia), a bottom-up pass may generate initial coarse clusters, which are then refined by the top-down LLM process.

2. Online: Calibrated, Path-Aggregated Traversal

At query time, a search LLM performs a best-first frontier search over the tree. To address the noise and context-dependence of LLM scoring, LATTICE introduces two key mechanisms:

• Cross-Branch Calibration: To mitigate "slate dependence" (where an LLM's score for a node changes based on its siblings), the search prompt includes reference nodes drawn from other branches of the tree. This provides a stable baseline for comparison across different search paths.
• Path-Aggregated Scoring: Raw LLM scores are noisy and local. LATTICE fits a latent-score model to the history of all scores to remove slate-specific bias. It then accumulates these latent scores along the root-to-leaf path using an exponential moving average. This ensures the search is guided by cumulative evidence rather than a single local decision.

Key Contributions

1. A New Search Paradigm: LATTICE shifts the LLM from a post-retrieval verifier to an active traversal agent, removing the embedding bottleneck for reasoning-intensive tasks.
2. Top-Down Index Construction: The paper demonstrates that top-down construction using LLM judgments over multi-level summaries produces a more navigable index than bottom-up embedding clustering, particularly for separating conceptually distinct but lexically similar documents.
3. Robust Traversal Mechanisms: The introduction of cross-branch calibration and path-aggregated scoring allows off-the-shelf LLMs to navigate large trees reliably, overcoming the inherent noise and slate-dependence of listwise LLM scoring.

Experimental Results

The authors evaluate LATTICE on the BRIGHT benchmark (reasoning-intensive retrieval) and traditional IR datasets (BEIR).

• BRIGHT Performance:

  • Base LATTICE: Using a single zero-shot LLM (Gemini-3-Flash) without ensembling, LATTICE achieves 46.7 nDCG@10. This matches or exceeds the best fine-tuned ensemble baselines (e.g., BGE-Reasoner-0928 at 46.4, DIVER v3 at 46.8) and significantly outperforms non-ensemble zero-shot baselines.
  • LATTICE++: A lightweight ensemble fusing LATTICE with BM25 and a dense retriever reaches 49.1 nDCG@10, setting a new state-of-the-art on the benchmark.
  • Scaling: LATTICE shows superior test-time scaling compared to sliding-window reranking. While reranking wins at low token budgets, LATTICE converges to a higher asymptote as the token budget increases, indicating its ability to find relevant documents through deeper exploration.
  • Cost Efficiency: LATTICE works effectively with open-weight models (e.g., Qwen3.5-27B), achieving performance comparable to proprietary models at roughly 4x lower cost per query.

• Traditional IR:

  • On standard BEIR datasets (SciFact, SciDocs, NQ), LATTICE remains competitive, achieving 75.8 nDCG@10 on SciFact and 66.8 on NQ (2.68M documents). However, the authors note that for these tasks, the high per-query cost of LLM traversal is not yet justified over standard retrieve-then-rerank pipelines.

• Ablations:

  • Top-down construction significantly outperforms bottom-up embedding clustering (e.g., +12 nDCG@10 on TheoremQA-T).
  • Cross-branch calibration is essential; without it, performance drops by ~22 points.
  • Depth (more iterations) is more valuable than breadth (larger beam size) under a fixed token budget.

Significance and Claims

The paper claims to demonstrate the promise of a search paradigm where an LLM interacts directly with a corpus through a navigable semantic hierarchy, bypassing the limitations of embedding-based retrieval for reasoning-intensive tasks.

• Limitations: The authors acknowledge that LATTICE is best suited for applications where latency is secondary to quality (e.g., deep research, legal QA, technical QA) due to the computational cost of LLM calls at each step. They also note that static hierarchical indices struggle with dynamic corpora where documents are excluded per query (a feature of some BRIGHT subsets), as pre-computed summaries become stale.
• Future Work: The paper suggests future directions include efficient localized index updates for dynamic corpora and distilling the traversal policy into smaller models.

In summary, LATTICE provides a viable alternative to the standard retrieve-then-rerank pipeline for complex reasoning tasks, proving that an LLM can effectively act as both the indexer and the searcher when provided with a properly structured, semantic navigation tool.