🤖 machine learning

Hierarchical Abstract Tree for Cross-Document Retrieval-Augmented Generation

This paper introduces $\Psi$ -RAG, a novel tree-based retrieval-augmented generation framework that employs an adaptive hierarchical abstract tree index and a multi-granular retrieval agent to overcome the limitations of existing methods in handling cross-document multi-hop questions, achieving state-of-the-art performance on relevant benchmarks.

Original authors: Ziwen Zhao, Menglin Yang

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziwen Zhao, Menglin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library containing millions of books, articles, and documents. You want to ask a complex question that requires connecting dots across many different pages, like, "Who is the wife of the man who produced the documentary about the singer who inspired Beyoncé?"

Traditional search engines (the "old way") are like a librarian who just grabs a few random pages that contain the words "Beyoncé" or "documentary." They often miss the crucial link to the specific man you're asking about because they don't understand the story connecting them.

Other advanced systems try to organize these books into a Tree (a hierarchy where big summaries sit on top of smaller details). However, the paper argues that existing tree systems have three major flaws:

They force a square peg into a round hole: They assume all information is evenly distributed, which isn't true. This confuses the system when some topics are rare and others are common.
They are isolated islands: The branches of the tree don't talk to each other well. If the answer requires jumping from one branch to another, the system gets stuck.
They are too vague: The "summary" at the top of the tree is so broad it forgets the specific details needed to answer a precise question.

The Solution: Ψ-RAG (Psi-RAG)

The authors propose a new system called Ψ-RAG. Think of it as a super-smart, adaptive librarian who doesn't just organize books; they actively think about how to find the answer.

Here is how it works, broken down into two main parts:

1. The "Merging and Collapsing" Tree (The Library Reorganization)

Instead of forcing books into rigid, pre-defined categories (like "Sports" or "History"), Ψ-RAG builds its own map based on how similar the texts actually are.

The Analogy: Imagine you have a pile of mixed-up LEGO bricks. Instead of sorting them by color first, you start snapping similar bricks together. If you find two bricks that fit perfectly, you glue them. If you find a small cluster that fits under a bigger cluster, you attach it there.
The Result: This creates a "Hierarchical Abstract Tree." The bottom layer has the actual text chunks. The layers above are summaries (abstracts) of the layers below.
Why it's better: Unlike other systems that try to make every branch the same size (which messes up rare topics), Ψ-RAG lets the tree grow naturally. If a topic is rare, it gets its own small, distinct branch. If a topic is huge, it gets a big branch. This preserves the "shape" of the information.

2. The "Multi-Granular Agentic Retrieval" (The Detective Agent)

This is the brain of the operation. It's not just a search bar; it's an AI detective that can talk to the library.

The Detective's Job: When you ask a question, the agent doesn't just look once. It looks at the top of the tree (the big summaries) to get the general idea. If the answer isn't there, it asks itself, "What specific detail am I missing?"
Query Reorganization: If the agent realizes it's stuck, it rewrites your question to be more specific.
- Original Question: "Who is the wife of the man who produced the documentary...?"
- Agent's Rewritten Question: "Who is the wife of David Gest?" (After figuring out the man's name in the first step).
The Hybrid Search: The agent uses two tools at once:
1. The Tree: To understand the big picture and follow the logical path.
2. A Keyword Search (Sparse Index): To find exact names and facts that might get lost in a broad summary.
The Loop: The agent keeps asking, "Do I have enough info?" If not, it digs deeper, rewrites the question, and searches again until it has the full story.

Why This Matters (According to the Paper)

The paper tested this system on difficult questions that require "multi-hop" reasoning (connecting A to B, then B to C, then C to the answer).

Speed vs. Accuracy: Other tree systems were fast but inaccurate on big datasets. Graph-based systems (which map relationships like a subway map) were accurate but very slow to build.
The Winner: Ψ-RAG was 25.9% more accurate than the previous best tree system (RAPTOR) and 7.4% more accurate than the top graph system (HippoRAG 2).
Efficiency: It built its index 6.5 times faster than RAPTOR and was much faster than the graph systems, making it practical for large collections of documents.

In a Nutshell

Ψ-RAG is a new way to organize and search through huge amounts of text. It builds a flexible map that respects the natural structure of the data and uses an AI "detective" to navigate that map, asking follow-up questions and combining broad summaries with specific facts to solve complex puzzles that other systems miss.

1. Problem Statement

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) with external knowledge. While Tree-RAG methods (e.g., RAPTOR) organize documents into hierarchical indexes to support multi-granularity queries, they face critical limitations when scaling to cross-document multi-hop questions:

Poor Distribution Adaptability: Existing methods rely on $k$ -means-type clustering (e.g., Gaussian Mixture Models). These algorithms assume spherical data distributions and suffer from the "uniform effect," forcing clusters to be of uniform size. This introduces noise by merging minor, skewed topics into major clusters, causing the retriever to overlook niche information.
Structural Isolation: Tree indexes typically lack explicit connections between leaf nodes across different branches. Unlike Graph-RAG, which hops between entities via edges, tree leaves are isolated. This prevents the retriever from capturing implicit causal dependencies required for multi-hop reasoning.
Coarse Abstraction: Standard tree abstraction often creates high-level summaries that obscure fine-grained details. Dense vector matching struggles to associate specific entities in a user query with these abstract concepts, leading to retrieval failures on factual, token-level questions.

2. Methodology: Ψ-RAG

The authors propose Ψ-RAG, a framework comprising two core components: a Hierarchical Abstract Tree Index and a Multi-granular Agentic Retriever.

A. Hierarchical Abstract Tree Indexing

Instead of $k$ -means clustering, Ψ-RAG uses an iterative "merging and collapse" process inspired by Agglomerative Hierarchical Clustering (AHC) but without a priori distribution assumptions.

Similarity Ranking: All document chunks are encoded into dense vectors, and pairwise similarities are computed and sorted.
Merging & Collapse:
- Merging: The most similar pair of chunks (or nodes) is linked under a new abstract node.
- Leaf Node Collapse: If one node is already linked to a parent, the isolated node is attached to that same parent.
- Abstract Node Collapse: If two nodes belong to different parents at the same depth, their roots are merged under a new node. If depths differ, the shallower root is linked onto the path of the deeper node to balance the tree.
Abstraction: An LLM agent generates two types of abstracts for each non-leaf node:
- Summative Abstract: A coherent paragraph summarizing logic and entity relationships.
- Keyword Abstract: High-density keywords/phrases preserving entity co-occurrence.
Rebalancing: Nodes with excessive children are split to ensure the context window of the abstraction agent is not exceeded.

B. Multi-granular Agentic Retrieval

To address structural isolation and coarse abstraction, Ψ-RAG employs a Retrieval and Answering (R&A) Agent that interacts with a Multi-granular Hybrid Index (Tree + Sparse Keyword Index).

Iterative Reasoning: The agent assesses if current information is sufficient. If not, it generates a reorganized query (enriched with context, e.g., adding descriptive appositions) and triggers a new retrieval step.
Hybrid Retrieval:
- Tree Retrieval: A top-down dense vector search starting from the root to the leaf layer.
- Sparse Retrieval: A BM25-based keyword search.
- Fusion: Results are combined via a reranker (parametric) or Reciprocal Rank Fusion (non-parametric).
Query Reorganization: The agent dynamically rewrites queries to include thematic keywords, aiding both the sparse retriever (finding specific entities) and the tree retriever (locating relevant high-level abstracts).

3. Key Contributions

Distribution Adaptability: Ψ-RAG theoretically proves (via Dasgupta's cost analysis) that its hierarchical merging process avoids the "uniform effect" of $k$ -means. It adaptively preserves skewed data distributions, ensuring minor clusters remain distinct rather than being absorbed by major ones.
First Cross-Document Tree-RAG: It is the first Tree-RAG framework capable of handling corpus-level, cross-document multi-hop retrieval, matching the capabilities of Graph-RAG while maintaining the efficiency of tree structures.
Hybrid Agentic Framework: It resolves the conflict between coarse abstraction and fine-grained retrieval by integrating a sparse keyword index and an iterative agent that reorganizes queries based on reasoning needs.
Flexibility: The framework is built entirely on open-source LLMs and is modular, allowing components (embedding, abstraction, agent, reranker) to be swapped without retraining.

4. Experimental Results

The authors evaluated Ψ-RAG on diverse benchmarks including single-hop/multi-hop QA (HotpotQA, 2Wiki, MuSiQue, MultiHop-RAG), narrative QA (NarrativeQA), and summarization (QMSum, WCEP).

Performance Gains:
- On cross-document multi-hop QA benchmarks, Ψ-RAG outperforms RAPTOR by 25.9% in average F1 score.
- It surpasses the state-of-the-art HippoRAG 2 (a Graph-RAG method) by 7.4% in average F1.
- It achieves significant gains in retrieval metrics (Recall@5) across all datasets.
Efficiency:
- Indexing: Ψ-RAG builds a corpus-level tree 10x faster than OpenIE-based Graph-RAG (e.g., GraphRAG) and 6.5x faster than RAPTOR.
- Retrieval: The top-down search offers $O(\log n)$ complexity, resulting in lower latency than Graph-RAG's traversal.
Ablation Studies:
- Removing the R&A agent causes a ~20% drop in multi-hop performance.
- Removing the sparse retriever significantly hurts performance on factual queries (e.g., -21.8% on PopQA).
- Query reorganization by the agent is critical for multi-hop success.

5. Significance

Bridging the Gap: Ψ-RAG successfully bridges the gap between the efficiency of Tree-RAG and the reasoning power of Graph-RAG, proving that hierarchical structures can effectively handle complex multi-hop reasoning without the heavy overhead of graph construction.
Scalability: By avoiding the "uniform effect" and utilizing efficient merging algorithms, it scales effectively to large, skewed corpora where traditional clustering fails.
Practicality: The framework demonstrates that open-source models can achieve state-of-the-art performance in structured RAG, making advanced multi-hop reasoning accessible without proprietary APIs or extensive fine-tuning.
Future Direction: It sets a new standard for handling cross-document retrieval, suggesting that hierarchical indexing combined with agentic reasoning is a viable and superior path for next-generation RAG systems.