💬 NLP

Don't Retrieve, Navigate: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG

The paper introduces Corpus2Skill, a method that transforms document corpora into hierarchical, navigable skill trees to enable LLM agents to actively explore and reason about enterprise knowledge structures, thereby outperforming traditional retrieval-based approaches on QA benchmarks.

Original authors: Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Searching, Start Exploring

Imagine you are a detective trying to solve a mystery in a massive, messy library.

The Old Way (Traditional RAG):
You ask the librarian, "Where is the book about changing my business from a sole proprietorship to an LLC?"
The librarian runs to the back, grabs the top 5 books that have the words "LLC" or "business" on the cover, and hands them to you.

The Problem: You get five books. One is about taxes, one is about opening a bank account, and one is about legal forms. None of them actually tell you how to switch your business type because that specific rule is hidden in a different section of the library. You have to read all five books, guess which one matters, and hope you didn't miss the real answer. You are a passive consumer of whatever the librarian found.

The New Way (CORPUS2SKILL):
Instead of handing you random books, the library gives you a detailed, organized map of the entire building before you even start looking.

You see a big sign: "Business & Finance."
You walk over there and see a sign: "Legal Entity Changes."
You walk down that hall and see a sign: "Switching Business Types."
Finally, you find the exact door with the answer.

If you walk down the wrong hallway (e.g., "Tax Forms"), you realize immediately, "Oh, this isn't right," and you backtrack to try a different path. You are an active explorer with a map.

How It Works: The "Cooking" Analogy

The paper describes a two-step process: Compiling (Cooking) and Serving (Eating).

1. The Compile Phase (The Chef Prepares the Meal)

Before anyone ever asks a question, the system takes the messy pile of 6,000+ documents (like a giant bag of mixed ingredients) and organizes them.

Clustering: It groups similar documents together (like putting all "vegetables" in one bin and all "spices" in another).
Summarizing: For every group, it writes a short summary card (like a label on the bin saying "Spices: Chili, Cumin, Salt").
Building the Tree: It creates a hierarchy.
- Level 1: Big categories (e.g., "Payments," "Marketing").
- Level 2: Sub-categories (e.g., "Credit Cards," "SEO").
- Level 3: Specific documents.
The Result: A clean, navigable "Skill Tree" (like a well-organized pantry) is saved. This happens once, offline.

2. The Serve Phase (The Diner Orders)

When a user asks a question, the AI Agent doesn't search a database. It reads the menu.

Step 1: The Agent sees the top-level categories. "Hmm, the user asked about changing business types. That sounds like 'Payments' or 'Legal'."
Step 2: It opens the "Payments" folder and reads the summary. "Ah, there's a sub-folder called 'Account Management'."
Step 3: It opens that folder, sees a list of specific articles, and picks the one titled "Changing Your Business Type."
Step 4: It reads that single article and gives the answer.

If the Agent realizes "Payments" is the wrong path, it can close that folder and open "Legal" instead. It has agency.

Why Is This Better?

The paper tested this on a real customer support benchmark (WixQA) and found it beat the old methods in three key ways:

It Sees the "Forest": Traditional search only sees the "trees" (individual documents). This system sees the whole forest. If a question spans two topics (e.g., "How do I change my currency for my online course?"), the Agent can look at the "Course" folder, realize it doesn't have the answer, and then look at the "Billing" folder to combine the two pieces of information.
It Can Backtrack: If the Agent goes down a dead end, it knows it's a dead end because the summary said so. It can turn around and try a different path. Traditional search just gives you the wrong answer and you have to guess.
No "Black Box": In traditional search, the AI doesn't know why it picked a document. In this system, the AI knows exactly which folder it came from and why it chose that path.

The Trade-off: Cost vs. Quality

The Cost: This method uses more "tokens" (computing power) per question because the AI has to read the summaries of the folders as it navigates. It's like reading the menu before ordering, which takes a few seconds.
The Benefit: The answers are much more accurate and factual. For complex business questions where getting it wrong is expensive (like legal or billing advice), the extra cost is worth it.

Summary Metaphor

Traditional RAG is like asking a random person on the street for directions. They might point you to the right street, or they might point you to a dead end because they only know a few blocks.
CORPUS2SKILL is like giving the traveler a GPS with a full map of the city. They can see the whole route, choose the best path, and if they take a wrong turn, they can instantly see the map and reroute.

The paper's main message is simple: Don't just let the AI search for answers. Give it a map so it can navigate to them.

1. Problem Statement

Current Retrieval-Augmented Generation (RAG) systems face a structural limitation: they treat Large Language Models (LLMs) as passive consumers of search results.

The "Forest for the Trees" Issue: Traditional RAG uses embedding-based similarity search to retrieve top- $k$ passages. The LLM receives these fragments but lacks a global view of the corpus structure. It cannot see what topics exist outside the retrieved set, cannot reason about the organization of knowledge, and cannot effectively backtrack or combine scattered evidence from different branches.
Limitations of Agentic RAG: While Agentic RAG allows LLMs to issue multiple queries iteratively, agents operate without a "map." They must guess search terms and lack a systematic way to survey the corpus structure, often leading to inefficient "shot in the dark" searches.
Limitations of Hierarchical RAG (e.g., RAPTOR, GraphRAG): These systems organize documents into trees or graphs with summaries, but the hierarchy remains invisible to the LLM at query time. The model still relies on vector similarity search to find nodes, rather than actively navigating the structure.

Core Challenge: How to transform a static document corpus into a navigable interface that allows an LLM agent to actively explore, reason about structure, and locate evidence systematically, rather than relying on opaque retrieval.

2. Methodology: CORPUS2SKILL

The authors propose CORPUS2SKILL, a framework that shifts the paradigm from "Retrieve" to "Navigate." It operates in two distinct phases:

A. Compile Phase (Offline)

This phase transforms a raw document corpus into a hierarchical "skill directory" (a file system of summaries).

Document Loading & Embedding: Documents are assigned unique IDs and embedded using a sentence embedding model.
Iterative Hierarchical Clustering:
- The system uses a bottom-up approach with a branching ratio $p$ (e.g., 10).
- Embed-Cluster-Summarize Loop: Documents are clustered (K-Means), and an LLM generates a concise summary for each cluster describing its topical coverage, question types, and key terms.
- These summaries are re-embedded and clustered again to form higher levels.
- This continues until a root level of $K$ clusters is reached.
Skill Tree Materialization: The hierarchy is converted into a filesystem structure:
- Root Clusters: Become top-level skill directories containing SKILL.md (summary + list of subgroups).
- Sub-Clusters: Become subdirectories containing INDEX.md (summary + list of further subgroups or document IDs).
- Leaf Documents: Full text is stored externally in a documents.json file, referenced by ID.
- Progressive Disclosure: Only metadata (names/descriptions) is pre-loaded; full content is loaded on demand.

B. Serve Phase (Online)

At query time, an LLM agent navigates this pre-compiled tree using two specific tools:

Code Execution (File Browsing): The agent uses commands (e.g., view, cat) to read SKILL.md and INDEX.md files. This provides a "bird's-eye view" of the corpus.
get_document(doc_id): Once the agent identifies a relevant document ID from a leaf INDEX.md, it retrieves the full text.

Workflow:

Overview: The agent sees pre-loaded skill descriptions and selects a relevant top-level skill.
Drill-down: It reads the SKILL.md to see subgroups, then reads the relevant INDEX.md to see document summaries.
Retrieval: It calls get_document for specific IDs.
Synthesis: It generates a grounded answer based only on the retrieved full documents.

3. Key Contributions

Corpus Navigability: Identified "navigability" as an underexplored dimension in RAG. The system gives the LLM a structured overview, enabling systematic evidence location and backtracking.
Informational Skills: Repurposed the concept of "Agent Skills" (typically procedural workflows) to encode informational knowledge. The corpus itself becomes a navigable skill tree.
Compile-Then-Navigate Framework: Introduced a pipeline that invests computation at compile time to create a browsable hierarchy, eliminating the need for vector databases or embedding indexes at serve time.
Progressive Disclosure Implementation: Leveraged filesystem-based skill packages to manage context windows efficiently, loading only necessary summaries before retrieving full documents.

4. Experimental Results

Evaluated on WixQA, an enterprise customer-support benchmark with 6,221 documents and 200 expert queries.

Baselines Compared:

Flat Retrieval: BM25, Dense (Vector), Hybrid.
Hierarchical Retrieval: RAPTOR (Tree-based).
Agentic RAG: Multi-turn agent using traditional search tools.

Key Findings:

Superior Quality: CORPUS2SKILL achieved the highest scores across all six metrics (Token F1, BLEU, ROUGE-1/2, Factuality, Context Recall).
- Token F1: 0.460 (vs. 0.388 for Agentic RAG, 0.363 for Dense).
- Factuality: 0.729 (vs. 0.724 for Agentic, 0.675 for RAPTOR).
- Context Recall: 0.652 (vs. 0.481 for Agentic), indicating it surfaces more relevant content than vector search.
Cost vs. Quality Trade-off:
- CORPUS2SKILL is more expensive per query ($0.172) than RAPTOR ($0.12) or Agentic RAG ($0.098) due to higher input token usage (reading navigation files).
- However, it produces significantly more accurate and grounded answers, justifying the cost for high-stakes enterprise queries.
Ablation Studies:
- Structure: Narrower trees ( $p=5$ ) slightly outperformed wider trees ( $p=20$ ), suggesting finer-grained routing is beneficial.
- Budget: Quality remained stable even with reduced interaction rounds, proving the hierarchy is well-organized.
- Model: A cheaper serving model (Claude Haiku) achieved comparable quality at half the cost, proving the hierarchy's robustness.

5. Significance and Future Directions

Paradigm Shift: The paper argues for shifting the computational burden from query time (searching) to compile time (structuring). By making the corpus navigable, the LLM's reasoning capabilities are fully utilized.
Scalability: The hierarchy depth grows logarithmically ( $O(\log_p N)$ ), allowing the system to scale to large corpora without a linear increase in navigation steps.
Limitations:
- Cost: High input token usage due to reading navigation files.
- Hard Clustering: Documents spanning multiple topics are forced into a single branch, potentially causing routing errors (though "cross-branch" navigation mitigates this).
- Static Nature: Requires full recompilation for document updates (no incremental updates yet).
Impact: Offers a complementary design point for knowledge-grounded systems, particularly for complex, multi-hop enterprise queries where understanding the structure of the knowledge base is as important as the content itself.

In conclusion, CORPUS2SKILL demonstrates that giving an LLM agent a "map" (a navigable skill tree) yields higher-quality, more factual answers than simply feeding it search results, effectively turning the LLM from a passive reader into an active explorer of enterprise knowledge.