Technical Summary: RAS (Retrieval-And-Structuring) for Knowledge-Intensive LLM Generation

Problem Statement

Large Language Models (LLMs) have demonstrated impressive capabilities in knowledge-intensive tasks but frequently struggle with multi-step reasoning. This limitation stems largely from the unstructured nature of retrieved context in standard Retrieval-Augmented Generation (RAG) frameworks. While RAG provides external information, the lack of explicit organization among retrieved passages forces models to implicitly bridge logical gaps, often leading to brittle reasoning pathways and hallucinations.

Existing attempts to integrate Knowledge Graphs (KGs) with LLMs typically rely on static, corpus-wide graphs. These approaches face two significant limitations:

1. High Cost: Building and maintaining global KGs (e.g., for Wikipedia) requires millions of LLM calls and substantial financial resources.
2. Ambiguity: Global graphs often blend evidence from conflicting documents, creating ambiguous or contradictory relations that obscure the specific context relevant to a user query.

Consequently, there is a need for a system that can dynamically construct knowledge structures tailored to specific queries, enabling more interpretable and robust reasoning without the prohibitive costs of offline indexing.

Methodology: The RAS Framework

The authors propose Retrieval-And-Structuring (RAS), a framework that dynamically constructs question-specific knowledge graphs through iterative retrieval and structured knowledge building. RAS interleaves targeted retrieval planning with incremental graph construction, allowing the model to assemble and reason over evolving knowledge structures tailored to each query.

The framework operates through three primary stages:

1. Knowledge-Aware Planning

The process is initiated and controlled by a planning module that dynamically assesses the current knowledge state.

• Initial Planning: Given an input query $Q$, the model determines if the query can be answered directly ([NO_RETRIEVAL]) or if retrieval is needed. If retrieval is required, the main question serves as the initial subquery ($q_0$).
• Iterative Planning: At each iteration $i$, the model analyzes the accumulated knowledge graph $G_i$ and the history of subqueries and extracted triples. It decides whether to:
  • Generate a new focused subquery ($q_{i+1}$) to fill specific knowledge gaps ([SUBQ]).
  • Terminate retrieval if the accumulated knowledge is sufficient ([SUFFICIENT]).

2. Text Retrieval and Structuring

Once a subquery is generated, the system executes a retrieval-and-structuring pipeline:

• Retrieval: A dense retriever fetches the top-$k$ semantically relevant passages ($t_i$) for the subquery.
• Text-to-Triples Conversion: A specialized model ($f_{t2t}$) converts the retrieved text into structured factual triples (subject-predicate-object). This model is trained on the WikiOfGraph dataset.
• Graph Construction: The extracted triples are converted into a graph structure $g'_i$ with semantic embeddings for nodes and edges.
• Iterative Enrichment: The new graph $g'_i$ is merged into an evolving, question-specific knowledge graph $G_Q$. This graph grows iteratively, accumulating evidence relevant to the main query.

3. Knowledge-Augmented Answering

When the planning module signals that knowledge is sufficient, the model generates the final answer.

• If no retrieval was needed, the answer is generated directly from the query.
• Otherwise, the answer is generated conditioned on the encoded question-specific KG ($G_Q$) and the chain of subqueries and their associated graph information. The model attends to this structured context to produce accurate, coherent outputs.

Training Strategy

RAS employs a Structure-Aware Multitask Learning approach. A single Graph LLM is trained to perform both planning and answering tasks using a parameter-efficient fine-tuning strategy (LoRA). The training data, HotpotQA-SUBQ, is constructed by filtering HotpotQA documents, generating subqueries for each supporting document, and labeling samples with the appropriate planning action ([SUBQ], [SUFFICIENT], or [NO_RETRIEVAL]) or final answer.

Key Contributions

1. RAS Framework: The proposal of a framework that dynamically builds question-specific knowledge graphs through iterative retrieval and structuring, avoiding the costs and noise of global KGs.
2. Unified Model Architecture: The design of a unified Graph LLM that jointly plans retrieval needs and generates answers over evolving knowledge graphs, utilizing a multitask learning setup.
3. Empirical Validation: Demonstration of consistent performance gains across seven diverse benchmarks, showing that dynamic structuring improves reasoning accuracy and robustness.

Experimental Results

The authors evaluated RAS on seven benchmarks covering open-domain QA (TriviaQA, PopQA, 2WikiMultihopQA), closed-set tasks (PubHealth, ARC-Challenge), and long-form generation (ASQA, ELI5).

• Performance Gains: RAS consistently outperformed strong baselines, including Self-RAG, RPG, and SuRe.
  • With open-source LLMs (LLaMA-2-7B/LLaMA-3-8B), RAS achieved gains of up to 7.0% over baselines.
  • With proprietary models (Claude-3.5-Sonnet), RAS achieved gains of up to 8.7%.
• Ablation Studies:
  • Iterative Planning: Removing the planning module (reverting to single-pass retrieval) caused significant performance drops, highlighting the necessity of dynamically determining retrieval needs.
  • Graph Construction: Removing the text-to-triple conversion or the graph encoder led to substantial degradation, confirming the value of structured knowledge representation.
  • Training Strategy: Joint multitask training and the use of LoRA were found to be critical for optimal performance.
• Efficiency: Unlike static graph methods (e.g., GraphRAG) that require expensive offline indexing (estimated at \85k–\500k for Wikipedia scale), RAS incurs negligible offline costs by constructing graphs dynamically at inference time.

Significance and Claims

The paper claims that dynamic, question-specific knowledge structuring offers a robust path to improving reasoning accuracy and robustness in language model generation. By explicitly organizing retrieved knowledge into evolving graphs, RAS mitigates the hallucination risks associated with unstructured context and the inefficiencies of global knowledge graphs.

The authors emphasize that their approach aligns with interpretability studies suggesting that LLM failures often stem from broken implicit reasoning chains. By making these chains explicit through structured intermediate reasoning, RAS provides a transparent and scalable solution for knowledge-intensive tasks. The framework is presented as a general-purpose approach for open-domain question answering rather than being limited to specific Knowledge Graph Question Answering (KGQA) tasks.