Technical Summary: DeepRefine

Problem Statement

Agent-compiled knowledge bases, which utilize Large Language Model (LLM) agents to ingest sources, synthesize articles, and maintain cross-references over time, offer a dynamic alternative to static Retrieval-Augmented Generation (RAG) systems. However, these dynamically compiled knowledge bases suffer from systematic quality defects: incompleteness (missing evidence chains or links), incorrectness (low-confidence or imprecise assertions), and redundancy (ambiguous or unresolved coreference issues).

These defects compound under iterative use, degrading retrieval fidelity and downstream task performance. While recent research focuses on optimizing the construction of knowledge bases, re-training constructors and re-compiling entire knowledge bases for existing deployments is often expensive and operationally impractical. This creates a need for a post-construction refinement paradigm that can improve the quality of already-deployed knowledge bases without full reconstruction.

Two core challenges hinder effective refinement:

1. Defect Localization: Identifying problematic regions in large, relationally connected knowledge bases without performing expensive exhaustive traversal.
2. Policy Optimization without Golden References: Learning effective refinement strategies when no "correct" sequence of refinement actions or golden knowledge bases are available for supervision.

Methodology: DeepRefine

The authors propose DeepRefine, a general LLM-based reasoning model designed to refine pre-constructed knowledge bases using only user queries. DeepRefine operates via a three-step reasoning process and is optimized using Reinforcement Learning (RL).

1. Reasoning Process

DeepRefine follows a multi-turn interaction loop with the knowledge base ($G_f$):

• Answerability Judgement Loop: DeepRefine first attempts to answer a query using the existing knowledge base. It performs dense retrieval to obtain a 0-hop subgraph. If the query is unanswerable, it iteratively expands the subgraph by retrieving connected triples (1-hop, 2-hop, etc.) and pruning them based on relevance to the query. This process continues until the query is answerable or a maximum hop limit is reached. The interaction history ($H_q$) is recorded.
• Error Abduction: If the query remains unanswerable, DeepRefine abductively reasons about the potential causes of failure based on the interaction history. It diagnoses issues from three perspectives: incompleteness (missing intermediate nodes/relations), incorrectness (conflicting or wrong facts), and redundancy (ambiguity or coreference issues).
• Refinement Actions Generation: Based on the abduced errors, DeepRefine generates a sequence of refinement actions to edit the full knowledge base directly. These actions are mapped to three pre-defined operators:
  • insert_edge(): Complements missing relations or items (addresses incompleteness).
  • delete_edge(): Removes incorrect or redundant relations (addresses incorrectness/redundancy).
  • replace_node(): Resolves ambiguity or disambiguates entities (addresses redundancy).

2. Policy Optimization via Reinforcement Learning

Since there are no golden references for refinement actions, DeepRefine is trained end-to-end using Group Relative Policy Optimization (GRPO).

• Reward Design (Gain-Beyond-Draft - GBD): The reward signal is derived from the downstream task performance. Specifically, the GBD reward quantifies the improvement in RAG generation accuracy achieved by the refined knowledge base ($\hat{G}_f$) relative to the draft knowledge base ($G_f$).
  $$GBD(q) = ACC(A_{refined}, A) - ACC(A_{draft}, A)$$
  Where $A$ is the ground truth answer, and $ACC$ is a task-specific metric (Generation Accuracy).
• Training Objective: The policy is optimized to maximize the group-relative advantage of the GBD reward, encouraging the model to select refinement actions that demonstrably improve downstream answerability.

3. Inference

During inference, DeepRefine processes online query streams asynchronously. It uses multi-step reasoning chains to identify and refine specific regions of the knowledge base, updating the agent-compiled store to better suit downstream tasks without introducing systematic latency.

Key Contributions

1. First General Refinement Model: DeepRefine is presented as the first general LLM-based reasoning model capable of refining any pre-constructed agent-compiled knowledge base using only user queries, without requiring re-construction.
2. Novel Reasoning Framework: The authors design a three-step process (Answerability Judgement, Error Abduction, Refinement Generation) that localizes defects via interaction history rather than exhaustive search.
3. GBD Reward Mechanism: The paper introduces the Gain-Beyond-Draft (GBD) reward, enabling the training of refinement policies via RL in the absence of golden reference actions, aligning knowledge updates directly with downstream task utility.
4. Empirical Validation: Extensive experiments across five datasets (Simple QA, Multi-hop QA, Conversation QA) demonstrate consistent downstream performance gains.

Experimental Results

The authors evaluated DeepRefine on Out-Of-Distribution (OOD) benchmarks using various knowledge base constructors (Naive, RL-tuned AutoGraph-R1, and Agent-compiled LLM-Wiki) and retrievers (Subgraph, ToG, HippoRAG, HippoRAG2).

• Performance Gains: DeepRefine (specifically the 8B variant) consistently improved RAG F1 scores across most configurations. Notably, it could sometimes outperform reconstruction methods (like AutoGraph-R1) even when applied to simpler, naive knowledge bases, suggesting that targeted refinement of specific regions is more effective than full regeneration which may introduce noise.
• Efficiency: DeepRefine is significantly more efficient than full reconstruction. On average, it consumed roughly half the time of the reconstruction method (e.g., ~3,200s vs ~6,780s for Simple QA) because it focuses only on problematic regions and uses code-based action generation rather than regenerating subgraphs.
• Ablation Studies:
  • RL Training: Models without RL training ("DeepRefine-8B w/o RL") showed marginal or negative improvements, confirming that the GBD reward is essential for learning effective refinement policies.
  • Model Scale: The 8B model outperformed the 4B model, indicating that larger models better leverage the interaction history for refinement.
  • Defect Types: In controlled benchmarks, DeepRefine was most effective at resolving incompleteness (adding missing evidence), moderately effective at incorrectness (fixing facts), and least effective at redundancy (coreference resolution), though it still outperformed the base model in all categories.

Significance and Claims

The paper claims that DeepRefine addresses a critical gap in the lifecycle of agent-compiled knowledge bases: the inability to efficiently correct systematic defects in deployed systems. By shifting the focus from construction to post-construction refinement, DeepRefine offers a scalable solution for maintaining high-quality knowledge bases.

The authors emphasize that their approach is generalizable (applicable to any pre-constructed KB) and data-efficient (requires only user queries and downstream task signals, not manual annotations of correct refinement steps). The work demonstrates that iterative, agent-driven refinement, guided by reinforcement learning and downstream utility signals, can significantly enhance the reliability of knowledge-intensive LLM applications.

The paper concludes modestly, acknowledging limitations such as the reliance on specific training data (HotpotQA) and the potential for further integration with robust systematic refinement frameworks.