Technical Summary: PathRouter

Problem Statement

Agentic Graph Retrieval-Augmented Generation (GraphRAG) systems utilize language model agents to iteratively retrieve and reason over graph-structured evidence. While these systems aim to improve decision-making by navigating complex information networks, existing training methods relying solely on outcome-based reinforcement learning (RL) suffer from two critical limitations:

1. Answer-Path Reward Aliasing: Standard RL assigns rewards based only on the final answer's correctness. This conflates trajectories where the correct answer is derived from useful evidence paths with those where the answer is retrieved from the model's parametric memory (shortcuts) despite failed evidence retrieval. Consequently, the model is reinforced for "lucky" successes that do not improve retrieval capabilities.
2. Search-Update Ambiguity: Scalar, trajectory-level feedback (e.g., a single reward score for the whole interaction) fails to indicate which specific retrieval actions (search queries or reasoning steps) contributed to the outcome. This leaves the agent without actionable guidance to adjust its search strategy when a trajectory fails or succeeds via shortcuts.

Methodology: PathRouter

The authors propose PathRouter, a path-aware training framework designed to align reward signals with retrieval quality. The framework operates through three core mechanisms:

1. Path-Aware Routing and Differentiated Advantage Scaling

Instead of treating all trajectories uniformly, PathRouter evaluates each trajectory along two axes:

• Answer Correctness ($C_i$): Whether the final answer is correct.
• Evidence-Path Overlap ($P_i$): The token-level F1 overlap between the retrieved passages and the gold supporting evidence.

Based on these metrics, trajectories are categorized into four types, each receiving a differentiated weight for Group Relative Policy Optimization (GRPO) advantage scaling:

• Faithful Success ($C \uparrow, P \uparrow$): Correct answer with high evidence overlap. Receives full weight.
• Shortcut Failure ($C \uparrow, P \downarrow$): Correct answer but low evidence overlap (relying on parametric memory). Receives reduced weight to suppress the reinforcement of shortcut behaviors.
• Evidence Retrieved ($C \downarrow, P \uparrow$): Incorrect answer but high evidence overlap (reasoning failure). Receives attenuated weight to avoid over-penalizing useful retrieval behavior.
• Joint Failure ($C \downarrow, P \downarrow$): Incorrect answer with low evidence overlap. Receives full weight.

This routing ensures policy updates reflect retrieval quality rather than just final answer correctness.

2. Selective Gold-Evidence Teacher for Search-Update Ambiguity

To resolve the ambiguity of which search actions to adjust, PathRouter introduces a frozen gold-evidence teacher for evidence-poor trajectories ($P_i < \theta_P$).

• Mechanism: The teacher is prompted with the query and the gold supporting passages (privileged context). It does not generate a new trajectory but scores the student's on-policy rollout.
• Token-Level KL Guidance: The teacher provides Kullback-Leibler (KL) divergence supervision specifically on reasoning tokens and search-query tokens.
• Exclusion: Answer tokens and retrieved observation blocks are excluded from KL supervision to prevent direct response imitation and ensure the agent learns to find evidence rather than just output the answer.
• Scheduling: A linear warmup is applied to the KL coefficient to allow the student to explore via GRPO rewards before the teacher constraint takes effect.

3. Training Objective

The total loss combines the routed GRPO loss with the selective teacher KL loss:
$$\mathcal{L} = \mathcal{L}_{GRPO} + \lambda_T(s) \cdot \mathcal{L}_{TKL}$$
The reward function also incorporates exploration shaping (bonuses for multi-turn retrieval, penalties for lazy stops, timeouts, and redundancy) to encourage thorough evidence gathering.

Key Contributions

1. Identification of Reward Aliasing: The paper identifies answer-path reward aliasing as a primary failure mode in agentic GraphRAG and proposes a route-conditioned GRPO advantage scaling mechanism to decouple answer correctness from retrieval quality.
2. Selective Token-Level Distillation: A novel selective teacher mechanism is introduced that applies token-level KL guidance to reasoning and search-query tokens for evidence-poor trajectories. This resolves search-update ambiguity without imitating final answers.
3. Empirical Validation: Extensive experiments demonstrate that PathRouter consistently improves both answer accuracy and evidence-path overlap across six QA benchmarks and three model sizes (1.5B, 3B, 7B).

Experimental Results

Evaluated on six benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, NQ, PopQA, TriviaQA) using Qwen2.5 models:

• Performance Gains: PathRouter consistently outperforms strong baselines, including Graph-R1.
  • On 7B models, it achieved an average F1 gain of 4.9 points (from 57.82 to 62.74) and an Exact Match (EM) gain of 4.69 points.
  • On 3B models, it achieved an average F1 gain of 3.1 points.
  • Improvements were most pronounced on multi-hop reasoning tasks (e.g., +7.4 F1 on HotpotQA, +8.2 on MuSiQue for 7B models).
• Trajectory Shift: Analysis of trajectory distributions shows PathRouter successfully shifts mass from "shortcut" and "joint failure" categories toward "faithful success" and "evidence-present reasoning failure," indicating improved evidence acquisition independent of answer correctness.
• Generalization: Cross-dataset transfer tests revealed high OOD generalization (average ratio of 95.7%), suggesting the learned retrieval strategies are robust and not tied to specific dataset distributions.
• Ablation Findings:
  • Removing the path reward ($r_p$) significantly degraded evidence quality (SF-F1) while retaining answer F1, confirming the signal targets retrieval.
  • Removing the selective teacher KL reduced performance, particularly on multi-hop tasks, confirming its role in resolving search-update ambiguity.
  • Teacher KL effectiveness was found to be capacity-dependent; smaller models (1.5B) showed mixed results or degradation with a stronger teacher, suggesting student capacity is a limiting factor.

Significance and Claims

The paper claims that PathRouter addresses the fundamental credit assignment limitations of outcome-only reinforcement learning in agentic systems. By jointly evaluating answer correctness and evidence-path overlap, the framework ensures that policy updates are driven by the quality of the retrieval process, not just the final output.

The authors assert that this approach enables agents to learn retrieval policies that are both answer-accurate and evidence-faithful. The work suggests that path-aware training yields generalizable retrieval behaviors that avoid dataset-specific shortcuts, offering a more robust foundation for complex reasoning tasks in GraphRAG systems. The paper acknowledges limitations regarding the additional hyperparameters required for routing and scheduling, as well as the increased computational cost per step due to teacher forward passes, noting a trade-off between training efficiency and improved faithfulness.