Technical Summary: Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflicts (RAPS-DA)

1. Problem Statement

Retrieval-Augmented Generation (RAG) enhances language models by grounding generation in external context. However, RAG systems face a "trust–resistance dilemma" when the retrieved context conflicts with the model's parametric knowledge (PK). Existing robustness methods often treat knowledge conflict as a homogeneous phenomenon, applying regime-agnostic supervision across a spectrum of reliability. This approach leads to conflicting optimization signals:

• Grounding: Reliable context should be integrated.
• Arbitration: Mixed-quality or contradictory evidence requires selective trust.
• Resistance: Adversarial or counterfactual context must be rejected in favor of PK.

Current solutions, including prompting, decoding strategies, and single-policy training (Supervised Fine-Tuning or Reinforcement Learning), fail to disentangle these incompatible behaviors. They either lack trainable adaptation mechanisms or impose a single training signal that causes cross-regime interference, leading to systematic underfitting across all reliability regimes. Furthermore, naive on-policy distillation (OPD) using a single teacher inherits these interference issues, while multi-teacher settings without careful routing introduce supervision noise and instability.

2. Methodology: RAPS-DA

The authors propose RAPS-DA (Regime-Aware Peer Specialization with Difficulty Annealing), a training framework designed to handle heterogeneous knowledge conflicts at both the sample and token levels. The framework utilizes three same-scale peer teachers trained from a shared base model, ensuring improvements stem from specialization rather than increased model capacity.

2.1 Regime-Routed Peer Supervision (Sample Level)

To address sample-level regime heterogeneity, the training data is partitioned into three regimes: Grounding (G), Arbitration (A), and Resistance (R).

• Peer Specialization: Three peer teachers ($\pi_{T_r}$) are fine-tuned via Supervised Fine-Tuning (SFT) on regime-specific subsets derived from a shared base model ($\pi_{base}$).
• Hard Routing: Each training sample is hard-routed to its regime-matched teacher. This eliminates the contradictory gradient updates that occur when a single teacher attempts to learn conflicting behaviors simultaneously.
• On-Policy Reverse-KL: The student generates on-policy rollouts and receives supervision via reverse-Kullback-Leibler (KL) divergence against the routed teacher. This is reformulated as a dense token-level reward maximization problem, where the base model serves as an implicit KL anchor. The implicit reward for a token $a_t$ is defined as $r^{(r)}_t = \log \frac{\pi_{T_r}(a_t|s_t)}{\pi_{base}(a_t|s_t)}$.

2.2 Conflict-Aware Token Selection (Token Level)

To address token-level supervision noise, a dual-layer selector filters and weights tokens based on three diagnostic signals:

1. Inter-teacher Conflict ($C(t)$): Jensen-Shannon divergence among all peer teachers, identifying tokens at the core of knowledge conflict.
2. Student–Teacher Gap ($G(t)$): Reverse-KL between the student and the routed teacher, measuring the need for correction.
3. Student Entropy ($H(t)$): Normalized entropy of the student's distribution, indicating predictive uncertainty.

The selection process involves:

• Hard Mask (Eligibility): Filters out tokens where the student is already aligned (low gap) or tokens where the student is highly uncertain and strongly disfavored by the teacher (indicating off-manifold rollouts). Crucially, it retains "confident-but-misaligned" tokens (low entropy, high gap) as they represent valuable correction opportunities.
• Soft Weight (Priority): Among eligible tokens, a weight $\omega(t, s)$ up-weights tokens where the student is confident but misaligned with the teacher, prioritizing the correction of entrenched errors.

2.3 Difficulty Annealing

To stabilize the interaction between routing and token selection, the framework employs a difficulty annealing schedule.

• Rationale: Starting training with only high-conflict tokens can hinder the student from building stable, regime-consistent foundations.
• Mechanism: The training begins with broad token coverage (retention ratio $p(s)=1$) and progressively narrows to the top $p_{final}$ most difficult tokens as training proceeds. Difficulty is scored based on conflict intensity and the interaction between conflict and student confidence.

2.4 Unified Objective

The final training objective aggregates the weighted, masked, reverse-KL loss across all regimes. The student is a single model that requires no regime labels or access to peer teachers during inference.

3. Key Contributions

1. Characterization of Conflict: The paper characterizes knowledge conflict in RAG as a spectrum of reliability regimes (Grounding, Arbitration, Resistance) and demonstrates that regime-agnostic supervision induces conflicting optimization signals.
2. RAPS-DA Framework: A novel framework combining regime-routed peer supervision, conflict-aware token selection, and difficulty annealing to address heterogeneous conflicts at both sample and token levels.
3. Empirical Validation: Extensive experiments across five in-distribution conflict scenarios and two out-of-distribution (OOD) benchmarks show that RAPS-DA outperforms prompting, decoding, fine-tuning, RL, and single-teacher OPD baselines.

4. Experimental Results

Experiments were conducted on Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct backbones, with generalization tested on 3B and 14B scales.

• In-Distribution Performance: RAPS-DA achieved state-of-the-art results across all regimes. On the Qwen-7B backbone, it improved over the strong Knowledgeable-R1 (KR1) baseline by 7.4 points in Grounding, 6.6 in Arbitration, and 11.2 in Resistance. Notably, it improved Resistance (adversarial rejection) by 11.2 points over KR1 without sacrificing Grounding accuracy, a trade-off often seen in other methods.
• Out-of-Distribution Generalization: On the ConflictQA benchmark, RAPS-DA achieved 42.7% Exact Match (EM), outperforming all baselines. On TruthfulQA (no-context), it maintained 4.4% EM, matching the base model and confirming that regime-specialized training does not erode parametric knowledge.
• Ablation Studies:
  • Single-Teacher vs. Multi-Teacher: A single-teacher OPD baseline (OPCD) suffered significant drops in Arbitration (37.3% vs. 78.6%) and Resistance (31.0% vs. 48.1%), validating the need for regime-specialized teachers.
  • Routing vs. Selection: Neither routing nor token selection alone achieved the full model's performance, confirming their complementary nature.
  • Annealing: Static masking or sliding-window schedules (hard-to-easy) performed significantly worse than the linear easy-to-hard annealing schedule, particularly on the Resistance regime, highlighting the necessity of broad initial coverage.
• Scale Generalization: The method yielded consistent improvements on both 3B and 14B models, with the Resistance regime showing the largest relative gains across all scales.

5. Significance and Claims

The paper claims that robust conflict resolution in RAG is fundamentally a specialization problem rather than a scaling problem. By decomposing conflict into distinct reliability regimes and assigning same-scale peer specialists to each, RAPS-DA avoids the cross-regime interference inherent in single-policy training.

The authors emphasize that the performance gains originate from the quality of regime-specialized supervision and the targeted token-level learning signals, not from using a larger or stronger teacher. The framework demonstrates that a single student model can learn to integrate, arbitrate, and reject evidence appropriately without requiring regime labels at inference time, approaching the performance of an oracle regime-specialist upper bound. This suggests that future robust RAG systems should prioritize disentangling conflicting learning signals over simply increasing model capacity or applying uniform supervision.