Technical Summary: Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

Problem Statement

Recent advancements in Retrieval-Augmented Generation (RAG) have shifted toward "agentic search," where Large Language Models (LLMs) are trained via Reinforcement Learning (RL) to dynamically plan, query search engines, and reason over retrieved evidence. While existing methods (e.g., Search-R1, ReSearch) achieve strong performance on final-answer correctness (Exact Match) using episode-level outcome-based rewards, they suffer from a critical limitation: chain-of-thought unfaithfulness.

These models often optimize solely for the final answer, overlooking the quality and integrity of intermediate reasoning steps. This leads to scenarios where the final answer is correct, but the intermediate steps (thoughts, search queries, and evidence synthesis) are logically disconnected, ignore retrieved evidence, or hallucinate information. The paper identifies a gap in the field: a lack of formal definitions and evaluation frameworks for reasoning faithfulness in the context of agentic search, and the absence of training mechanisms that incentivize faithful intermediate traces without sacrificing task performance.

Methodology

1. Evaluation Framework for Faithfulness

The authors propose a comprehensive evaluation framework centered on three distinct, turn-level faithfulness metrics that assess the logical flow of the agentic search trajectory:

• Think-Search Faithfulness: Assesses whether a search query is a direct and necessary consequence of the agent's preceding reasoning (i.e., "searching what it thinks it needs").
• Information-Think Faithfulness: Evaluates whether the agent's reasoning block following a retrieval step properly synthesizes, summarizes, or deduces from the newly acquired evidence, ensuring the model does not ignore or contradict retrieved information.
• Think-Answer Faithfulness: Measures whether the final answer is entailed by the cumulative knowledge and reasoning in the preceding thought block, preventing the introduction of unverified information at the final step.

These metrics are operationalized using a combination of Natural Language Inference (NLI) models and LLM-as-a-Judge (LLMaaJ) approaches.

2. VERITAS Framework

To address the identified gaps, the paper introduces VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search). This is a training framework that integrates fine-grained, structured faithfulness rewards into the RL training loop (specifically using PPO).

• Reward Design: The total reward function ($R$) is a weighted sum of the traditional outcome-based reward ($R_{EM}$) and structured faithfulness rewards ($R_{info-think}$, $R_{think-answer}$, $R_{think-search}$).
  $$R = w_{EM} \cdot R_{EM} + w_{info-think} \cdot R_{info-think} + w_{think-search} \cdot R_{think-search} + w_{think-answer} \cdot R_{think-answer}$$
  The authors empirically found that $R_{think-search}$ could be omitted ($w=0$) as baseline models already exhibited high faithfulness in this dimension, allowing for a streamlined training process.
• Distilled Reward Model: To overcome the latency and cost of using LLMaaJ for on-policy RL, the authors train a smaller, distilled Reward Model (RM) using LoRA fine-tuning on a Qwen2.5-14B-Instruct model. This RM is trained to replicate the judgments of a larger LLM judge (Claude-3.7-Sonnet) regarding Information-Think faithfulness.
• Curriculum Learning: To prevent "reward hacking" (where the model over-optimizes for faithfulness at the expense of exploration and task accuracy), the authors employ a curriculum strategy. The faithfulness rewards are initially set to zero and linearly warm up over the first 1.5 epochs of training, allowing the model to first learn basic task performance before refining its reasoning fidelity.

Key Contributions

1. Formal Definition and Evaluation: The paper establishes the first formal evaluation framework for faithfulness in agentic search, defining and quantifying three specific dimensions: Think-Search, Information-Think, and Think-Answer faithfulness.
2. Empirical Analysis of Baselines: Through extensive evaluation of leading RL-based search agents (Search-R1, ReSearch, DeSA), the authors reveal a critical disconnect: high task accuracy does not guarantee high reasoning faithfulness. Specifically, models trained with pure outcome rewards often fail to integrate retrieved evidence into their reasoning (low Information-Think faithfulness).
3. VERITAS Training Paradigm: The authors propose and validate VERITAS, a method that incorporates structured, checklist-style faithfulness rewards into RL training. This approach successfully improves reasoning faithfulness while maintaining or improving final task accuracy.

Experimental Results

The authors evaluated VERITAS-R1 (Search-R1 trained with the VERITAS framework) against baselines across seven QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle).

• Faithfulness Improvements: VERITAS-R1 significantly improved faithfulness metrics. For instance, on general QA datasets, Information-Think faithfulness increased from 0.734 (baseline) to 0.853. Think-Answer faithfulness also saw notable gains on multi-hop datasets.
• Task Performance: Contrary to the hypothesis that enforcing faithfulness might constrain performance, VERITAS-R1 achieved higher Exact Match (EM) scores than the baseline Search-R1. On multi-hop datasets, the average EM improved from 0.361 to 0.380.
• Ablation Studies:
  • Reward Components: The Information-Think reward was found to be the most stable driver of both faithfulness and accuracy improvements.
  • Curriculum Learning: Training without the curriculum (applying faithfulness rewards immediately) led to reward hacking and lower final EM scores, confirming the necessity of the warm-up strategy.
  • Reward Model: The distilled Qwen2.5-14B RM showed high consistency (0.910) with human annotations and the larger judge model, validating its use as a scalable proxy.

Significance and Claims

The paper claims that rewarding the reasoning process does not trade off with performance; instead, it creates a positive synergy. By shifting from purely outcome-based rewards to structured, fine-grained faithfulness rewards, the authors demonstrate that it is possible to develop agents that are not only more accurate but also more transparent and reliable.

The work underscores the value of structured supervision over intermediate reasoning steps. It suggests that as outcome-only rewards saturate, the next axis of improvement for agentic systems lies in investing compute in richer, finer-grained evaluation signals (rubrics/checklists) and propagating these signals back into the training loop. The authors position VERITAS as a step toward developing more trustworthy AI agents that can be deployed in scenarios requiring rigorous evidence synthesis and logical consistency.