FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization

1. Problem Statement

Current Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved Large Language Model (LLM) reasoning in domains like math and coding. However, standard RLVR pipelines rely on sparse, outcome-based rewards (e.g., binary correct/incorrect signals). This approach suffers from two critical flaws:

• Over-confidence and Hallucination: Models are incentivized to guess answers to maximize rewards, even when evidence is insufficient, leading to "lucky guesses" that are factually correct but logically unsound.
• Process-Outcome Mismatch: Scalar rewards obscure the validity of intermediate reasoning steps. A model may receive a positive reward for a correct final answer derived through irrelevant or hallucinated steps, or conversely, be penalized for a valid partial derivation that accidentally leads to a wrong final answer.
• Over-conservativeness: Methods attempting to reduce hallucinations often force models to refuse answering valid questions (over-refusal) to avoid the risk of being wrong.

The core problem is the lack of step-level supervision that ensures every reasoning step is strictly supported by provided evidence (faithfulness), rather than just optimizing for the final answer.

2. Methodology: FaithRL

The authors propose FaithRL, a general RL framework designed to maximize reasoning faithfulness. The framework consists of three core components:

A. Optimization Objective: Maximizing Reasoning Faithfulness

Instead of maximizing correctness (Objective A) or minimizing hallucinations (Objective B), FaithRL optimizes for Reasoning Faithfulness (Objective C).

• Definition: A trajectory is "faithful" if the set of knowledge items used in the reasoning process strictly matches the minimal required evidence set ($E(q)$) for the query.
• Theoretical Guarantee: The paper proves that optimizing for faithfulness avoids the degenerate behaviors of other objectives. It prevents over-confidence (guessing on unanswerable questions) and over-conservativeness (refusing answerable questions), stabilizing the model's refusal strategy based on its actual capability.

B. Geometric Reward Design ($R_{geo}$)

To align outcome-level optimization with the model's intrinsic capability, the authors introduce a Truthful Helpfulness Score (THS) based on a geometric interpretation.

• Concept: Model performance is mapped to a 2D plane where the x-axis is Correctness and the y-axis is Hallucination Rate. The "ideal" state is (1, 0).
• Mechanism: The reward is derived from the signed area of the triangle formed by the baseline performance, the current performance, and the ideal point.
• Adaptive Coefficients: The reward function assigns coefficients based on the baseline's initial performance ($x_0, y_0$):
  • $+y_0$ for correct answers (encourages accuracy).
  • $-x_0$ for hallucinations (penalizes errors).
  • $0$ for misses (refusals).
• Theorem: Maximizing this geometric reward is mathematically equivalent to maximizing the THS, ensuring the model improves both honesty and helpfulness simultaneously without arbitrary hyperparameters.

C. Faithfulness-Aware Advantage Modulation (FAAM)

To enforce step-level fidelity, FaithRL integrates a Verifier ($V$) that checks if each reasoning step is supported by the evidence.

• Mechanism: The standard Group Relative Policy Optimization (GRPO) advantage is modulated by a scalar $M_{i,t}$ for each token.
  • If a trajectory has a positive advantage (correct outcome), faithful steps receive full credit, while unfaithful steps are discounted.
  • If a trajectory has a negative advantage (incorrect outcome), unfaithful steps receive the full penalty, while faithful steps (valid reasoning that led to a wrong answer due to other factors) are protected from full penalty.
• Result: This prevents the model from being punished for valid reasoning chains that fail at the end and prevents it from being rewarded for lucky guesses with flawed logic.

3. Key Contributions

1. Novel Objective: Theoretical formulation of "Maximizing Reasoning Faithfulness" as a primary objective, proving it theoretically stabilizes refusal strategies and prevents degenerate behaviors.
2. Unified Framework: Introduction of FaithRL, the first work to mitigate hallucinations by directly optimizing reasoning faithfulness via objective evidence verification, combining geometric rewards with step-level modulation.
3. Geometric Alignment: A novel reward design that mathematically aligns with the Truthful Helpfulness Score (THS), balancing utility and honesty dynamically based on baseline capabilities.
4. Step-Level Credit Assignment: The FAAM mechanism effectively filters learning signals, ensuring the model learns from faithful reasoning regardless of the final outcome's correctness.

4. Experimental Results

The authors evaluated FaithRL on three multi-hop QA benchmarks (2WikiMultiHopQA, HotpotQA, MuSiQue) and two Out-of-Distribution (OOD) math benchmarks (GSM8k, MATH500).

• Performance Gains:
  • FaithRL achieved an average 4.7 point reduction in hallucination rates and a 1.6 point improvement in correctness compared to strong RLVR baselines (like GRPO and TruthRL).
  • It achieved a 16.0 point improvement in the Truthful Helpfulness Score (THS).
  • On OOD tasks (GSM8k, MATH500), FaithRL improved accuracy by 10.8 points and reduced hallucinations by 8.4 points compared to GRPO, demonstrating strong generalization.
• Faithfulness Evolution: During training, FaithRL increased the proportion of faithful reasoning steps by 31.1% (in-domain) and 6.5% (OOD), whereas baseline methods showed negligible improvement in step-level faithfulness.
• Efficiency: Despite adding step-level verification, FaithRL incurred only a ~13.7% increase in total GPU hours and 4.7% increase in FLOPs compared to GRPO, thanks to KV-cache reuse and efficient server concurrency.

5. Significance

FaithRL addresses a fundamental limitation in current LLM reasoning: the decoupling of "being right" from "being right for the right reasons." By shifting the optimization focus from the final outcome to the faithfulness of the reasoning process, the framework:

• Reduces Hallucinations: Forces models to rely strictly on provided evidence, reducing reliance on parametric memory or guessing.
• Improves Reliability: Creates models that are more trustworthy in high-stakes domains (healthcare, law) by ensuring they refuse to answer when evidence is insufficient, rather than guessing.
• Generalizes Robustly: The learned capability of faithful reasoning transfers effectively to unseen domains (math), suggesting that faithfulness is a general cognitive skill rather than a dataset-specific artifact.

The work lays a foundation for future research into interpretable and verifiable AI, moving beyond black-box optimization toward transparent, evidence-grounded reasoning.