💬 NLP

FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision

FARCA is a policy optimization framework that mitigates hallucination in reinforcement learning with factual supervision by decomposing noisy credit assignment into localization and reliability ambiguities, thereby generating fine-grained, reliability-weighted token-level training signals that improve model factuality without compromising general reasoning capabilities.

Original authors: Qiming Xie, Wenjie Zheng, Xiangqing Shen, Rui Xia

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qiming Xie, Wenjie Zheng, Xiangqing Shen, Rui Xia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, large language models have become powerful tools for reasoning, capable of solving complex problems by generating step-by-step explanations. These systems are often trained using a method called reinforcement learning, where the computer learns by trial and error, receiving a reward only when it produces a correct final answer. While this approach has unlocked impressive capabilities, it carries a hidden risk: the model might arrive at the right conclusion by weaving together a story filled with made-up facts. Because the training system only checks the final result, it inadvertently teaches the model that fabricating details is a valid strategy, as long as the ending is correct. This phenomenon, known as hallucination, undermines the reliability of these systems, especially when they are asked to answer questions that require precise knowledge of the real world.

To fix this, researchers have begun introducing a new layer of supervision that checks the facts within the reasoning process itself, not just the final answer. However, a new study reveals that simply adding these fact-checks is not enough. If the system cannot pinpoint exactly which words in a sentence were true and which were false, or if it blindly trusts every fact-checker it uses, the training can actually become noisier and less effective. The researchers found that existing methods often treat a whole sentence or a reasoning step as a single unit, assigning the same credit or blame to every word within it. This creates a mismatch where a correct fact and a hallucinated one might share the same reward signal, confusing the model. Furthermore, the tools used to verify facts are not perfect; they can be misled by language tricks or irrelevant information, yet current training methods treat their judgments as absolute truth.

To solve these problems, a team of researchers from Nanjing University of Science and Technology and Nanjing University developed a new framework called FARCA. Their approach is designed to align the granularity of fact-checking with the way the model learns. Instead of judging an entire sentence at once, the system breaks the model's reasoning down into small, independent claims, or atomic facts. For each of these tiny claims, the system traces it back to the specific words in the text that generated it. This process, which the authors call token provenance, ensures that the model receives a reward or penalty only for the specific words that correspond to a true or false statement. If a sentence contains one correct fact and one incorrect one, the system can now reward the correct part while penalizing the incorrect part, rather than blurring the two together.

The researchers also addressed the issue of unreliable fact-checkers. They realized that not all fact-checking signals are equally trustworthy. Some judgments are based on clear, identifiable evidence, while others might be guesses driven by the model's language patterns. To handle this, FARCA uses a technique called counterfactual evidence attribution. Before accepting a fact-check, the system asks a simple question: if we removed the most important piece of evidence supporting this fact, would the fact-checker change its mind? If the answer is yes, the judgment is considered highly reliable. If the fact-checker gives the same answer even without the key evidence, the system treats that judgment as weak and reduces its influence on the training process. This allows the model to learn from strong, evidence-backed corrections while ignoring noisy or misleading signals.

The team tested this new method on two different language models using a variety of challenging datasets that require deep knowledge and reasoning. They compared their results against several existing methods that use factual supervision. The findings showed that the new framework significantly improved the models' ability to stick to the facts without sacrificing their ability to solve complex problems. On benchmarks designed to measure how often models make things up, the new method outperformed all previous approaches, reducing hallucinations by a noticeable margin. For instance, on one test measuring truthfulness in general knowledge, the improvement was substantial, with the new method achieving scores that were several percentage points higher than the next best system. Crucially, the models also maintained or even improved their performance on mathematical reasoning tasks, proving that making the training more fact-focused did not make the models worse at logic.

The study suggests that the key to training reliable AI lies in precision and skepticism. By pinpointing exactly where a fact lives in a sentence and by questioning the strength of the evidence behind every fact-check, the researchers created a training environment that is both more accurate and more robust. Their work demonstrates that the path to better artificial intelligence is not just about checking more facts, but about checking them in a way that respects the complexity of language and the limitations of verification tools. This approach offers a clearer, more stable way to guide these powerful systems, ensuring that when they reason, they do so with a foundation of truth rather than a collection of plausible-sounding lies.

Technical Summary: FARCA – Fact-Aligned Reliability-Aware Credit Assignment

Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has enhanced the reasoning capabilities of Large Language Models (LLMs), outcome-driven rewards often fail to penalize factually flawed reasoning that coincidentally leads to correct answers. This phenomenon amplifies hallucinations in knowledge-intensive tasks.

Recent mitigation strategies introduce process-level factual supervision but suffer from a critical bottleneck termed noisy factual credit assignment. This noise arises from two distinct ambiguities:

Credit Localization Ambiguity: A granularity mismatch exists between fact verification (typically at the atomic fact or reasoning-step level) and policy updates (per-token). When a reasoning trajectory contains mixed factuality, aggregating verification signals into a single scalar for a step or trajectory causes tokens associated with correct facts and hallucinated content to share the same credit signal, blurring the correspondence between verification outcomes and their source tokens.
Credit Reliability Ambiguity: Existing methods often treat verifier outputs (e.g., from Natural Language Inference models) as ground truth without assessing their reliability. Verifiers can be misled by linguistic priors or irrelevant spans, producing misjudgments that contaminate policy gradients.

Methodology: FARCA Framework

The authors propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework designed to transform factual supervision into localized, reliability-weighted token-level training signals.

1. Atomic Fact Extraction and Token Provenance

To resolve localization ambiguity, FARCA decomposes reasoning text into independently verifiable atomic facts.

Decomposition: Sentences are segmented and filtered to remove non-verifiable content (e.g., greetings). GPT-4o is prompted to break sentences into semantically self-contained claims, performing decontextualization where necessary.
Token Provenance: For each atomic fact, the system traces it back to its specific source token span in the original generation. Rule-based checks correct for issues like repeated entities or pronoun rewriting to ensure accurate mapping.
Continuous Scoring: Instead of discrete labels, a factual verifier (NLI-based) provides a continuous score $h \in [0, 1]$ , mapped to a signed factual score $r \in [-1, 1]$ .

2. Counterfactual Evidence Attribution

To resolve reliability ambiguity, FARCA estimates the reliability of each verification signal using counterfactual evidence attribution.

Mechanism: For each atomic fact, the system identifies the most relevant evidence sentences. It then constructs a counterfactual evidence set by removing these key sentences and re-verifying the fact.
Reliability Weight: The magnitude of the score change ( $\Delta$ ) between the original and counterfactual verification serves as a proxy for reliability. A large $\Delta$ indicates the judgment depends on identifiable evidence (high reliability), while a small $\Delta$ suggests the judgment relies on linguistic priors (low reliability).
Weighting: This dependency strength is mapped to a continuous reliability weight $w \in (0, 1)$ via a sigmoid function.

3. Reward Design and Advantage Reshaping

FARCA integrates these signals into the training objective:

Rewards: The total reward combines format constraints, answer correctness, and a reliability-weighted factual reward (the average of reliability-weighted factual scores).
Advantage Reshaping: The framework performs a soft interpolation between the original Group Relative Policy Optimization (GRPO) advantage and a fact-corrected advantage.
- The fact-corrected advantage uses the signed factual score to determine the update direction for specific tokens.
- The reliability weight $w$ modulates this: high $w$ shifts the advantage toward the fact-corrected direction; low $w$ retains the original advantage, preventing unreliable signals from reversing gradients.
Token-Level Update: The final advantage for each token is the average of the reliability-aware advantages of all atomic facts covering that token.

Key Contributions

Problem Identification: The paper identifies "noisy factual credit assignment" as a primary bottleneck in RL with factual supervision, decomposing it into credit localization and credit reliability ambiguities.
Framework Proposal: FARCA is introduced as a unified framework that aligns fact verification granularity with policy updates (via token provenance) and models signal reliability (via counterfactual evidence attribution).
Empirical Validation: Experiments demonstrate that FARCA significantly improves model factuality while preserving general reasoning capabilities, outperforming existing factual RL baselines.

Experimental Results

The authors evaluated FARCA on Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct using knowledge-intensive datasets (HotpotQA, 2WikiMultiHopQA) and mathematical reasoning datasets (SimpleRL).

Hallucination Benchmarks: FARCA achieved the best results across four benchmarks (SimpleQA, TruthfulQA, HalluQA, HaluEval-QA). On Qwen2.5-3B, it improved average performance by 1.75 percentage points over the strongest baseline (FaithRL), with specific gains of 2.09 and 2.67 points on TruthfulQA and HalluQA, respectively.
Reasoning Capabilities: Unlike some baselines that sacrifice reasoning for factuality, FARCA also delivered the best performance on mathematical reasoning benchmarks (AIME2026, AIME2025, MATH-500, GSM8K).
Ablation Studies: Removing token provenance (broadcasting signals to all tokens) or reliability estimation (setting weights to 1) resulted in performance degradation, confirming the necessity of both fine-grained localization and reliability modeling.
Analysis: Further analysis showed that ~98.3% of extracted facts were successfully aligned to token spans and that counterfactual attribution produced stable, non-degenerate reliability weights (average $w \approx 0.51$ ), effectively down-weighting unreliable signals.

Significance and Claims

The paper claims that FARCA addresses the fundamental mismatch between how facts are verified and how policies are updated. By transforming noisy, coarse-grained verification signals into fine-grained, reliability-aware credit, FARCA enables factual supervision to guide policy optimization more faithfully.

The authors assert that effective factual optimization requires not just the presence of factual signals, but their precise localization to the correct tokens and reliable weighting based on evidence dependence. FARCA demonstrates that these factors are critical for mitigating hallucinations without compromising the model's general reasoning abilities, offering a more robust path for training factually grounded LLMs.