💬 NLP

CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models

The paper introduces CARE-RFT, a novel reinforcement finetuning method that employs skew reverse KL divergence to resolve the trade-off between reasoning performance and model trustworthiness, achieving the high reasoning capabilities of unconstrained RFT while preserving the calibration and reliability of the base model.

Original authors: Shuozhe Li, Jincheng Cao, Bodun Hu, Aryan Mokhtari, Leqi Liu, Amy Zhang

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuozhe Li, Jincheng Cao, Bodun Hu, Aryan Mokhtari, Leqi Liu, Amy Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, well-read student (the AI model) how to solve complex puzzles. You want them to become a master reasoner, capable of thinking step-by-step and finding creative solutions. However, you also want them to remain honest and not make up facts when they aren't sure.

This paper, CARE-RFT, tackles a specific problem that arises when we try to train these students using a method called "Reinforcement Finetuning" (RFT).

The Problem: The "Overconfident Dreamer" vs. The "Cautious Bureaucrat"

The authors found that current training methods force you to choose between two bad options:

The Unconstrained Approach (The Overconfident Dreamer):
If you let the student learn purely by trial and error without strict rules, they get very good at solving hard puzzles. They start thinking outside the box! But, they also start hallucinating. They become so confident in their answers that they will confidently invent fake facts or lie about things they don't know. They lose their "calibration," meaning their confidence doesn't match their actual accuracy.
- Analogy: It's like a student who memorizes the answer key to a math test but doesn't understand the math. If the test changes slightly, they just guess wildly and confidently, getting the right answer by luck but failing to be reliable.
The RKL-Constrained Approach (The Cautious Bureaucrat):
To stop the lying, researchers usually add a "safety leash" called Reverse KL (RKL). This forces the student to stay very close to their original, pre-trained personality. This keeps them honest and factual.
- The Catch: The leash is too tight. It punishes the student for trying anything new or different. Because they are afraid to deviate from the "safe" path, they stop learning how to solve hard, complex puzzles. They stay honest, but they stop being smart.

The Solution: CARE-RFT (The "Confidence-Anchored" Coach)

The authors propose a new method called CARE-RFT. Think of this as a smart coach who knows exactly when to loosen the leash and when to pull it tight.

Instead of a single, rigid rule, CARE-RFT uses a special tool called Skew Reverse KL. Here is how it works using a simple metaphor:

The "Anchor" Concept: Imagine the student is a boat, and the original, well-trained model is a heavy anchor.
Standard RKL (The Old Way): The anchor is a giant, unbreakable chain. If the boat tries to drift even a little bit into new, uncharted waters (exploring new reasoning paths), the chain pulls it back violently. The boat stays safe but never explores.
Unconstrained RFT (The Other Way): The anchor is cut loose. The boat can go anywhere, but it might crash into rocks (hallucinations) or drift off a cliff (miscalibration).
CARE-RFT (The New Way): The anchor is a smart, adjustable tether.
- When the student is unsure: If the boat is drifting into foggy, uncertain waters, the tether pulls hard, keeping the student grounded in the safe, factual knowledge of the base model. This prevents hallucinations.
- When the student is confident and rewarded: If the student tries a new path and it works (they get a high reward), the tether relaxes. It allows the student to drift further out to explore and learn complex reasoning, but only because they have proven they are on the right track.

What Happens When You Use CARE-RFT?

The paper ran experiments on different AI models (like Qwen2.5) and found that CARE-RFT achieves the "best of both worlds":

It keeps the "Dreamer" smarts: The models became just as good at solving hard math and reasoning problems as the unconstrained, hallucination-prone models.
It keeps the "Bureaucrat" honesty: The models stayed just as trustworthy and factually accurate as the cautious, leash-wearing models. They stopped making up facts and their confidence levels matched their actual performance.

The "Entropy" Secret

The authors also looked at the "entropy" (a measure of uncertainty) of the models during training.

The Unconstrained models became "brittle"—their uncertainty collapsed to zero too fast, making them overconfident and prone to errors.
The RKL models stayed too "fuzzy" and unsure to learn hard tasks.
CARE-RFT found a "Goldilocks" zone. It allowed the models to become confident enough to solve hard problems, but not so confident that they stopped questioning themselves.

Summary

In short, CARE-RFT is a new training technique that acts like a smart safety net. It allows Large Language Models to explore new, complex reasoning strategies without losing their grip on reality. It proves that you don't have to choose between being a brilliant reasoner and a trustworthy fact-checker; with the right "confidence-anchored" approach, you can be both.

Technical Summary: CARE-RFT

Problem Statement

Reinforcement Finetuning (RFT) has emerged as a pivotal paradigm for unlocking emergent reasoning capabilities in Large Language Models (LLMs), exemplified by models like DeepSeek-R1-Zero. However, the authors identify a critical trade-off between reasoning performance and model trustworthiness:

Unconstrained RFT: While algorithms like Group Relative Policy Optimization (GRPO) without constraints achieve superior reasoning accuracy, they severely compromise model reliability. The removal of constraints leads to "indiscriminate reinforcement," where the model reinforces entire chains of thought (including logically flawed intermediate steps) if the final outcome is correct. This results in increased hallucination rates, degraded factuality on retrieval benchmarks, and poor calibration (high Expected Calibration Error, or ECE).
RKL-Constrained RFT: To mitigate hallucinations, standard approaches impose a Reverse Kullback–Leibler (RKL) divergence penalty to keep the trained policy ( $\pi_\theta$ ) close to a reference policy ( $\pi_{ref}$ ). While this preserves calibration and factuality, the RKL penalty is unbounded. It imposes infinite penalties on tokens where the reference model assigns low probability, effectively stifling the exploration of novel, high-reward reasoning paths that deviate from the base model. Consequently, reasoning gains are limited compared to unconstrained RFT.

The core challenge is to design a regularization method that allows for the exploration necessary for reasoning while preventing the unanchored deviations that cause hallucinations and miscalibration.

Methodology: CARE-RFT

The authors propose CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), which replaces the standard RKL penalty with a Skew Reverse KL (SRKL) divergence.

Theoretical Foundation

Standard RKL divergence ( $D_{RKL}$ ) penalizes deviations where $\pi_\theta$ assigns probability to tokens that $\pi_{ref}$ deems unlikely. This penalty grows unbounded as $\pi_{ref}(o_t) \to 0$ , preventing the model from exploring low-probability but potentially correct reasoning strategies.

CARE-RFT introduces a skew parameter $\alpha \in (0, 1)$ to define a mixture distribution. The SRKL divergence is defined as:
$D_{SRKL}^\alpha (\pi_{ref} \| \pi_\theta) = \sum_{o_t \in V} \pi_\theta(o_t) \log \frac{\pi_\theta(o_t)}{\alpha \pi_\theta(o_t) + (1-\alpha)\pi_{ref}(o_t)}$

Key Mechanisms

Confidence-Anchored Penalty: The SRKL penalty is confidence-sensitive.
- Upward Deviations (Exploration): When the model is confident ( $\pi_\theta$ is high) and consistently rewarded, the penalty for increasing probability on a token is bounded. The term $\log(1/\alpha)$ serves as a finite upper bound. This allows the model to accumulate probability mass on novel, high-reward reasoning patterns without being suppressed by the reference model's low probability.
- Downward Deviations (Preservation): When the model attempts to reduce probability on tokens that the reference model favors (or when the model is uncertain), the penalty remains unbounded (diverging to $-\infty$ as the ratio approaches 0). This ensures the model does not drift away from the well-calibrated priors of the base model, preserving factual grounding.
Gradient Stability: The gradient coefficient of SRKL is bounded for upward moves, mitigating the "gradient blow-up" and estimator variance often seen with standard RKL when the reference model assigns near-zero probability to sampled tokens.

Experimental Setup

The authors evaluated CARE-RFT across multiple model scales (Qwen2.5-3B and 7B) and three distinct RFT algorithms:

GRPO: The foundational group relative policy optimization.
DAPO: An aggressive, constraint-free variant.
GSPO: A recent sequence-level optimization approach.

Benchmarks:

Reasoning: MATH and GSM8K (measured by Pass@4).
Trustworthiness/Factuality: TruthfulQA and SelfAware (measured by Pass@1).
Calibration: Expected Calibration Error (ECE) on TruthfulQA.

Key Results

The experiments demonstrate that CARE-RFT successfully breaks the accuracy–calibration trade-off:

Reasoning Performance: CARE-RFT matches the reasoning performance of unconstrained RFT. For instance, on Qwen2.5-3B using GRPO, CARE-RFT achieved a MATH score of 0.600, nearly identical to the unconstrained version (0.610) and significantly higher than RKL-constrained GRPO (0.510).
Trustworthiness and Calibration: Unlike unconstrained RFT, which suffered severe drops in TruthfulQA accuracy and high ECE, CARE-RFT recovered the trustworthiness of the base model.
- TruthfulQA: CARE-GRPO scored 0.465, close to the base model (0.489) and RKL-GRPO (0.480), whereas unconstrained GRPO dropped to 0.350.
- ECE: CARE-RFT maintained low ECE (0.132), comparable to RKL-constrained methods (0.125) and significantly better than unconstrained methods (0.210).
Training Dynamics: Analysis of token entropy revealed that unconstrained RFT leads to a catastrophic collapse in entropy (overconfidence), while RKL maintains high entropy but limits sharpening. CARE-RFT achieves a controlled entropy reduction, allowing the model to sharpen distributions for reasoning without collapsing into overconfidence.

Significance and Claims

The paper claims that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.

Resolution of Trade-off: CARE-RFT resolves the tension between the need for exploration (to unlock reasoning) and the need for constraint (to prevent hallucination) by introducing a one-sided, bounded penalty.
Generalizability: The method is presented as a "drop-in" regularization layer compatible with various RFT algorithms (GRPO, DAPO, GSPO) and model scales.
Mechanistic Insight: The work establishes that the failure of unconstrained RFT is not merely a lack of constraints, but specifically the indiscriminate nature of outcome-level rewards. By shifting to token-level, confidence-anchored constraints, the model can learn complex reasoning patterns without sacrificing the factual priors of the base model.

The authors conclude that their approach enables models to achieve the "aha moment" of emergent reasoning while maintaining the reliability required for high-stakes applications, without requiring the unbounded penalties that stifle learning in traditional RKL-constrained RFT.