💬 NLP

From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

The paper introduces CARE-PPO, a reinforcement learning framework that leverages a Confidence-Aligned Reward to jointly optimize language-based quantitative predictions and their reliability by repurposing the PPO critic as a robust confidence estimator, thereby outperforming existing baselines in accuracy and uncertainty calibration across healthcare and finance domains.

Original authors: Mehak Dhaliwal, Rasta Tadayon, Andong Hua, Haewon Jeong, Yao Qin

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mehak Dhaliwal, Rasta Tadayon, Andong Hua, Haewon Jeong, Yao Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can read a messy description of your lunch and guess exactly how many grams of carbohydrates are in it. This is great for people with diabetes who need to know their insulin dosage. But here's the catch: sometimes this robot is wildly confident about a completely wrong answer. It might say, "I'm 100% sure this pizza has 5 grams of carbs!" when it actually has 40. That's dangerous.

The paper introduces a new training method called CARE-PPO to fix this. Think of the robot as having two brains working together: an Actor (the guesser) and a Critic (the judge).

The Old Way vs. The New Way

Usually, when we train these robots, we just tell them, "You were right" or "You were wrong." It's like a teacher giving a student a red 'X' or a green checkmark. The student learns to get the number right, but they don't learn when to trust themselves. They might guess wildly and still get a green checkmark if they get lucky, or feel unsure even when they are right.

The authors argue that this "binary" (right/wrong) approach isn't enough. They explicitly rule out just using the robot's own internal "confidence" scores (like how sure it feels based on its own thoughts) or asking it to say "I'm 80% sure" in words, because those methods often lead to the robot being overconfident and wrong.

The Magic of CARE-PPO

CARE-PPO changes the game by giving the robot a Critic that acts like a strict but fair coach. Here's how it works:

The Reward System: Instead of just a "Right/Wrong" checkmark, the Critic gives a score based on how far off the guess was. If the robot guesses 44g and the real answer is 44g, it gets a huge reward. If it guesses 17g, it gets a small reward (or a penalty). The closer the guess, the higher the score.
The Critic's Secret Job: While the Actor is trying to get the number right, the Critic is learning to predict how good the Actor's guess will be. The paper suggests that by training the Critic to predict these scores, it naturally learns to become a confidence meter.
- If the Critic sees a situation where the Actor usually makes big mistakes, it gives a low "confidence score."
- If the situation is easy and the Actor usually gets it right, the Critic gives a high "confidence score."

It's like a video game where the Critic isn't just watching the player; it's learning to predict the player's future performance. Over time, the Critic becomes so good at this that its prediction is the confidence score. You don't need to ask the robot, "How sure are you?" because the Critic already knows.

What the Numbers Say

The authors tested this on two real-world tasks:

Nutrition: Guessing carbs from meal descriptions (like "a slice of pepperoni pizza").
Finance: Guessing the price of products (like a blender) from their descriptions.

They used two versions of a model called Qwen-3 (one with 4 billion parameters and one with 8 billion).

The results were promising:

Accuracy: The CARE-PPO models got the numbers right almost as well as the best other methods.
Confidence: This is where it shined. The Critic's confidence scores were much better at matching reality than other methods. In the tests, when the model was wrong, the Critic gave a low score, and when it was right, it gave a high score.
Robustness: The method worked even when the robot was tested on things it hadn't seen before, like meal descriptions in different languages (Spanish, Italian, etc.) or products from a different category (electronics instead of appliances).

The "Task Overfitting" Problem

One cool finding is that this method helps the robot stay "task-aware." If you ask a robot trained on food to write a poem, a robot trained with the old "Supervised Fine-Tuning" (SFT) method might get confused and try to calculate the carbs in the poem anyway! The CARE-PPO robots, however, were much better at realizing, "Hey, this isn't food, I shouldn't guess carbs." They kept their general "I can do anything" personality while being good at the specific job.

The Bottom Line

The paper suggests that by linking the "confidence" of a prediction directly to the "error" of that prediction during training, we can build AI that not only gives better numbers but also knows when to say, "I'm not sure about this one." It's a step toward making AI safer and more trustworthy in high-stakes situations like healthcare and finance, without needing extra steps or asking the AI to guess its own confidence in words. The authors found this works well in simulations and real-world datasets, though they note they are still exploring how it scales to even larger models.

Technical Summary: CARE-PPO for Language-Based Quantitative Prediction

Problem Statement

Large Language Models (LLMs) have demonstrated capability in performing quantitative prediction tasks from unstructured natural language inputs, such as estimating carbohydrate content from meal descriptions or predicting product prices. However, these models remain susceptible to hallucinations and overconfident errors, a critical limitation in high-stakes domains like healthcare and finance. Existing approaches for confidence estimation in LLMs largely focus on classification or open-ended generation, often relying on post-hoc methods (e.g., verbalized confidence, token probabilities, or entropy) that are not optimized during training. Furthermore, current quantitative prediction methods prioritize accurate point estimates while neglecting the generation of reliable confidence signals, leaving users without guidance on when to trust a prediction.

Methodology: CARE-PPO

The authors introduce CARE-PPO (Confidence-Aligned Reward for Estimation with PPO), a reinforcement learning framework that establishes a novel connection between loss prediction for uncertainty estimation and actor-critic Proximal Policy Optimization (PPO) fine-tuning.

Core Insight

The framework is built on the insight that a model's predictive uncertainty at a given input can be characterized as the expected loss incurred by the predictor. By defining the reward in an actor-critic setup as a function of prediction error, the critic's value function naturally converges to an estimate of the actor's expected error. This allows the critic to serve as a confidence estimator without requiring explicit confidence supervision or a separate calibration stage.

Framework Architecture

Actor-Critic Setup: The system models language generation as a Markov Decision Process (MDP). The actor ( $\pi_\theta$ ) generates a reasoning trajectory and a final numerical estimate. The critic ( $V_\phi$ ) estimates the expected return from a given state.
Confidence-Aligned Reward: The terminal reward $R(s_T)$ $R (s_{T})$ is defined as a monotonic function of the absolute prediction error $\xi = |\hat{y} - y|$ $ξ = ∣ \overset{y}{^} - y ∣$ , where $\hat{y}$ $\overset{y}{^}$ is the predicted value and $y$ $y$ is the ground truth. The authors propose two instantiations:
- Linear Reward: $h(\xi) = -\xi$ , providing a dense, unbounded signal.
- Gaussian Reward: $h(\xi) = \alpha \exp(-\xi^2 / 2\sigma^2)$ , providing a bounded, interpretable signal where $\sigma$ controls error tolerance.
Training Objective: The actor is optimized via PPO to maximize the expected return (minimize error), while the critic is trained to regress to the empirical return.
Inference: At inference time, the critic's value estimate is repurposed directly as a confidence score. High values indicate low expected error (high confidence), and low values indicate high expected error.

Key Contributions

Theoretical Connection: The paper establishes a link between loss prediction theory and actor-critic RL, demonstrating that when rewards are aligned with prediction error, the PPO critic naturally learns to estimate the actor's expected loss.
CARE-PPO Framework: A unified framework that jointly trains an actor for accurate numerical estimation and a critic for error-aligned confidence estimation, eliminating the need for explicit confidence supervision.
Comprehensive Evaluation: The method is evaluated on two real-world tasks (Nutrition Estimation and Product Price Prediction) using Qwen-3 4B and 8B models.
Robustness Analysis: The study demonstrates that CARE-PPO maintains performance and confidence alignment under realistic out-of-distribution (OOD) settings, including linguistic shifts (multilingual inputs) and domain shifts (different product categories).

Experimental Results

The authors evaluated CARE-PPO against baselines including Base Models, Supervised Fine-Tuning (SFT), and a PPO variant with binary rewards (PPO-Binary). Confidence estimation was compared against logit-based methods (Answer Probability, EntropyInv) and verbalized methods (Verb-1S, Verb-2S).

Prediction Accuracy: CARE-PPO variants achieved competitive Mean Absolute Error (MAE), significantly outperforming the Base Model and PPO-Binary. While SFT achieved the lowest MAE in some cases, it suffered from poor confidence alignment.
Confidence Alignment: Measured by the Area Under the Sparsification Error (AUSE), the critic-based confidence estimator in CARE-PPO consistently outperformed all baselines (verbalized and logit-based) across all tasks, model scales, and settings.
- Even with the binary reward (PPO-Binary), the critic achieved lower AUSE than verbalized baselines, suggesting the actor-critic dynamic inherently encourages confidence-aware representations.
- CARE-PPO with error-aligned rewards (Linear/Gaussian) further amplified these gains.
Robustness:
- Linguistic Shift: In multilingual nutrition estimation, CARE-PPO maintained lower MAE and significantly better confidence alignment (lower AUSE) compared to SFT and Base models.
- Domain Shift: In price prediction (transferring from Appliances to Electronics), critic-based confidence remained robust, whereas SFT and Base models showed degraded alignment.
Task Awareness: Unlike SFT models, which frequently overfitted to the task and incorrectly generated carbohydrate estimates for unrelated queries (e.g., writing a limerick), CARE-PPO models preserved general instruction-following behavior. The authors note that CARE-PPO-Linear showed slightly higher overfitting than the Gaussian variant, potentially due to the unbounded nature of the linear reward.

Significance and Claims

The paper claims that CARE-PPO represents a significant step forward in making LLMs reliable for quantitative prediction in safety-critical domains. Its primary significance lies in:

Emergent Confidence: It demonstrates that reliable confidence estimation can emerge as a byproduct of standard actor-critic RL training when rewards are aligned with prediction error, removing the need for complex post-hoc calibration or auxiliary models.
Efficiency: The approach introduces minimal inference overhead (approximately 2.4% increase in latency) because the critic performs only a single forward pass on the completed response.
Generalization: The framework reduces task-specific overfitting on general prompts compared to SFT, preserving the model's general-purpose capabilities while specializing for quantitative tasks.

The authors conclude that confidence-aware language-based quantitative prediction is a promising direction, with CARE-PPO offering a scalable and efficient solution that couples prediction accuracy with trustworthy uncertainty estimation.