Technical Summary: Rewarding Intellectual Humility in Large Language Models

Problem Statement

Large Language Models (LLMs) frequently generate hallucinated or unverifiable content, particularly in factual domains, because their training objectives typically reward response generation regardless of certainty. Existing mitigation strategies, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often rely on costly human-in-the-loop feedback and do not explicitly model uncertainty or teach models to abstain when unsure. This work addresses the need for a training paradigm that explicitly incentivizes "intellectual humility"—the ability to say "I don't know"—to reduce hallucinations without relying solely on dense human preference data.

Methodology

The authors propose a modified Reinforcement Learning with Verifiable Rewards (RLVR) framework. This approach utilizes a ternary reward structure to guide model behavior:

• Correct Answer ($r_{correct} = 1$): Rewarded.
• Incorrect Answer ($r_{wrong} = -1$): Penalized.
• Abstention ($r_{abs}$): Rewarded or penalized based on a tunable hyperparameter. The abstention token is defined as "I don't know."

Experimental Setup

• Models: The study evaluates Granite-3.3-2B-Instruct and Qwen-3-4B-Instruct.
• Datasets:
  • MedMCQA: A multiple-choice medical dataset augmented with an explicit "I Don't Know" option to facilitate verification.
  • Hendrycks Math: A dataset of open-ended, multi-step mathematical problems used to test generalization to non-multiple-choice scenarios.
• Training Strategies: The authors compare three primary settings using Group Relative Policy Optimization (GRPO):
  1. RL-only: Direct training with the ternary reward signal.
  2. RL-SFT-Random: A two-stage process where the model is first SFT-trained on MedMCQA with 30% of ground truths randomly replaced with "I don't know" to encourage abstention, followed by RLVR fine-tuning.
  3. RL-RTuning: Similar to the above, but the SFT step specifically targets questions the base model answered incorrectly, teaching the model to abstain only on known failures (based on R-Tuning principles).

Reward Design

The reward function $R(y)$ is defined as:
$$R(y) = \begin{cases} 1 & \text{if } y \text{ is correct} \\ r_{abs} & \text{if } y = \text{"I don't know"} \\ -1 & \text{if } y \text{ is incorrect} \end{cases}$$
For MedMCQA, a format reward is also applied to enforce strict XML-style tagging (<reasoning>, <answer>) to ensure verifiable output structures.

Key Results

1. Trade-off Between Accuracy and Abstention

The experiments reveal a clear, tunable trade-off controlled by the abstention reward ($r_{abs}$):

• Moderate Rewards: Setting $r_{abs}$ to small positive or slightly negative values (e.g., $-0.25$ to $0.3$) consistently reduces incorrect responses (hallucinations) without causing severe degradation in overall accuracy on multiple-choice tasks.
• Model Sensitivity: Larger models (Qwen-3-4B) exhibit greater robustness to abstention incentives compared to smaller models (Granite-3.3-2B). For instance, on MedMCQA, Qwen-3-4B with $r_{abs}=0.3$ reduced incorrect answers from 32.5% to 10.3% while maintaining 48.0% correctness, though it increased the "I don't know" rate to 41.0%.

2. Performance of Training Variants

• RL-only: Effective in reducing hallucinations on MedMCQA but struggled with Hendrycks Math. The authors attribute this to "insufficient exploration," where the base model's initial hesitancy to use the "I don't know" token prevents the RL algorithm from learning the behavior in open-ended settings.
• RL-SFT-Random: This approach successfully mitigated the exploration issue. On Hendrycks Math, it reduced incorrect answers by approximately 28% with only a 10% loss in accuracy. However, on MedMCQA, higher abstention rewards in this setting led to a monotonic decline in correctness as the model became overly cautious.
• RL-RTuning: Contrary to expectations, this method underperformed RL-SFT-Random. The authors hypothesize that the SFT step in RL-RTuning was dominated by "I don't know" responses (approx. 60%), causing the model to collapse in its ability to answer confidently, a state it could not recover from during the subsequent RL phase.

3. Open-Ended vs. Multiple-Choice

While the ternary reward scheme worked well for structured multiple-choice questions, open-ended QA (Hendrycks Math) required the initial SFT step to overcome the model's reluctance to abstain. Without this pre-training, RL-only failed to induce abstention behavior in the open-ended domain.

Significance and Claims

The paper claims that verifiable reward design offers a practical, flexible, and cost-effective alternative to human-heavy alignment methods for mitigating hallucinations. Key takeaways include:

• Feasibility: Explicitly rewarding abstention is a viable strategy to promote intellectual humility in LLMs.
• Tunability: The abstention reward ($r_{abs}$) acts as a direct control knob, allowing practitioners to balance the trade-off between caution (reducing hallucinations) and correctness (maintaining task accuracy) based on specific application needs.
• Scalability: The approach is particularly effective for larger models, which demonstrate better internal calibration under moderate abstention incentives.
• Limitations: The study acknowledges that while the method reduces hallucinations, it does not eliminate them entirely, and the optimal reward structure varies significantly across models and dataset types (MCQ vs. open-ended).

The authors conclude that while the current results are promising, future work is needed to optimize the ratio of abstention samples in SFT steps (suggesting 10–30% rather than 30% or higher) to prevent capability collapse and further improve performance in open-ended reasoning tasks.