KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning

Technical Summary: KARL (Knowledge-Boundary-Aware Reinforcement Learning)

KARL is a novel reinforcement learning (RL) framework designed to mitigate hallucinations in Large Language Models (LLMs) by enabling them to appropriately abstain from answering questions that fall outside their knowledge boundaries.

1. The Problem: The "Abstention Trap" and Accuracy-Hallucination Trade-off

Existing methods to reduce hallucinations typically face a zero-sum trade-off:

• Non-abstention methods (e.g., standard SFT or Binary RL) maximize accuracy but fail to suppress hallucinations, as models are incentivized to "guess" to avoid being penalized for silence.
• Abstention-aware methods (e.g., Ternary Rewards) attempt to reward "I don't know" responses. However, they often suffer from "Abstention Collapse" (the "Abstention Trap"). Because models are initially inaccurate, they encounter many groups of responses containing only incorrect answers and abstentions. In static ternary reward structures, any reward for abstention higher than the penalty for an incorrect answer creates a structural bias that drives the model to abstain on everything, leading to a massive drop in accuracy and a loss of learning signals.

2. Methodology: The KARL Framework

To solve this, the authors propose a framework that aligns abstention behavior with the model's evolving knowledge boundary through two core innovations:

A. Knowledge-Boundary-Aware Reward (KAR)
Instead of using a static reward, KAR performs online knowledge boundary estimation using within-group response statistics during the RL process. For each query, the model generates a group of $G$ responses. The reward is determined by a "solvability indicator":

• If the query is "Solvable" (at least one correct answer exists in the sampled group): The reward encourages correct answers ($r_{pos} = +1$) and penalizes both incorrect answers and unnecessary abstentions ($r_{neg}, r_{abs} = -1$).
• If the query is "Unsolvable" (no correct answers exist in the group): The reward encourages honest abstention ($r_{abs} = +1$) and penalizes incorrect answers ($r_{neg} = -1$).

B. Two-Stage RL Training Strategy
To prevent the early-stage "abstention trap," the training is decoupled into two distinct phases:

1. Stage I: Knowledge Boundary Exploration: The model uses a mixed-reward strategy. A portion of the data is trained with a Binary Reward (rewarding only correctness). This acts as a safeguard, providing strong gradients for accuracy and preventing the model from collapsing into total silence. This stage builds a robust factual foundation.
2. Stage II: Abstention Calibration: Once the model has a high accuracy baseline, the full KAR mechanism is applied to all data. The goal shifts to converting the remaining "persistent errors" (queries the model still gets wrong) into appropriate abstentions without erasing the accuracy gained in Stage I.

3. Key Contributions

• Dynamic Boundary Estimation: Moving from offline, static labeling to an online, sampling-based approach that evolves as the model learns.
• Mitigation of Policy Collapse: The two-stage curriculum effectively bypasses the "abstention trap" by prioritizing knowledge recall before calibration.
• Unified Reliability Metric: The paper utilizes a "Rely" score that mathematically balances the trade-off between accuracy ($T$), abstention ($U$), and hallucination ($F$).

4. Results and Significance

Experimental Results:

• Superior Trade-off: On the NaturalQuestions (NQ) benchmark, KARL achieved a significant reduction in hallucinations (-38.6% for Llama-3.1) while actually improving accuracy (+4.0%).
• Robust Generalization: Unlike SFT or R-Tuning, which often fail on out-of-distribution (OOD) data, KARL demonstrated robust performance on TriviaQA, BioASQ, and ARC-Challenge, maintaining high reliability across different domains and formats.
• Preservation of Reasoning: Evaluations on GSM8K and BBH showed that training on knowledge-intensive tasks did not degrade the model's core mathematical and logical reasoning capabilities.

Significance:
KARL represents a significant step toward building reliable and trustworthy AI. By treating the knowledge boundary as a moving target rather than a static property, it provides a scalable way to train models that "know what they don't know," making them safer for high-stakes applications in fields like medicine and law where factual errors are critical.