Technical Summary: BALTO – Balanced Token-Level Policy Optimization for Hallucination Mitigation

1. Problem Statement

Large Language Models (LLMs) face significant challenges in knowledge-intensive settings due to faithfulness hallucinations—generating content that lacks factual support or contradicts provided evidence. While Reinforcement Learning (RL) offers a promising avenue for mitigation, current approaches suffer from a granularity mismatch.

Most existing RL methods employ response-level rewards, assigning a single scalar score to an entire output. This coarse-grained supervision fails to distinguish between hallucinated spans and faithful content within the same response. Consequently, a localized error (e.g., an incorrect date or entity) can trigger a uniform negative penalty across all tokens, erroneously suppressing correct information and inducing "reward hacking" behaviors where models generate shorter, overly conservative responses to avoid penalties.

Recent fine-grained approaches (e.g., claim-level or token-level rewards) address detection but often fail to solve the credit assignment problem. Unbalanced token-level rewards can introduce biases toward verbosity or length, while methods tied to global response-level advantages still inherit noise from the global signal. The core problem is the lack of a mechanism that precisely penalizes hallucinated tokens while compensating for faithful tokens within the same response, ensuring the model redistributes probability mass toward truthfulness rather than simply suppressing generation.

2. Methodology: BALTO Framework

The authors propose BALTO (Balanced Token-Level Policy Optimization), a reinforcement learning framework designed to localize hallucinations and assign balanced credit at the token level. The framework operates in two primary stages:

A. Fine-Grained Hallucination Token Detection

BALTO utilizes an LLM-based faithfulness evaluator to process sampled responses:

1. Claim Extraction: The system extracts checkable factual claims (e.g., entities, numbers, dates, relations) from the response.
2. Verification: Each claim is verified against the reference context (retrieved documents). Claims are partitioned into Faithful ($C^+$) and Hallucinated ($C^-$) sets.
3. Token Projection:
   • Tokens in faithful claims are labeled as Faithful ($F$).
   • For hallucinated claims, the system attempts to localize the minimal erroneous token subset ($e_{i,m}$) responsible for the error (e.g., only the incorrect number "11.4%" rather than the whole sentence). If isolation is impossible, the entire claim span is penalized. These tokens are labeled as Hallucinated ($H$).
   • Tokens outside checkable claims or faithful tokens within a hallucinated claim (where the error cannot be isolated) are labeled as Neutral ($N$).

B. Balanced Token-Level Credit Assignment

Unlike standard GRPO which broadcasts a single advantage to all tokens, BALTO assigns specific advantages ($A^{bal}_{i,t}$) based on token labels:

• Hallucinated tokens ($H$): Receive a negative advantage of -1.
• Faithful tokens ($F$): Receive a positive compensatory advantage of $N^-_i / N^+_i$, where $N^-$ and $N^+$ are the counts of hallucinated and faithful tokens, respectively.
• Neutral tokens ($N$): Receive 0 advantage.

Key Mechanism: This design ensures an intra-response zero-sum property: $\sum A^{bal}_{i,t} = 0$. The total negative signal from hallucinations is exactly balanced by the positive signal from faithful content. This encourages the policy to redistribute probability mass from hallucinated content to faithful content, rather than suppressing the entire response.

C. Optimization Objective

BALTO optimizes the policy using a clipped objective similar to GRPO but restricted to factual tokens ($H \cup F$). The objective function maximizes the expected balanced advantage, normalizing updates by the number of factual tokens ($Z_i = \max(1, |H| + |F|)$).

3. Theoretical Analysis

The paper provides theoretical proofs demonstrating BALTO's superiority over response-level GRPO:

• Variance Reduction (Theorem 1): Response-level GRPO induces gradient variance scaling linearly with the total response length ($O(T)$). In contrast, BALTO restricts variance to scale only with the number of hallucinated tokens ($O(N^-)$). Since hallucinations are sparse, this significantly reduces gradient noise and stabilizes training.
• Optimization Efficiency (Theorem 2):
  • Cold Start ($p \to 0$): Response-level GRPO suffers from "gradient starvation" as the normalization coefficient vanishes when faithful responses are rare. BALTO maintains strong, bounded local signals ($O(1)$), ensuring efficient early-stage exploration.
  • Convergence ($p \to 1$): Response-level GRPO triggers unbounded divergent penalties ($A^- \to -\infty$) for rare errors, destabilizing late-stage convergence. BALTO maintains strictly bounded advantages ($[-1, 1]$), ensuring stable and graceful convergence.

4. Experimental Results

Experiments were conducted on three benchmarks: ConFiQA (counterfactual QA), RAGTruth (RAG hallucination corpus), and FinLLM-Eval (financial domain QA). Models used included Qwen3-8B and Qwen3-4B.

Key Findings:

• Faithfulness: BALTO achieved the highest faithfulness scores across all six model-benchmark settings, outperforming response-level baselines (GRPOB, GRPOD) and token-level baselines (FSPO).
• Q-Score: BALTO consistently achieved the highest Q-Score (defined as Faithfulness $\times$ Informativeness), demonstrating a superior trade-off between correctness and information retention.
  • Compared to the strongest post-training baseline, BALTO improved Q-Score by an average of 2.7 points (up to 4.9) and Faithfulness by 3.1 points (up to 6.7).
• Training Dynamics:
  • Gradient Norms: BALTO exhibited gradient norms an order of magnitude smaller and more concentrated than response-level methods, confirming reduced variance.
  • Parameter Updates: BALTO achieved high faithfulness with significantly fewer parameter updates (~12% vs. ~17% for GRPOB), indicating more efficient learning focused on relevant parameters.
• Ablation Studies: Removing the positive compensatory advantage for faithful tokens (setting $PA=0$) resulted in substantial drops in both faithfulness and informativeness, validating the necessity of the balanced credit mechanism.

5. Significance and Claims

The paper claims that BALTO addresses a critical gap in hallucination mitigation: the need for balanced, fine-grained credit assignment.

• Precision: By localizing errors to specific tokens and balancing the reward signal, BALTO avoids the "all-or-nothing" penalty of response-level RL, preserving the informativeness of the response.
• Stability: The theoretical and empirical evidence suggests that balanced token-level assignment leads to more stable training trajectories and higher optimization efficiency, particularly in the early stages of training and during convergence.
• Trade-off: The results demonstrate that it is possible to significantly reduce hallucinations without collapsing into overly conservative or verbose generation, achieving a stronger faithfulness-informativeness trade-off than prior post-training approaches.

The authors note limitations, primarily that the method relies on an external LLM-based evaluator for claim extraction and verification, meaning its performance is bounded by the evaluator's accuracy. Additionally, the framework is designed for retrieval-grounded settings where reference documents are available.