Reducing Hallucinations in LLMs via Factuality-Aware Preference Learning

1. Problem Statement

Large Language Models (LLMs) often exhibit hallucinations—generating fluent and confident responses that are factually incorrect. While alignment methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) improve instruction following, they often inadvertently reinforce hallucinations.

• The Core Conflict: Preference labels (human or automated) often prioritize fluency, confidence, and elaboration over factual correctness. Annotators may prefer a confident but incorrect answer (e.g., "Sydney is the capital of Australia") over a less elaborate but accurate one ("Canberra").
• Limitations of Existing Methods: Current factuality-focused approaches often rely on:
  • Auxiliary reward models or verifiers (increasing complexity).
  • Multi-stage training pipelines.
  • Token-level supervision (fine-grained annotations).
  • They often fail to correct the root cause: misordered preference pairs where the "chosen" response is hallucinated and the "rejected" one is factual.

2. Methodology: F-DPO (Factuality-aware Direct Preference Optimization)

The authors propose F-DPO, a single-stage extension of standard DPO that requires only binary factuality labels (Factual vs. Hallucinated) and no external reward models.

A. Data Construction

The authors construct a factuality-aware preference dataset by augmenting the Skywork Reward-Preference corpus:

1. Binary Labeling: Each response in a preference pair is annotated with a binary label $h \in \{0, 1\}$, where $0$ is factual and $1$ is hallucinated, using an automated LLM judge (GPT-4o-mini).
2. Synthetic Augmentation: To ensure balanced coverage of factuality configurations, synthetic hallucinated variants are generated by prompting an LLM to introduce plausible but incorrect claims into factual answers.
3. Balancing: The dataset is rebalanced across configurations of $(h_{winner}, h_{loser})$.

B. Core Mechanisms

F-DPO introduces two key modifications to the standard DPO objective:

1. Label Flipping (Deterministic Correction):

   • Standard DPO assumes the "winner" ($y_w$) is always better. However, if $y_w$ is hallucinated ($h_w=1$) and $y_l$ is factual ($h_l=0$), training on this pair reinforces hallucinations.
   • F-DPO calculates the factuality differential $\Delta h = h_l - h_w$.
   • If $\Delta h < 0$ (i.e., the chosen response is less factual), the labels are flipped: the factual response becomes the winner, and the hallucinated one becomes the loser. This ensures the model is never trained to prefer a hallucinated response over a factual one.

2. Factuality-Aware Margin Penalty:

   • Standard DPO maximizes the margin between the log-probabilities of the winner and loser.
   • F-DPO modifies the margin $m$ by subtracting a penalty term proportional to the factuality difference:
     $$m_{fact} = m_{\pi, \pi_{ref}} - \lambda \cdot \Delta h$$
   • Effect:
     • If $\Delta h = 1$ (Winner is factual, Loser is hallucinated), the effective margin is reduced by $\lambda$. This forces the model to assign a significantly higher probability to the factual response to minimize the loss, thereby amplifying the learning signal for factuality.
     • If $\Delta h = 0$ (Both factual or both hallucinated), the objective reduces to standard DPO, preserving original preference signals for other qualities (e.g., style, helpfulness).

C. Training Objective

The final loss function is:
$$L_{F-DPO} = -\mathbb{E} \left[ \log \sigma \left( \beta \cdot (m_{\pi, \pi_{ref}} - \lambda \cdot \Delta h) \right) \right]$$
Where $\lambda$ is a tunable hyperparameter controlling the strength of the factuality penalty.

3. Key Contributions

1. Novel Algorithm: Introduction of F-DPO, a simple, single-stage, response-level method that corrects preference-factuality misalignment using only binary labels.
2. Mechanism Innovation: The combination of label flipping (to correct noisy supervision) and a factuality-conditioned margin (to emphasize correctness gaps) without auxiliary models.
3. Comprehensive Evaluation: Validation across seven open-weight LLMs (ranging from 1B to 14B parameters) from diverse families (Qwen, LLaMA, Gemma).
4. Generalization: Demonstration of strong out-of-distribution (OOD) performance on the TruthfulQA benchmark.

4. Experimental Results

The paper reports significant improvements in factuality and reductions in hallucination rates compared to both Base Models and Standard DPO.

• Hallucination Reduction:
  • On Qwen3-8B, F-DPO reduced the hallucination rate by 5× (from 0.424 to 0.084).
  • On Qwen2.5-14B, it achieved a near-zero hallucination rate of 0.008 (an order of magnitude improvement over the base model).
• Factuality Scores:
  • Factuality scores improved by 50% on Qwen3-8B (from 5.26 to 7.90).
• Out-of-Distribution Generalization (TruthfulQA):
  • On Qwen2.5-14B, F-DPO improved MC1 accuracy by +17% (0.500 $\to$ 0.585) and MC2 accuracy by +49% (0.357 $\to$ 0.531).
  • In contrast, Standard DPO often degraded performance on TruthfulQA, confirming that standard preference optimization can harm truthfulness.
• Ablation Studies:
  • Label Flipping: Essential for correcting misordered pairs; combining it with the margin penalty yields the best results.
  • Penalty Strength ($\lambda$): Performance improves monotonically with $\lambda$ up to 100, after which diminishing returns occur.
  • Data Efficiency: The method achieves comparable performance using only 25% of the training data.
  • Helpfulness: F-DPO generally preserves or improves helpfulness scores, though a slight trade-off was observed in Qwen3-8B where extreme factuality optimization slightly reduced stylistic richness.

5. Significance and Impact

• Simplicity and Efficiency: F-DPO eliminates the need for complex multi-stage pipelines, token-level annotations, or auxiliary reward models, making it highly compute-efficient and easy to implement.
• Addressing the "Fluency Trap": It directly tackles the issue where LLMs are rewarded for sounding confident rather than being correct, a critical failure mode in high-stakes domains like medicine and law.
• Scalability: The method scales effectively across model sizes (1B–14B) and generalizes to unseen benchmarks, suggesting it is a robust solution for aligning LLMs with truthfulness.
• Ethical Alignment: Qualitative analysis shows that F-DPO improves the model's ability to refuse harmful or unethical requests (e.g., bypassing encryption) while maintaining factual grounding, indicating a synergy between factuality and safety alignment.

In summary, F-DPO provides a pragmatic and effective framework for reducing hallucinations by explicitly correcting the preference signal to prioritize factual correctness over mere fluency.