Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

Technical Summary: Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

Problem Statement
While large language models (LLMs) demonstrate impressive fluency, they frequently struggle with factual consistency, often generating summaries containing hallucinated details or inconsistencies with the source text. Reinforcement Learning (RL) has been proposed to mitigate this by using evaluation metrics as reward signals. However, existing automatic factuality metrics are individually unreliable and often struggle with subtle errors. Furthermore, using a single metric as a scalar reward can lead to unstable training, while combining metrics typically requires manual weighting (reward shaping), which is complex and prone to error. Alternative approaches like Reinforcement Learning from Human Feedback (RLHF) often fail to prioritize factuality, as human annotators may overlook factual errors in favor of other qualities like fluency or coherence. Constructing high-quality, factuality-focused preference datasets via human annotation is resource-intensive and difficult to scale.

Methodology
The authors propose a fully automated training pipeline that improves factual consistency without relying on human annotations or reference summaries. The core methodology consists of three main stages:

1. Lexically Similar Summary Generation:
   Instead of pairing diverse samples, the method generates pairs of summaries that are lexically similar to minimize confounding stylistic or structural differences. This allows the model to focus specifically on factual distinctions.

   • For a given source document, the model generates a primary summary using Beam Search (BS#1).
   • A second summary is generated either by selecting the second most probable sequence from the same beam search (BS#2) or by using Greedy Decoding.
   • This approach ensures that the differences between the pair are subtle, primarily driven by decoding variations rather than major structural shifts.

2. Automated Preference Labeling via Metric Ensembles:
   To avoid the unreliability of single metrics, the pipeline employs an ensemble of factuality metrics (specifically SBERTScore and SummaC-Conv in the experiments) to score the generated pairs.

   • Scores from different metrics are converted into binary preference labels.
   • A conflict filter is applied: a summary pair is retained for training only if all selected metrics agree on the preference label. Pairs with high disagreement between metrics are filtered out to remove noisy annotations.
   • This process constructs a high-quality preference dataset using only source documents.

3. Preference Optimization (DPO):
   The filtered preference dataset is used to train the language models via Direct Preference Optimization (DPO). Unlike standard RL, DPO optimizes the model parameters directly to increase the likelihood of the preferred summary without requiring a separate, complex reward model. This reduces training complexity and mitigates issues like catastrophic forgetting.

Key Contributions

• Automated Pipeline: A novel, fully automated preference learning pipeline that optimizes summarization factuality without human annotations or reference summaries.
• Metric Aggregation Strategy: A method to adapt multiple existing, imperfect factuality metrics into training targets by converting scores to preferences and filtering out cases with high inter-metric disagreement, thereby enhancing data reliability.
• Lexical Similarity Focus: An analysis demonstrating that training on lexically similar summary pairs is more effective for enhancing factuality than training on diverse pairs, as it isolates factual errors from stylistic variations.
• Performance on Smaller Models: The method successfully elevates the factuality performance of smaller models (e.g., BART) to levels comparable to significantly larger, modern LLMs.

Experimental Results
The approach was evaluated on the XSUM and TL;DR datasets using four models of varying architectures and scales: BART-large, GPT-J, LLaMA-3, and DeepSeek-R1.

• Factuality Gains: The proposed method consistently outperformed baselines, including Supervised Fine-Tuning (SFT), RLHF, and Model-based Preference Optimization (MPO) which relies on heuristic assumptions about decoding strategies.
  • On XSUM, BART achieved an AlignScore increase of +24.7 points over SFT.
  • On TL;DR, BART saw an increase of +9.3 points.
  • Smaller models (BART) achieved factuality scores comparable to or exceeding larger models (LLaMA, DeepSeek) after training.
• Robustness: The method demonstrated stability across different model families and scales, avoiding the catastrophic forgetting observed in other RL-based approaches.
• Trade-offs: While factuality improved significantly, there was a slight trade-off in ROUGE-L scores and overall quality preferences when judged by ChatGPT. The authors note that the preferred summaries (SFT/RLHF) often contained excessive details, whereas the factuality-optimized summaries were more abstract to reduce the risk of inconsistency.

Significance and Claims
The paper claims that this approach provides a stable and scalable alternative to human-annotated RLHF for improving factuality. By leveraging the agreement of multiple weak metrics and focusing on subtle lexical differences, the method effectively trains models to avoid generating plausible but factually incorrect text. The authors highlight that this pipeline "revitalises" the potential of older, smaller models to produce accurate summaries at lower computational costs, making high-factuality summarization more accessible. The work suggests that preference learning, when grounded in metric agreement rather than human bias or single-metric rewards, offers a robust path toward more faithful language model outputs.