💻 computer science

Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding

The paper introduces ConSUM, a reranking framework that enhances the factuality of generated summaries by leveraging Minimum Bayes Risk decoding to balance consensus among candidate outputs with consistency to the source document, achieving superior performance over existing methods.

Original authors: Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura, Taro Watanabe

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Riza Setiawan Soetedjo, Yusuke Sakai, Hidetaka Kamigaito, Jingun Kwon, Manabu Okumura, Taro Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a news editor trying to write a short summary of a long, complicated article. You ask a team of AI assistants to write their own versions for you. The problem is, even though these AIs are smart, they sometimes make up facts or get details wrong (like saying a car crash happened on Tuesday when it was actually Wednesday). This is called a "factual error."

The paper introduces a new method called ConSUM to fix this. Think of ConSUM not as a single writer, but as a smart editor-in-chief who uses a two-step voting system to pick the best summary from the team.

Here is how it works, using simple analogies:

1. The Problem: The "Single Source" Trap

Usually, when an editor checks a summary, they only look at the original article (the source). They ask, "Does this match the article?"

The Flaw: Sometimes, an AI writes a summary that sounds like it matches the article but actually swaps a name or a number. Because the AI is so confident, the editor might miss the mistake. It's like a student copying a textbook but changing one crucial number; if you only skim the text, you might not catch it.

2. The Solution: The "ConSUM" Editor

ConSUM improves the process by looking at two things at once: Consistency and Consensus.

Step A: Consistency (Checking the Source)

This is the traditional check. The editor compares the summary against the original article to make sure the facts are there.

Analogy: It's like a teacher checking a student's homework against the answer key to ensure they didn't invent new rules.

Step B: Consensus (The "Wisdom of the Crowd")

This is the new, clever part. The AI generates many different versions of the summary (let's say 16 different drafts).

The Idea: If 15 out of 16 AI assistants agree that "The rocket carried 6 tourists," but one assistant says "The rocket carried 600 tourists," the one saying 600 is likely hallucinating (making things up).
The Mechanism: ConSUM uses a technique called Minimum Bayes Risk (MBR). Imagine a panel of judges. Instead of just picking the "most likely" sentence, they look at all the drafts and ask, "Which sentence appears most often across all the different versions?"
Analogy: It's like a game of "Telephone." If everyone in the circle whispers the same thing, it's probably the truth. If one person whispers something totally different, they are likely the one who got confused. ConSUM picks the version that aligns with the "group vote."

3. The Final Decision: The Scorecard

ConSUM combines these two checks into a final score:

Consistency Score: How well does it match the original article?
Consensus Score: How well does it agree with the other AI drafts?

The system picks the summary with the highest combined score. It's like a hiring manager who checks a candidate's resume (Consistency) but also asks for references from their past coworkers (Consensus) to see if they are actually reliable.

What Did They Find?

The researchers tested this on news articles (like CNN and XSum datasets).

The Result: The summaries picked by ConSUM were much more accurate than summaries picked by older methods.
Human Proof: When real humans read the summaries, they preferred the ones made by ConSUM. They found them to be more trustworthy and factual, without losing the quality or flow of the writing.

The Catch (Limitations)

The paper notes that this "voting" process takes a bit more computer power and time because the AI has to generate many drafts and compare them all. It's like asking a whole committee to vote instead of just one person deciding; it's more accurate, but it takes longer.

In short: ConSUM stops AI from making up facts by making it "vote" on the truth. If the AI team agrees on a fact, it's probably true. If they disagree, the system filters out the weird, made-up answers.

Technical Summary: Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding

Problem Statement
Improving the factuality of model-generated summaries—defined as the accuracy of a summary with respect to its source content—remains a significant challenge in Natural Language Generation (NLG). While reranking methods exist to select optimal outputs from multiple candidates, current approaches suffer from three primary limitations:

Inability to use reference-based metrics: High-quality "gold" references are labor-intensive to create, forcing reliance on reference-free metrics.
Source-only reliance: Reranking based solely on the source document often fails to penalize specific factual errors due to the broad scoring nature of reference-free metrics, leading to factual inconsistencies.
Metric bias: Relying on a single metric can cause overfitting and inherit specific biases of that metric.

Existing methods often struggle to balance fluency and coherence with strict factual alignment, particularly when gold references are unavailable or of low quality.

Methodology: ConSUM
The authors propose ConSUM (Consistency and Consensus in Summarization), a novel reranking framework that combines two distinct signals to select the best summary candidate. The method operates in three steps:

Generation: Given a source document $s$ , the model generates two distinct sets:
- Candidate Summaries ( $Y$ ): A pool of potential outputs.
- Pseudo-References ( $R$ ): A set of "proxy" ground truths sampled from the same model but potentially using different decoding strategies (e.g., epsilon sampling) to better capture the model's true distribution. The authors distinguish $Y$ and $R$ to prevent the model from reinforcing its own biases.
Scoring: Each candidate $y_i$ is scored using two factors:
- Consistency Score ( $S_{sis}$ ): Measures the alignment between the candidate and the source document using reference-free factuality metrics (specifically FENICE and FIZZ).
- Consensus Score ( $S_{sen}$ ): Measures the agreement between the candidate and the set of pseudo-references using Minimum Bayes Risk (MBR) decoding. The utility function for MBR is a reference-based metric, MENLI (Metrics from Natural Language Inference), which evaluates entailment, contradiction, and summarization quality. This step identifies the candidate that maximizes the expected utility against the pseudo-references.
Reranking: The final score ( $S_{fin}$ ) is a weighted linear combination of the normalized consistency and consensus scores:
$S_{fin}(y_i, s, R) = w \cdot Z(S_{sen}(y_i, R)) + (1-w) \cdot Z(S_{sis}(y_i, s))$
where $w$ is a hyperparameter (set to 0.75 in the main experiments) balancing the importance of consensus versus source consistency. The candidate with the highest $S_{fin}$ is selected as the final output.

Key Contributions

Dual-Signal Reranking: The paper introduces a framework that simultaneously leverages source consistency (via reference-free metrics) and model consensus (via MBR with pseudo-references) to improve factuality.
Decoupling Candidates and References: Unlike prior work that often treats candidates and references as the same set, ConSUM explicitly separates them. Pseudo-references serve as an unbiased proxy for ground truth to estimate the model's distribution, while candidates serve as the pool for selection.
Integration of MBR and Reference-Free Metrics: The method successfully combines MBR decoding (typically used in Machine Translation) with state-of-the-art reference-free factuality metrics (FENICE, FIZZ) for summarization, addressing the lack of gold references.

Experimental Results
The method was evaluated on the CNN/DailyMail and XSum datasets using three Pretrained Language Models (BART, PEGASUS, T5) and one Large Language Model (Llama-3).

Factuality Improvements: ConSUM consistently achieved superior factuality scores compared to baselines (standard decoding and consistency-only rerankers). For instance, on CNN/DM, the FENICE-0.75 configuration improved the FIZZ score from 39.36 (Baseline) to 52.44. On XSum, the same configuration raised the FIZZ score from 16.91 to 27.79.
Quality Preservation: The improvements in factuality did not come at the cost of summary quality. ROUGE, BERTScore, and MoverScore scores remained competitive with or significantly better than baselines.
Human Evaluation: In a human evaluation involving 50 source documents, the FENICE-0.75 system achieved the highest overall average scores for Informativeness, Factuality, and Fluency. Annotators consistently preferred ConSUM-generated summaries over baselines and even the "gold" references (which were noted to be of low quality in recent literature).
Hyperparameter Sensitivity: The optimal weight $w$ varied by dataset (0.75 for CNN/DM, 0.25 for XSum with FENICE), but a unified setting of $w=0.75$ was chosen for its balanced performance. The study found that 16 candidates and 64 pseudo-references (generated via epsilon sampling) yielded optimal results.

Significance and Claims
The authors claim that ConSUM demonstrates that combining source-based factuality with model-based consensus effectively improves the reliability of generated summaries without requiring gold references. The results suggest that the "wisdom of the crowd" (consensus among model outputs) can serve as a robust signal to filter out hallucinations and factual errors, particularly in highly abstractive tasks like XSum where hallucinations are frequent.

The paper concludes that while there is still room for improvement (as evidenced by the gap between ConSUM and Oracle scores), the proposed method offers a practical, unsupervised approach to enhancing summary factuality that is preferred by human evaluators. The authors acknowledge limitations, including the computational complexity of MBR ( $O(n^2)$ ) and the current restriction to English news datasets, suggesting future work could explore diverse text types and languages.