When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisation

1. Problem Statement

As automated multi-document news summarisation systems become ubiquitous, they risk distorting public understanding by exhibiting political bias. Existing research has identified biases such as position bias (over-reliance on early text), entity-based bias (different outputs for different political figures), and demographic under-representation. However, a critical gap exists in evaluating how Large Language Models (LLMs) handle multiple news sources with varying political perspectives simultaneously.

Key challenges addressed include:

• Lack of Specialized Data: Existing multi-document datasets lack comprehensive political orientation labels at the article level.
• Evaluation Gaps: Current methods often fail to assess multiple fairness dimensions (e.g., neutrality, representation balance, sentiment preservation) simultaneously.
• The "Scaling" Myth: It is unclear whether increasing model size (parameter count) inherently leads to fairer outputs, or if debiasing techniques (like prompting) are effective across different architectures.

2. Methodology

A. The FairNews Dataset

The authors introduced FairNews, the first multi-document news summarisation dataset with explicit political orientation labels.

• Source: Constructed from All the News 2.0 (2.7M articles).
• Labeling: Articles were assigned political leanings (Left, Center, Right) using AllSides publisher ratings.
• Clustering: Articles were grouped into events based on temporal proximity (±3 days) and semantic similarity (TF-IDF/Cosine).
• Constraints: Clusters must contain representation from all three political perspectives. Non-political content (sports, entertainment) was filtered. The dataset contains 181 events and 742 articles.

B. Evaluation Framework

The study employs a five-dimensional fairness metric framework (visualized in Figure 1 of the paper):

1. Neutralisation: The percentage of sentences in the summary that use a neutral tone (measured via NewsSentiment classifier).
2. Equal Fairness: The "equality gap" between the highest and lowest represented political perspectives in the summary (measured via a political ideology BERT classifier).
3. Ratio Fairness: The deviation of the output political distribution from the input distribution, measured using Wasserstein Distance.
4. Entity Coverage: The ratio of named entities (people, organizations) preserved from source documents to the summary (using spaCy).
5. Entity Sentiment Similarity: The Wasserstein Distance between the sentiment distributions of entities in the source vs. the summary.

C. Experimental Setup

• Models: 13 LLMs across three families: Gemma-3 (1B, 4B, 12B, 27B), Llama-3 (1B, 3B, 8B, 70B), and Qwen2.5 (1.5B, 3B, 7B, 32B, 72B).
• Baselines: Simple summarisation prompts.
• Debiasing Interventions:
  • Prompt-based: Four strategies including "Debias Instruction," "Debias Persona," "Structured Prompt" (step-by-step guidelines), and "Debias Reference" (providing metadata).
  • Agent-based: A "Judge" model (largest variant in the family) performs pairwise comparisons to select the fairest summary from a set of generated outputs.

3. Key Results

A. Model Performance vs. Size

• Quality: Mid-sized models (e.g., Gemma-3 12B, Llama-3 8B, Qwen2.5 7B) often achieved the best balance of lexical similarity (ROUGE-L), semantic similarity (BERTScore), and factual consistency (AlignScore).
• Scaling Paradox: The largest models (e.g., Llama-3 70B, Qwen2.5 72B) did not consistently outperform mid-sized variants. In some cases, larger models showed degradation in similarity metrics.
• Fairness: Mid-sized models consistently outperformed both smaller and larger counterparts in fairness metrics. The largest variants were not necessarily the most equitable.

B. Inherent Biases

• Centrist Under-representation: All model families exhibited a systematic bias against centrist perspectives, regardless of input balance.
• Family-Specific Skews:
  • Gemma-3: Skewed heavily Right.
  • Qwen2.5: Skewed heavily Left (especially smaller variants).
  • Llama-3: Moderate Right skew.

C. Effectiveness of Debiasing

• Prompt-Based Interventions:
  • Highly Model-Dependent: Effectiveness varied significantly by architecture.
  • Structured Prompts: Generally yielded the most consistent improvements across metrics.
  • Entity Sentiment Resistance: The Entity Sentiment Similarity metric proved the most stubborn; it resisted all prompt-based interventions, suggesting that surface-level instructions cannot alter deep semantic sentiment encodings.
  • Coverage Trade-off: Some debiasing prompts improved fairness but degraded Entity Coverage, narrowing the representational scope.
• Agent-Based Debiasing:
  • Results were mixed and architecture-dependent.
  • The largest "Judge" models struggled to effectively guide smaller models (e.g., Gemma-3) without causing performance degradation, likely due to instruction-following capability gaps between small and large variants.

D. Position Bias

• Statistical analysis confirmed no significant position bias. The order of input documents (random vs. lead-bias) did not significantly impact fairness or quality metrics.

4. Key Contributions

1. FairNews Dataset: A novel, labeled dataset for multi-document summarisation that enables systematic political bias evaluation.
2. Multi-Dimensional Framework: A comprehensive evaluation suite covering coarse-grained (neutrality, ratio) and fine-grained (entity coverage, sentiment) fairness dimensions.
3. Empirical Evidence Against "Bigger is Better": Demonstrated that scaling up LLMs does not linearly improve fairness; mid-sized models often offer the optimal trade-off between performance, fairness, and computational cost.
4. Limitations of Prompting: Highlighted that current prompt-based debiasing is insufficient for fine-grained sentiment preservation and can inadvertently harm entity coverage.

5. Significance and Implications

• For Practitioners: The study advises against blindly scaling models for fairness. Instead, it recommends using mid-sized models (e.g., 7B–12B range) combined with structured prompts for the best fairness-efficiency trade-off.
• For Research: It underscores the need for architecture-aware debiasing strategies. Since sentiment preservation is resistant to prompting, future work must explore internal representation editing or more sophisticated generation strategies beyond surface-level instructions.
• Democratic Impact: By providing a rigorous framework to detect and measure political bias, this work contributes to the development of more responsible AI systems that support informed public discourse rather than amplifying polarization.

Conclusion

The paper concludes that fairness in multi-document news summarisation is a multi-faceted challenge that cannot be solved by simply increasing model parameters. It requires targeted, architecture-specific interventions and a recognition that different fairness dimensions (e.g., coverage vs. sentiment) often involve inherent trade-offs. The "sweet spot" for deployment lies in mid-sized models, which currently offer the most robust balance of quality and equity.