🤖 machine learning

Fair Representation in Parliamentary Summaries: Measuring and Mitigating Inclusion Bias

This paper evaluates five large language models summarizing European Parliament debates, revealing systematic biases against speakers based on order, language, and political affiliation, and proposes a hierarchical summarization method to effectively mitigate these inclusion biases where prompting strategies fail.

Original authors: Eoghan Cunningham, James Cross, Derek Greene

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eoghan Cunningham, James Cross, Derek Greene

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the European Parliament as a massive, bustling town hall meeting where hundreds of people from different countries, speaking different languages, and holding different political views gather to debate important issues. The transcripts of these meetings are thousands of pages long—too long for any regular citizen to read.

Enter Large Language Models (LLMs), the "AI scribes." Their job is to read these massive transcripts and write a short, easy-to-read summary so the public can understand what happened. The goal is to make democracy more accessible.

However, this paper asks a critical question: Is the AI scribe being fair? Or is it accidentally (or intentionally) ignoring certain people, twisting their words, or only listening to the loud voices at the start and end of the meeting?

Here is a breakdown of the paper's findings using simple analogies.

1. The "Middle Curse": The Lost Voices

The Problem:
Imagine a long line of people waiting to speak at a town hall. The AI scribe tends to listen very carefully to the first few people and the last few people. But the people speaking in the middle? The AI often zones out on them. It's like a radio that has a strong signal at the start and end of a song, but the middle part is static.

In technical terms, this is called the "Lost-in-the-Middle" problem. The study found that speakers in the middle of a debate were systematically ignored or poorly represented in the summaries, regardless of how important their points were.

The Fix:
The researchers tried a new method called Hierarchical Summarization.

Old Way (Flat): The AI tries to swallow the whole 50-page document at once and spit out a summary. It gets overwhelmed and forgets the middle.
New Way (Hierarchical): The AI acts like a smart editor. First, it summarizes each individual speech into a tiny note. Then, it groups those notes by topic (e.g., "All points about taxes," "All points about education"). Finally, it combines those topic notes into one final summary.
Result: This "step-by-step" approach fixed the problem. By breaking the task down, the AI stopped ignoring the middle speakers. It was like giving the scribe a checklist so they didn't miss anyone.

2. The "Language Barrier": The Translation Gap

The Problem:
The European Parliament has 24 official languages. The study found that if a politician spoke in a "low-resource" language (a language with less data available on the internet for the AI to learn from, like some smaller European languages), the AI did a worse job representing them.

Even worse, this bias happened even when the AI was summarizing official English translations of those speeches. It's as if the AI had a "bad ear" for certain accents or dialects, and that bias persisted even after the words were translated. The speakers using these languages were less likely to be included accurately in the final summary.

3. The "Political Filter": The Left-Wing Bias

The Problem:
The researchers also checked if the AI favored certain political parties. They found a subtle but clear bias: Left-of-center parties (generally more progressive or social-democratic groups) were represented more fully than right-of-center parties.

The AI didn't necessarily lie about what the right-wing parties said; it just omitted more of their points. It was like a camera that focused perfectly on the people on the left side of the stage but kept the people on the right side slightly out of frame.

Note: This bias was hidden at first because the "Middle Curse" was so bad that it drowned out the political bias. Once they fixed the "Middle Curse" using the step-by-step method, the political bias became visible.

4. The "Magic Prompt" Didn't Work

The researchers tried a simple trick: they told the AI, "Hey, please make sure you give equal attention to everyone!"
Result: It didn't work. The AI ignored the instruction. This is like telling a distracted student, "Please pay attention to the whole book," without actually changing how they study. The bias is baked into how the model processes information, not just a lack of politeness.

The Big Takeaway

This paper is a warning and a guide for the future of AI in democracy.

The Warning: If we just plug AI into government systems without checking, we might accidentally silence the middle voices, disadvantage speakers of smaller languages, and skew political representation.
The Guide: We can't just rely on "magic prompts." We need to change how the AI works. By breaking big tasks into smaller, structured steps (like the hierarchical method), we can fix some of the biggest fairness issues.

In short: AI is a powerful tool for making democracy accessible, but like any tool, it needs to be calibrated carefully. If we don't, the "summary" of our democracy might only reflect the voices of the first, the last, the English-speaking, and the left-leaning, leaving everyone else in the dark.

1. Problem Statement

The paper addresses the critical challenge of fairness and inclusion bias when using Large Language Models (LLMs) to summarize parliamentary proceedings. While LLMs offer a promising solution to make democratic processes more accessible to the public, they risk distorting political discourse through systematic biases. The authors identify three specific dimensions of bias that threaten the integrity of these summaries:

Positional Bias (The "Lost-in-the-Middle" Problem): LLMs tend to under-represent speakers who intervene in the middle of a debate, favoring those at the beginning and end.
Cross-lingual Bias: Speakers using low-resource languages (or languages with fewer training data points) receive less faithful representation compared to high-resource languages (like English).
Partisan Bias: LLMs may systematically favor or exclude speakers based on their political affiliations, leading to unequal inclusion or misrepresentation of different political groups.

A significant gap in existing literature is that standard evaluation metrics (e.g., ROUGE, BERTScore) measure overall semantic overlap but fail to penalize misattribution (assigning a statement to the wrong speaker) or omission (ignoring a speaker entirely).

2. Methodology

Data and Models

Dataset: The study uses 1,242 speeches from 50 plenary debates of the 6th European Parliament (2005–2006). The dataset includes original multilingual transcripts and official English translations, covering 708 speakers from 7 political groups and 24 official languages.
LLMs Evaluated: Five models were tested, including proprietary (Claude Sonnet 4, Claude Haiku 4.5, GPT-5) and open-weight models (Phi-4, Gemma3).
Evaluation Model: An open-weight model (Qwen3) was used specifically for the "reconstructor" function to extract speaker-specific content from summaries.

Experimental Design: Summarization Approaches

The authors compared four distinct summarization workflows to test mitigation strategies:

Flat Summarization (Baseline): Directly summarizing the entire debate text in one step.
Flat + Prompt: Same as above, but with an explicit system prompt instructing the model to "give equal attention to all speakers."
One-Level Hierarchical Summarization:
- Step 1: Summarize each individual intervention (speech) into a structured format (Headline, Issue, Position, Argument, Proposal, Quotes).
- Step 2: Aggregate these structured summaries into a final debate summary.
Two-Level Hierarchical Summarization:
- Step 1: Summarize individual interventions.
- Step 2: Perform thematic aggregation (e.g., grouping all "Issues," then all "Positions," etc.) before the final aggregation.

Evaluation Framework: Attribution-Aware Metrics

To address the limitations of standard metrics, the authors developed a reconstruction-based evaluation framework:

Reconstruction Function ( $f_{REC}$ ): Given a final debate summary, the model attempts to reconstruct the specific contribution of a single speaker.
Fidelity Measurement: The reconstructed summary is compared to the original intervention using BERTScore.
Bias Decomposition:
- Inclusion Bias (Recall): Measured by how much of the original speaker's content is recovered in the reconstruction. Low recall indicates systematic omission.
- Hallucination Bias (Precision): Measured by how much of the reconstructed content is actually supported by the original speech. Low precision indicates fabrication or misattribution.

3. Key Contributions

Attribution-Aware Evaluation Framework: The paper extends Steen and Markert's bias framework to multi-speaker contexts, distinguishing between inclusion bias (omission) and hallucination bias (misrepresentation) at the speaker level.
Mitigation of Positional Bias: The authors demonstrate that hierarchical summarization (specifically using domain-informed structures like "Issue/Position/Argument") significantly mitigates the "lost-in-the-middle" problem, whereas simple prompting does not.
Novel Partisan Analysis: The study reveals that once positional bias is controlled for, a distinct partisan inclusion bias emerges, where left-of-center parties are more frequently included in summaries than right-of-center parties.
Cross-Lingual Bias Quantification: The work provides systematic evidence that language resource levels significantly impact representation quality, a bias that persists even when using official translations or hierarchical methods.

4. Key Results

Positional Bias ("Lost in the Middle")

Finding: Under flat summarization, all models (except GPT-5) exhibited a U-shaped attention pattern: high fidelity for early and late speakers, but near-zero fidelity for middle speakers.
Mitigation: Hierarchical summarization successfully flattened this curve across all models.
Ineffective Strategy: Explicit prompting ("give equal attention") had no significant effect on reducing positional bias.

Cross-Lingual Bias

Finding: Speakers using low-resource languages received significantly lower fidelity scores than English speakers.
Persistence: This bias persisted regardless of whether the input was the original multilingual text or the official English translation. This suggests the bias stems from the LLM's training data limitations regarding low-resource languages, not just the translation process.
Interaction: Positional bias was a stronger predictor of fidelity than language bias.

Partisan Bias

Finding: Once the "lost-in-the-middle" effect was removed via hierarchical summarization, a partisan bias became visible.
Direction: Models (particularly Claude and Phi-4) showed a bias toward left-of-center parties, exhibiting higher recall (more complete inclusion of their arguments) compared to right-of-center parties.
Mechanism: The bias manifested primarily as inclusion bias (omission of right-leaning speakers) rather than hallucination (fabrication of left-leaning speakers).

5. Significance and Conclusion

This paper underscores that architectural changes to summarization pipelines (specifically hierarchical, domain-informed structures) can effectively solve algorithmic attention biases like the "middle curse." However, these structural fixes cannot compensate for biases rooted in training data, such as language resource disparities and political partisanship.

Implications:

For Deployment: Simply deploying LLMs for democratic transparency is insufficient; systems must employ hierarchical processing to ensure all voices are heard.
For Evaluation: Standard metrics are inadequate for political summarization. New metrics must explicitly measure speaker-level attribution and inclusion.
For Ethics: There is a need for domain-sensitive oversight. While hierarchical methods improve fairness regarding when people speak, they do not resolve inherent biases regarding who speaks (language and politics), requiring further research into training data curation and prompt engineering for specific demographic groups.

The study concludes that while LLMs hold promise for enhancing democratic engagement, their current deployment in multilingual, multi-party environments requires rigorous, attribution-aware evaluation to prevent the systematic silencing of specific groups.