💬 NLP

Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation

This paper proposes a novel computational argumentation framework to evaluate the faithfulness of LLM-generated summaries of parliamentary debates by focusing on the formal preservation of reasoning structures, addressing the limitations of existing metrics in capturing argumentative consistency.

Original authors: Eoghan Cunningham, Derek Greene, James Cross, Antonio Rago

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eoghan Cunningham, Derek Greene, James Cross, Antonio Rago

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the European Parliament as a massive, chaotic kitchen where hundreds of chefs (politicians) are arguing over a giant, complex recipe (a new law). They shout about adding salt, removing sugar, or changing the cooking time. The final recipe has dozens of specific instructions (provisions).

Now, imagine you are a regular person trying to understand what happened in that kitchen. You can't read the 500-page transcript of everyone shouting. So, you ask a super-smart robot (an AI or Large Language Model) to write a short summary for you.

The Problem:
The robot writes a nice, short summary. But is it telling the truth?

Did it say the chefs wanted to add salt when they actually wanted to remove it?
Did it hide the fact that half the kitchen was screaming "No!" about a specific ingredient?
Did it make a controversial ingredient look like everyone loved it?

Current ways of checking if a summary is good are like checking if the summary uses the same words as the original. If the robot uses different words but gets the meaning wrong, those old checks fail. They are like judging a movie summary by counting how many times the word "explosion" appears, rather than checking if the hero actually died.

The Solution: The "Argument Map"
This paper proposes a new way to check the robot's work. Instead of just looking at words, they turn the debate into a logical battle map (called a Quantitative Bipolar Argumentation Framework).

Think of it like a Tug-of-War game:

The Rope: The specific parts of the law being debated (e.g., "Extend the health agency's power").
The Teams:
- Team Green (Support): Chefs pulling the rope one way, saying "This is good!"
- Team Red (Attack): Chefs pulling the other way, saying "This is bad!"
The Score: The final position of the rope depends on how many people are on each team and how hard they are pulling.

How the New System Works:
The researchers built a system that creates two maps:

Map A: The map of the real debate (the full transcript).
Map B: The map of the robot's summary.

Then, they compare the two maps using five simple rules (properties) to see if the summary is a "faithful" representation of the fight:

Rule 1: Did anyone get left out? (Relevance) Did the summary forget to mention a chef who was shouting loudly?
Rule 2: Is the score balanced? (Pro-Con Ratio) If 10 people said "Yes" and 2 said "No" in the real debate, does the summary show that same 10-to-2 split? Or did it accidentally show 10 "Yes" and 0 "No"?
Rule 3: Who won the tug-of-war? (Acceptability) In the real debate, did the "Yes" team win? Did the summary also show the "Yes" team winning?
Rule 4: Is the intensity right? (Balance) If the "Yes" team was winning by a little bit in reality, does the summary show them winning by a huge landslide? (This is a common AI mistake: making things look more certain than they are).
Rule 5: Is the exact score close? (Accuracy) If the real score was 55% "Yes," is the summary's score close to 55%, or is it wildly off?

What They Found:
They tested this on real European Parliament debates using different AI models (like Claude and Llama).

The Old Way (Word Counting): The AI summaries looked great! They used similar words and sounded smart.
The New Way (The Battle Map): The summaries failed the logic test.
- The AI often ignored the "No" team. It would summarize a heated argument as if everyone agreed.
- It turned controversial issues (where the rope was in the middle) into settled facts (where the rope was pulled all the way to one side).
- Essentially, the AI was "smoothing over" the conflict, making the debate look much more peaceful and one-sided than it actually was.

The Takeaway:
If we want AI to help us understand democracy, we can't just ask it to "summarize the text." We have to ask it to "summarize the argument."

This paper gives us a ruler to measure if an AI is lying to us about who won the debate, who lost, and how hard they fought. It's like checking a referee's scorecard to make sure the game wasn't rigged, even if the referee's report looks well-written.

1. Problem Statement

Parliamentary debates generate vast amounts of complex content that are difficult for citizens and researchers to digest. While Large Language Models (LLMs) can automate the summarization of these debates at scale, current evaluation methods fail to assess faithfulness (or factual consistency) regarding the argumentative structure of the text.

Limitations of Existing Metrics: Traditional metrics like ROUGE (lexical overlap) and BERTScore (semantic similarity) correlate poorly with human judgments of consistency. They often reward summaries that are semantically similar but argumentatively distorted.
The Core Challenge: There is a lack of a formal, interpretable framework to determine if an LLM-generated summary faithfully preserves the reasoning used to justify or oppose specific policy outcomes (provisions) within a debate.

2. Methodology

The authors propose a novel evaluation framework grounded in Computational Argumentation, specifically using Quantitative Bipolar Argumentation Frameworks (QBAFs).

A. Formal Representation (QBAF)

The authors model parliamentary debates as QBAFs, defined as a quadruple $\langle X, A, S, \tau \rangle$ :

$X$ (Arguments): Divided into Proposal Arguments ( $X_p$ ), representing the specific policy provisions being debated, and Speech Arguments ( $X_s$ ), representing the statements made by speakers.
$A$ (Attacks) & $S$ (Support): Binary relations where speech arguments attack or support other speech arguments or proposal arguments.
$\tau$ (Base Scores): Proposal arguments are assigned a neutral base score (0.5) to allow the balance of attacks and supports to determine their final strength. Speech arguments have tunable base scores (e.g., 0.2).
Semantics: The framework uses DF-QuaD gradual semantics to calculate the final strength ( $\sigma$ ) of arguments, ranging from 0 to 1, based on the network of attacks and supports.

B. Evaluation Properties

The framework evaluates a summary ( $Q^*$ ) against the original debate ( $Q$ ) using five formal properties. These properties measure how well the summary preserves the argumentative logic:

Speech Argument Relevance: Measures the proportion of speech arguments connected to at least one proposal argument (ensuring arguments aren't orphaned).
Pro-Con Ratio: Checks if the ratio of supporting vs. attacking arguments for a specific proposal is preserved.
d-Preferability Consistency: A pairwise property checking if the "acceptability status" (based on d-preferred extensions) of proposal arguments remains consistent between the source and summary.
Balance Consistency: A gradual property checking if proposal arguments that had net support ( $>0.5$ ) or net opposition ( $<0.5$ ) in the source maintain that status in the summary.
$\epsilon$ -Accuracy: A stricter property measuring if the calculated strength of proposal arguments in the summary falls within a specific error margin ( $\epsilon$ ) of the original.

C. Experimental Setup

Dataset: 511 arguments from three European Parliament (EP) debates (2006–2008).
Argument Mining: Used Relation-Based Argument Mining (ARC) to identify attack/support relations between sentences.
- Evaluated four LLMs (Claude Sonnet 4.5, Claude Haiku 4.5, Qwen3, Llama 3.1) for ARC.
- Finding: Zero-shot/few-shot larger reasoning models (Claude Sonnet/Haiku) outperformed smaller fine-tuned models on this specific domain.
Comparison: The proposed QBAF metrics were compared against standard NLP metrics (ROUGE, BERTScore, Sentence-NLI) on LLM-generated summaries.

3. Key Contributions

Formal Framework for Faithfulness: Introduced a method to ground summary evaluation in the contested policy outcomes rather than general text similarity. This shifts the focus from "does it sound right?" to "does it preserve the logic of the debate?"
Five Formal Properties: Defined specific, quantifiable metrics (Relevance, Pro-Con Ratio, Consistency, Balance, Accuracy) that allow for a granular assessment of argumentative distortion.
Demonstration of Metric Failure: Showed that traditional metrics (ROUGE/BERTScore) can yield high scores for summaries that fundamentally misrepresent the argumentative balance (e.g., turning a contested issue into a consensus).
Scalable Argument Mining: Validated that modern LLMs can effectively mine argument relations from complex political texts, with zero-shot large models performing better than fine-tuned smaller models in this context.

4. Results

Argument Mining Performance:
- Claude Sonnet 4.5 achieved the highest accuracy (0.74) and F1 (0.78) in zero-shot settings.
- Llama 3.1 (fine-tuned) performed poorly (0.21 accuracy), highlighting that domain-specific fine-tuning on general ARC datasets does not transfer well to complex political debates without specific adaptation.
Summary Evaluation:
- Traditional Metrics: Showed high semantic coherence (high BERTScore, NLI) but low lexical overlap (low ROUGE), indicating successful abstraction. However, they failed to distinguish between summaries with different argumentative qualities.
- Proposed QBAF Metrics: Revealed significant degradation in argumentative faithfulness:
  - Pro-Con Balance: LLM summaries often failed to preserve the balance of evidence (e.g., Haiku summaries showed a Pro-Con ratio of 0.05 vs. 0.83 in the source).
  - Strength Distortion: Key proposals that were contested in the original debate (strength $\approx 0.5$ ) were often presented as overwhelmingly supported in the summaries (strength $\gg 0.5$ ).
  - Accuracy: The $\epsilon$ -accuracy property showed that summaries rarely maintained the precise strength of arguments (scores ranged from 0.00 to 0.52).
- Model Comparison: The larger model (Claude Sonnet) consistently outperformed the smaller model (Claude Haiku) across all argumentative properties, though both struggled with precise strength preservation.

5. Significance

Democratic Transparency: The framework provides a tool to ensure that automated summaries of parliamentary debates do not inadvertently bias public perception by distorting the balance of arguments.
Beyond Semantic Similarity: It establishes that for argumentative texts, semantic similarity is insufficient; the structure of reasoning must be preserved.
Future Direction: The paper lays the groundwork for "Argumentative Faithfulness" as a standard evaluation metric for LLMs in legal, political, and policy domains. It suggests that future work should focus on matching specific speech arguments across source and summary to detect omissions and hallucinations more precisely.

In conclusion, the paper argues that without a formal argumentation-based evaluation, LLM summaries of parliamentary debates risk being "plausible but misleading," and proposes a rigorous mathematical framework to detect and quantify such distortions.