💬 NLP

ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech

This paper introduces ParliaBench, a comprehensive framework and dataset for evaluating and improving LLM-generated parliamentary speeches by combining standard linguistic metrics with novel political authenticity measures, demonstrating that fine-tuning significantly enhances models' ability to produce ideologically consistent and coherent political content.

Original authors: Marios Koniaris, Argyro Tsipi, Panayiotis Tsanakas

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marios Koniaris, Argyro Tsipi, Panayiotis Tsanakas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to sound like a politician. You don't just want the robot to speak in perfect grammar; you want it to sound like a specific politician from a specific party, arguing for a specific cause with the right amount of passion and style.

This paper, ParliaBench, is essentially a "driving test" and a "training manual" for robots trying to do exactly that: generate authentic parliamentary speeches.

Here is the breakdown of what the authors did, using simple analogies:

1. The Problem: The Robot is Too Generic

Currently, if you ask a standard AI to write a speech, it might sound smooth and polite, but it lacks "soul." It doesn't know the difference between a Labour Party member and a Conservative Party member.

The Analogy: Imagine a robot actor who can recite Shakespeare perfectly but sounds exactly the same whether they are playing a king, a beggar, or a villain. They have the words, but they lack the character.
The Gap: Existing tests for AI only check if the words make sense (grammar) or if they are similar to other texts. They don't check if the AI is "politically authentic."

2. The Solution: Building a "Political Gym" (The Dataset)

To fix this, the authors built a massive training ground using real speeches from the UK Parliament.

The Collection: They gathered nearly 450,000 real speeches from the UK Parliament.
The Cleanup: They acted like librarians, organizing these speeches by who said them, which party they belonged to, what topic they were discussing, and when they happened. They filtered out the "noise" (like procedural announcements) to keep only the real debates.
The Result: A clean, organized library of 448,000 speeches covering everything from Brexit to the pandemic, ready to teach AI how real politicians talk.

3. The Training: Teaching the Robots

The researchers took five different types of AI models (the "robots") and gave them a crash course using this library.

The Method: They didn't just show the robots the speeches; they fine-tuned them. Think of this as taking a general student and giving them a specialized boot camp to become a political speechwriter.
The Output: After training, these robots generated 28,000 new speeches to see if they actually learned anything.

4. The Exam: A New Way to Grade (The Evaluation Framework)

This is the most important part of the paper. The authors realized that standard grading (checking for spelling or sentence structure) isn't enough. They created a three-part grading system:

Linguistic Quality (The "Grammar Check"): Does it sound like human English? Is it confusing or repetitive?
Semantic Coherence (The "Logic Check"): Does the speech make sense? Do the arguments flow logically from start to finish?
Political Authenticity (The "Vibe Check"): This is the new invention.
- The "Left-Right" Compass: They created a new metric called Political Spectrum Alignment. Imagine a line from Far-Left to Far-Right. The test checks if the AI's speech lands in the correct spot on that line for the party it's pretending to be.
- The "Party ID" Badge: They created another metric called Party Alignment. This checks if the speech sounds like it came from the specific party (e.g., does it sound like a Labour speech, or does it accidentally sound like a Conservative one?).

They used a "Judge AI" (another robot trained to be a critic) to read the speeches and give them a score from 1 to 10 on how authentic they felt, just like a human judge at a debate.

5. The Results: Did the Training Work?

The results were a clear "Yes."

Before Training: The robots sounded generic and often got the political "vibe" wrong.
After Training: The robots got significantly better. They spoke more naturally, argued more logically, and, most importantly, they sounded like the specific politicians they were supposed to mimic.
The Proof: The new "Political Authenticity" metrics (the Compass and the Badge) were very good at spotting the difference between a trained robot and an untrained one. They could tell when a robot was "faking" a political stance.

6. The Takeaway

The paper concludes that if you want an AI to write political speeches, you can't just use a general AI. You need to:

Feed it a massive amount of real, specific data (like the UK Parliament speeches).
Fine-tune it specifically for that job.
Grade it using special tests that check for political "soul," not just grammar.

Important Note from the Authors:
The paper explicitly states this is for research and education only. They are not suggesting these robots should actually run parliaments or replace real politicians. The goal is to understand how AI works in political contexts and to build better tools for studying them, not to deploy them in real democratic processes.

In short: ParliaBench is a new gym, a new teacher, and a new grading system that helps AI learn to sound like a real politician, and proves that with the right training, they can actually do it.

Technical Summary: ParliaBench

Problem Statement

Generating authentic parliamentary speech presents challenges distinct from standard text generation tasks. While large language models (LLMs) have demonstrated utility in political applications such as sentiment analysis, election forecasting, and synthetic survey generation, they struggle with the specific requirements of parliamentary discourse. Unlike general text, parliamentary speeches demand not only linguistic quality but also political authenticity and ideological consistency. A speech by a Labour MP must differ fundamentally from one by a Conservative counterpart, not merely in policy stance but in their approach to governance.

Current evaluation methods for text generation rely on surface-level similarity metrics (e.g., BLEU, ROUGE) or semantic embedding approaches (e.g., BERTScore, MoverScore). These fail to capture political authenticity. Furthermore, while domain-specific benchmarks exist for legislative summarization or opinion alignment, there is no established framework to simultaneously assess ideological consistency and parliamentary conventions in generative tasks.

Methodology

The authors propose ParliaBench, a comprehensive framework comprising a dataset, an evaluation protocol, and baseline model performance.

1. Dataset Construction (ParliaBench Dataset)

Source: Derived from the UK subset of the ParlaMint corpus, retaining raw XML files for granular control.
Processing Pipeline:
- Ingestion & Parsing: Isolated individual speeches while preserving hierarchy.
- Metadata Enrichment: Extracted speaker identity, political affiliation, chamber, and session dates. Crucially, temporal alignment was applied to cross-reference speech dates with affiliation histories, as MPs often change roles.
- Filtering: Distinguished substantive content from procedural noise. A 1,000-speech minimum threshold was applied to political affiliations, reducing the set from 28 to 11 stable groups for training.
- Thematic Classification: Mapped content to EuroVoc (standard for European parliamentary systems) rather than the original CAP classification, utilizing hybrid mapping rules for complex categories.
Statistics: The final dataset contains 447,778 speeches from 1,901 unique speakers across 11 political affiliations, totaling ~99.94 million words. It covers diverse political contexts, including Brexit and the COVID-19 pandemic.

2. Model Development

Architectures: Five distinct LLMs were selected: Mistral 7B v0.3, Llama 3.1 8B, Gemma 2 9B, Qwen2 7B, and YI 6B.
Fine-Tuning: Models were fine-tuned using QLoRA (Quantized Low-Rank Adaptation) with $r=16, \alpha=16$ over 2 epochs. Training inputs were structured with political metadata (party, topic, orientation, section, house).
Generation: A total of 27,560 speeches were generated using nucleus sampling ( $T=0.7, p=0.85$ ) across baseline and fine-tuned models, ensuring identical input prompts for fair comparison.

3. Evaluation Framework

The framework evaluates generation quality across three dimensions using a dual-track approach: Computational Metrics and LLM-as-a-Judge assessments.

Linguistic Quality:
- Computational: Perplexity (PPL), Distinct-n (Dist-N), Self-BLEU.
- LLM-Judge: Coherence (J_Coh), Conciseness (J_Conc).
Semantic Coherence:
- Computational: GRUEN, BERTScore, MoverScore (compared against top-5 human speeches in the training set).
- LLM-Judge: Relevance (J_Rel).
Political Authenticity (Novel Metrics):
- Computational:
  - Political Spectrum Alignment (PSA): Measures ideological positioning on a Left-Right scale (-6 to +6). It combines semantic similarity to a political orientation centroid with the distance of that orientation from the expected one.
  - Party Alignment (Party Align): Measures cosine similarity between the generated speech and a party-specific centroid.
- LLM-Judge: Authenticity (J_Auth), Political Appropriateness (J_PolApp), Overall Quality (J_Qual).
Consistency: Cross-Context Stability is measured via the coefficient of variation across political parties, topics, and orientations.

Key Results

The evaluation of 27,560 generated speeches yielded the following findings:

Impact of Fine-Tuning: Fine-tuned models consistently outperformed baselines. Pairwise t-tests confirmed statistically significant improvements in 45 out of 70 metric comparisons.
- Llama 3.1 8B and YI 6B showed the most notable improvements (79% of metrics for YI).
- Gemma 2 and Qwen2 exhibited trade-offs, improving in some categories while declining in others.
Novel Metrics Validation: The proposed PSA and Party Align metrics demonstrated strong discriminative power. All five models showed significant improvements in PSA ( $p < 0.001$ ) and Party Align after fine-tuning, validating that these embedding-based metrics capture political dimensions inaccessible to conventional NLP metrics.
Contextual Performance:
- Parties: Major parties (Conservative, Labour) achieved stable performance due to abundant training data. Minor parties showed higher variability.
- Topics: Science and Geography were the most difficult domains, likely due to terminological specialization. Economics and European Union topics yielded the highest scores.
- Ideology: Centrist positions dominated the dataset and achieved higher scores. Extreme positions required specialized training approaches.
Stability: Fine-tuned models maintained high cross-context stability (91.4–96.2), with Mistral achieving the highest consistency despite absolute performance trade-offs.

Significance and Claims

The paper claims to present the first benchmark resource specifically designed for evaluating parliamentary speech generation. Its significance lies in:

Standardization: Providing a curated dataset of 448k speeches and a reproducible evaluation framework.
Methodological Advancement: Introducing Political Spectrum Alignment and Party Alignment as novel metrics that quantify ideological positioning, addressing the gap in existing evaluation methods which focus on surface-level similarity.
Empirical Evidence: Demonstrating that domain-specific fine-tuning is essential for generating politically authentic speech and that architectural choices (e.g., context window size) significantly impact performance consistency.

The authors explicitly state that their work is intended for research and educational purposes to assist human experts, not for deployment in actual democratic processes. They acknowledge limitations, including the lack of human validation for argument structure or factual accuracy and the inherent biases of the LLM-as-a-Judge approach. Future work is proposed to include multilingual evaluation, human validation protocols, and systematic assessment of political bias.