Persona-driven Simulation of Voting Behavior in the European Parliament with Large Language Models

1. Problem Statement

The paper addresses the challenge of simulating the voting behavior of Members of the European Parliament (MEPs) using Large Language Models (LLMs). While LLMs have demonstrated capabilities in understanding political discourse, they inherently exhibit a progressive, left-leaning bias by default. This bias limits their ability to accurately represent the diverse political spectrum of the EU, particularly for right-wing or far-right groups.

Existing methods for predicting legislative votes often rely on:

• Historical data: Requiring vast amounts of past voting records or social media data, which is not always available.
• Traditional ML: Approaches like Random Forests that often perform only marginally better than majority baselines.
• US-centric studies: Most LLM-based agent research focuses on the two-party US system, which lacks the multi-party complexity of the European Parliament.

The core research question is: Can zero-shot persona prompting with limited, publicly available information accurately simulate individual MEP voting decisions and aggregate group positions, even for political groups that diverge from the LLM's inherent bias?

2. Methodology

Data Collection

• Source: The authors collected data from HowTheyVote, focusing on roll-call votes where individual MEP votes are recorded.
• Dataset: 47 key proposals from the 2024 European Parliament session, resulting in 27,770 individual voting decisions from 710 politicians.
• Labels: Votes are categorized as FOR, AGAINST, or ABSTENTION.
• Context: For each proposal, the authors collected speeches from group representatives. To avoid bias, speaker identities and group affiliations were anonymized (e.g., replaced with [GROUP]).

Model Selection

To prevent data leakage (ensuring models were not trained on the 2024 votes), the authors selected models with training cut-offs prior to January 16, 2024:

• Llama3-70B and Llama3-8B (Western models).
• Qwen2.5-72B and Qwen2.5-7B (Non-Western models to test bias generalizability).
• Note: Mixtral and Phi-3 were tested but rejected due to poor adherence to prompt formats.

Prompt Engineering Strategies

The study employed two main strategies for Persona Construction:

1. Attribute Prompting: Providing structured attributes: Name, Gender, Age, Birthplace, Country, National Party, and European Group.
2. Wikipedia Summary: Using Llama3-70B to summarize the English Wikipedia article of each MEP to provide richer context.

Proposal Information: The model was fed anonymized speeches representing the debate on a specific proposal.

Response Formats:

• No Reasoning (nr): Direct classification (FOR/AGAINST/ABSTENTION).
• Reasoning (r): The model first generates an open-text chain of thought explaining its decision, followed by the classification.

Evaluation Metrics

Due to class imbalance (77% FOR, 17% AGAINST, 6% ABSTENTION), the primary metric was the Weighted F1 Score.

3. Key Contributions

1. First Large-Scale EU Simulation: This is the first study to systematically simulate the voting behavior of the entire 2024 European Parliament using LLM agents, moving beyond the US two-party context.
2. Persona Prompting Efficacy: Demonstrates that zero-shot persona prompting can effectively steer LLMs away from their default biases to simulate specific political identities.
3. Ablation Studies on Bias and Robustness:
   • Counterfactual Debates: The authors generated counterfactual speeches (opposite stance) to test if the model's vote would change based on argumentation or if it remained anchored to the persona.
   • Attribute Analysis: Identified which specific attributes (e.g., National Party vs. Age) drive the most accurate predictions.
4. Open Release: The authors released the persona dataset and simulation code to facilitate further research in political science and NLP.

4. Key Results

Overall Performance

• The best-performing configuration (Llama3-70B with Attribute Prompting and Reasoning) achieved a Weighted F1 score of 0.793.
• This significantly outperforms the majority baseline (always predicting "FOR"), which scores 0.666.
• Stability: Predictions were highly stable across multiple runs (standard deviation < 0.002).

Group-Specific Performance

• Best Prediction: Center-left and progressive groups (S&D, Renew, Greens/EFA) were predicted most accurately (F1 > 0.90 for S&D and Renew).
• Worst Prediction: Groups at the political extremes (ID, ECR, GUE/NGL) were predicted least accurately (F1 ~0.53–0.59).
• Bias Persistence: The model struggled to simulate "AGAINST" votes for right-wing groups, often defaulting to the model's inherent progressive bias.

Impact of Reasoning and Attributes

• Reasoning: The "Reasoning" (chain-of-thought) strategy significantly improved performance for most models, particularly in identifying minority classes like ABSTENTION and AGAINST. For Qwen-7B, reasoning shifted predictions from 95% "FOR" to a more balanced distribution, improving the F1 score for abstentions from 0.02 to 0.11.
• Attributes:
  • National Party was identified as the single strongest predictor. Providing the National Party and European Group alone achieved performance nearly identical to providing all attributes.
  • Wikipedia vs. Attributes: Smaller models (Llama-8B, Qwen-7B) benefited slightly more from detailed Wikipedia summaries, while larger models (Llama-70B) performed best with concise attribute lists.

Counterfactual Analysis

• When presented with counterfactual debates (arguments opposite to the real ones), models simulating centrist personas (S&D, Renew) largely maintained their original voting stance.
• Models simulating extremist personas (ID, ECR, GUE/NGL) were more susceptible to changing their votes based on the counterfactual arguments, suggesting LLMs struggle to consistently replicate the rigid ideological positions of politically divergent personas.

Limitations Identified

• Abstention Prediction: Models consistently under-predicted abstentions (F1 ~0.11), likely because real-world abstentions are often strategic or due to external factors (illness, conflict of interest) not captured in the prompt.
• Data Bias: The model's inherent left-leaning bias made it difficult to accurately simulate far-right voting patterns, even with persona prompting.

5. Significance and Implications

• Political Science: The study validates that National Party affiliation is the primary determinant of MEP voting behavior, aligning with existing political research (e.g., Willumsen, 2023).
• LLM Alignment: It highlights the difficulty of "steering" LLMs to represent views that contradict their training data biases, even with explicit persona instructions.
• Methodological Advancement: The paper establishes a new baseline for political simulation using LLMs, demonstrating that open-ended reasoning followed by classification is superior to direct classification, especially for smaller models.
• Ethical Considerations: The authors discuss the risks of using LLMs to impersonate politicians, noting the potential for generating plausible but false arguments about a politician's motives, which could be used to spread misinformation.

In conclusion, the paper demonstrates that while LLMs cannot perfectly replicate the complex, strategic, and sometimes contradictory nature of human political decision-making (especially regarding abstentions and extreme ideologies), they can serve as powerful tools for approximating voting behavior and understanding the structural drivers (like party affiliation) of legislative outcomes in the European Parliament.