Can Commercial LLMs Be Parliamentary Political Companions? Comparing LLM Reasoning Against Romanian Legislative Expuneri de Motive

1. Problem Statement

The paper addresses the empirical question of whether commercial Large Language Models (LLMs) can serve as reliable political advisory tools for legislators. While existing literature focuses on LLMs as objects of study (e.g., detecting ideological bias) or simulators of voting behavior, there is a lack of practitioner-oriented evaluation regarding their ability to generate legislative reasoning (specifically expuneri de motive or explanatory memoranda) in real-world, non-Anglophone contexts.

The authors frame the relationship between a legislator (Principal) and an LLM (Agent) through Principal-Agent Theory and Bounded Rationality. The core challenge is information asymmetry: legislators cannot inspect the LLM's training data or alignment objectives, creating risks of adverse selection and moral hazard. The study investigates whether LLMs reduce the information deficit for legislators or amplify it through "contextual ignorance" and confabulation.

2. Methodology

Dataset

• Source: 15 real law proposals from the Romanian Senate and Chamber of Deputies.
• Structure: Each sample pairs a structured law change (amendments, references) with its official explanatory memorandum (expuneri de motive), serving as the ground truth.
• Scope: Covers labor, criminal, and administrative law; document lengths range from 2,000 to 35,000 characters.
• Language: Romanian (a low-resource language relative to English in LLM training data).

Models Evaluated

Six models across three provider families were tested to analyze the effects of provider, scale, and architecture:

1. OpenAI: GPT-5-mini (Reasoning/CoT) and GPT-5-chat (Standard).
2. Anthropic: Claude Haiku 4.5 (Efficient Frontier).
3. Meta (Open-Weight): Llama 4 Maverick (17B/400B MoE), Llama 3.3 70B (Dense), and Llama 3.1 8B (Dense).

Experimental Design

• Generation: Each model generated 5 independent runs per law proposal (Total $N=450$ traces) using a fixed prompt instructing the model to act as a Romanian legislative expert.
• Evaluation Framework (Dual Approach):
  1. LLM-as-Judge: A custom prompt evaluated outputs on a 5-point scale across three dimensions:
     • Argument Coverage: Did it capture key arguments?
     • Factual Alignment: Accuracy of facts, dates, and citations.
     • Exposé des Motifs Closeness: Holistic semantic similarity.
  2. Programmatic Metrics: Nine metrics including ROUGE, Jaccard similarity, embedding cosine similarity, and domain-specific legal entity overlap (article numbers, law citations).

3. Key Contributions

1. First Practitioner-Oriented Evaluation: Provides the first empirical assessment of commercial LLMs as political advisory tools using real parliamentary data from a non-Anglophone EU member state (Romania).
2. Reframing Political Bias: Shifts the focus from stable ideological bias (detected in abstract quizzes) to task-dependent confabulation. The study argues that the primary risk is not ideological leaning, but "contextual ignorance" where models generate plausible but unfounded reasoning for politically idiosyncratic proposals due to training data gaps.
3. Theoretical Innovation: Introduces the concept of Cascading Bounded Rationality. This describes how failures compound across three layers:
   • The Bounded Principal (legislator with limited time).
   • The Bounded Agent (LLM with limited training coverage).
   • The Bounded Monitor (LLM-as-Judge lacking domain expertise).
4. Methodological Insight: Demonstrates the Metric Divergence Problem, showing that standard embedding-based similarity metrics (e.g., Cosine Similarity) correlate poorly with semantic quality in legal reasoning, often failing to distinguish between high-quality and hallucinated outputs.

4. Key Results

A. The Two-Tier Structure

A sharp performance gap exists between "Frontier" models and "Open-Weight" models:

• Tier 1 (Frontier): GPT-5-mini, GPT-5-chat, and Claude Haiku 4.5 achieved statistically indistinguishable semantic closeness scores (Mean $\approx$ 4.6/5.0).
• Tier 2 (Open-Weight): Llama models clustered significantly lower (Mean $\approx$ 3.75–4.00).
• Significance: The gap (Cohen's $d > 1.4$) is driven by training data composition and alignment, not parameter scale. Scaling Llama from 8B to 70B parameters yielded negligible gains ($d=0.41$).

B. The "Competence Trap" & Asymmetric Profiles

• Claude Haiku 4.5 matched GPT-5 on overall closeness but exhibited a significant factual alignment gap ($\Delta = 0.36$). It correctly identified argument structures but hallucinated specific details (article numbers, dates).
• Implication: This creates a "competence trap" where fluent, structurally correct outputs mask factual errors, potentially deceiving time-constrained legislators (System 1 heuristic failure).

C. Reasoning vs. Knowledge

• GPT-5-mini (with Chain-of-Thought) outperformed GPT-5-chat on Argument Coverage but not on overall quality.
• Conclusion: The bottleneck for legislative reasoning is domain knowledge, not reasoning depth. CoT helps enumerate arguments but cannot fix missing factual grounding.

D. Task Difficulty Hierarchy

• Easy: Standardized templates (e.g., EU directive transpositions) where rationale follows predictable structures.
• Hard: Politically idiosyncratic proposals (e.g., specific religious funding) requiring deep contextual knowledge absent in training data.
• Result: All models struggled with "hard" tasks, generating plausible but unfounded reasoning.

E. Metric Divergence

• Cosine Similarity showed near-zero correlation ($r = -0.04$) with human-aligned judge scores.
• Length Bias: Judge scores correlated strongly with response length ($r = 0.56$), suggesting a verbosity bias in the evaluation mechanism.

5. Significance and Implications

• For Governance: Commercial LLMs can function as effective advisory tools for standardized, well-templated legislative tasks (Tier 1 models), acting as bounded rationality mitigators. However, they are unreliable for idiosyncratic political proposals, where they risk introducing "contextual ignorance" disguised as expert reasoning.
• For Policy & Regulation: The study challenges the narrative that LLM risk is primarily ideological. Instead, it highlights epistemological risk: the inability of models to distinguish between valid and invalid reasoning in under-represented domains.
• For AI Development: General-purpose benchmarks and parameter scaling are poor predictors of performance in specialized legal/political domains. Future improvements require better domain-specific training data and alignment, not just larger models.
• For Monitoring: The "Cascading Bounded Rationality" framework suggests that relying on automated LLM judges to evaluate other LLMs in legal contexts is insufficient. Human-in-the-loop verification remains critical, particularly for tasks outside standard templates.

In conclusion, the paper argues that while frontier LLMs show promise as political companions, their deployment requires a task-selection strategy and robust institutional design to manage the risks of structural misalignment and confabulation.