Technical Summary: Polistemics – Evaluating LLMs as Information Mediators in Politics & Elections

Problem Statement

As Large Language Models (LLMs) increasingly function as primary information mediators for citizens—replacing or augmenting traditional search engines and Voting Advice Applications (VAAs)—there is no standardized framework to assess whether they mediate political information responsibly. Prior evaluations have largely treated this task as reproduction (checking if an LLM correctly recalls a party's stance) rather than mediation (how the model selects, frames, and communicates information under imperfect conditions). This oversight leaves critical epistemic dimensions unaddressed: specifically, how models handle absent, vague, or contradictory information, and whether they preserve the epistemic agency of citizens by avoiding unwarranted confidence or persuasive steering.

Methodology: The POLISTEMICS Benchmark

The authors introduce POLISTEMICS, a theory-grounded benchmark designed to evaluate LLMs as mediators of political information in electoral contexts. The methodology involves three core components:

1. Theoretical Framework: Epistemic Modesty

The evaluation is grounded in the normative standard of Epistemic Modesty, derived from the concept of citizen epistemic agency. This framework defines three core behavioral traits required for responsible mediation:

• Faithfulness: Accurately representing provided context without fabrication, distortion, or omission.
• Impartiality: Communicating information without unwarranted evaluative steering, framing, or normative judgment (preserving the pluralistic character of political rhetoric).
• Epistemic Calibration: Expressing certainty levels that match the limits of the evidence (e.g., abstaining or signaling uncertainty when evidence is missing or vague).

2. Experimental Design

The benchmark operationalizes these traits through controlled experiments simulating real-world information retrieval complexities.

• Data Source: Official party-position data from the 2025 German Bundestag election (7 parties, 38 statements) and the 2025 Dutch Tweede Kamer election (8 parties, 30 statements), sourced from Wahl-O-Mat and StemWijzer.
• Information Environments (IEs): The authors construct five controlled variants of the context to test model robustness:
  • Baseline: Clear, single-source evidence.
  • Inconclusive Conditions: Absent (no evidence), Vague (stance obscured by rhetorical evasion), and Contradictory (conflicting evidence from opposing parties).
  • Interfering Conditions: Noisy (evidence buried among distractors) and Counterfactual (evidence conflicting with the model's internal priors).
• Models Evaluated: Three state-of-the-art models powering public chatbots: Qwen3.6 Flash, GPT-5.4, and Claude Sonnet 4.6.
• Evaluation Protocol: An LLM-as-Judge framework using a panel of three distinct judge models (Gemini 3 Flash, DeepSeek V4, MiMo-V2.5) to score outputs on 13 sub-questions (SQs) across the three rubrics. Scores are aggregated into an Adherence Index per rubric and a final Epistemic Modesty Index (EMI).

3. Ablation Studies

To isolate the drivers of bias, the authors conducted targeted ablations:

• Label Anonymization: Replacing party names with generic labels to test if disparities are driven by party identity (priors) rather than evidence.
• Output Language: Forcing English output while keeping evidence in the native language to test if translation affects the "sanitization" of political rhetoric.

Key Results

The evaluation of the three models across the 2025 German and Dutch elections reveals that while aggregate scores are high, they mask systematic failures under specific conditions.

1. Overall Performance

• Claude achieved the highest Epistemic Modesty Index (92.6%), followed by GPT (91.2%) and Qwen (87.1%).
• Impartiality was the highest-scoring rubric (98%), while Epistemic Calibration was the most challenging (87%).
• Models performed near-ceiling levels on Interfering conditions (Noisy, Counterfactual), suggesting current models are robust to distraction and conflicting external evidence.

2. Failure Modes in Inconclusive Environments

Significant performance drops occurred in Inconclusive conditions, revealing distinct failure modes:

• Absent Evidence: GPT demonstrated ideal Epistemic Modesty by abstaining entirely. Claude and Qwen acknowledged missing evidence but frequently fell back on internal parametric knowledge to answer anyway, matching ground truth in many cases but violating epistemic boundaries.
• Vague Evidence: Performance dropped primarily due to failures in Epistemic Calibration (failing to signal uncertainty) and Faithfulness (fabricating stances). Models often answered with confidence despite the evidence being non-determinable.
• Contradictory Evidence: Models struggled to represent both sides. GPT and Qwen often resolved contradictions by committing to the first evidence chunk, while Claude referenced both positions but failed to fully cover either.

3. Even-Handedness and Party Priors

• Party Disparities: Under baseline conditions, models were largely even-handed. However, disparities widened under inconclusive conditions, particularly for specific parties (e.g., Die Linke in Germany, SP in the Netherlands).
• Role of Priors: Ablation studies suggest these disparities are driven by party priors. When party names were anonymized, disparities decreased. Models appeared to rely on internal knowledge to fill gaps for parties with strong training-data representation, while struggling with newer or less prominent parties (e.g., BSW, BBB).
• Sanitization: A recurring weakness was I4 Sanitization, where models (particularly GPT and Qwen) flattened the intensity and distinctiveness of political language, pulling rhetoric toward a neutral register. This effect was most pronounced for parties with rhetoric far from the model's implicit "neutral" baseline (e.g., AfD).

Significance and Claims

The paper claims that POLISTEMICS provides the first standardized, theory-grounded audit for LLMs as political information mediators. Its primary contributions are:

1. Shifting the Evaluation Paradigm: Moving beyond simple stance accuracy (reproduction) to evaluate the process of mediation (faithfulness, impartiality, calibration) under imperfect information.
2. Identifying Systemic Risks: Demonstrating that high aggregate scores can conceal localized failures where they matter most: when evidence is missing, vague, or contradictory. In these scenarios, models often present ambiguous evidence as settled, creating an "illusion of certainty" that undermines citizen epistemic agency.
3. Highlighting the Role of Priors: Showing that model behavior is not purely evidence-driven but is shaped by internal party priors and training data prominence, leading to uneven mediation across different political actors.

The authors conclude that while reliable mediation is achievable (as shown by GPT's abstention in absent conditions), no current model delivers it consistently across all information environments. They argue that democratic information mediation requires models that are not only politically impartial but also faithful to evidence and epistemically calibrated under uncertainty. The paper does not claim that LLMs are inherently unsafe, but rather that their current deployment in civic infrastructure without standardized auditing poses risks to the quality of public discourse and electoral decision-making.