Graph-Augmented LLMs for Swiss MP Ideology Prediction

Technical Summary: Graph-Augmented LLMs for Swiss MP Ideology Prediction

Problem Statement
Approximating the ideological positions of Members of Parliament (MPs) is a fundamental task in political science, traditionally achieved by analyzing legislative behavior, speeches, and voting records. While Large Language Models (LLMs) have demonstrated promise in estimating these stances from textual sources, existing approaches suffer from two primary limitations:

1. Independence Assumption: They typically treat MPs as independent text generators, ignoring the rich relational structure of parliamentary systems (e.g., co-sponsorship networks, committee memberships, party blocs).
2. Data and Dependency Constraints: LLM-based methods often struggle in low-data settings or when long-term dependencies across multiple parliamentary sessions are required.

Current methods fail to leverage the "wider and more informative picture" provided by the complex interplay of actors and elements within a parliament, despite evidence that relational information can substantively enrich representations.

Methodology: PG-RAG Framework
The authors propose PG-RAG (Political Graph Retrieval-Augmented Generation), a framework designed to integrate graph-structured knowledge into LLM prediction pipelines. The methodology proceeds in three stages:

1. Political Knowledge Graph (KG) Construction:
   The system utilizes a KG built from the Bulletin of the Swiss Parliament, encoding the policy-making process. The schema includes entities such as Pursuit (legislative proposals) and Speech, linked by temporal dependencies and enriched with metadata regarding MPs, parties, and committees.

2. Subgraph Extraction:
   For a target MP, the system extracts specific subgraphs from the KG based on three paradigms, ordered by complexity:

   • Speech-centric (SP): Retrieves all speeches linked to the MP. This serves as a textual baseline.
   • MP-centric (MP): Extracts a 1-hop subgraph capturing institutional relationships, including party affiliation, parliamentary group, committee memberships, and geographic representation.
   • Pursuit-centric (PR): Extracts a 2-hop subgraph focusing on legislative initiatives sponsored or co-sponsored by the MP.

3. Context Encoding and Generation:
   The extracted subgraphs are encoded into the LLM context using two strategies:

   • Summarization (S): The subgraph triplets are serialized into natural language statements and summarized by an LLM (GPT-5) to create a structured textual representation of the MP's political context.
   • Raw-Graph (R): The subgraph is provided directly as a raw set of nodes and relations (JSON format from Neo4j), preserving explicit relationship types and attributes without summarization.

The framework evaluates these combinations (e.g., MP-S, MP-R, SP-R) against standard baselines to predict ideology scores (mapped to a left-right scale) based on the MP's identity, party, and group information.

Experimental Setup

• Dataset: Data from the Swiss National Council (lower chamber) covering the 50th legislative period (2015–2019), comprising 225 unique MPs.
• Ground Truth: Ideology scores derived from a random sample of 1,000 recorded votes, scaled using a two-parameter Item Response Theory (IRT) model.
• Baselines:
  • Naive: Global Mean, Party Mean, Party Bloc Mean.
  • LLMs: GPT-5, Qwen3-8B, Qwen3-32B, and Apertus-8B (Swiss LLM), tested in zero-shot and few-shot settings.
• Metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Spearman's Rank Correlation (RC).

Key Results

• Performance Improvement: Incorporating graph-derived knowledge significantly improves prediction accuracy. In the zero-shot setting, the MP-centric subgraph with summarization (MP-S) reduced MAE from 0.75 (GPT-5 baseline) to 0.72.
• Error Reduction: The graph-augmented approach notably reduced MSE (from 1.06 to 0.74 for MP-R in zero-shot), demonstrating the ability to correct predictions that deviate substantially from ground truth.
• Model Size Sensitivity: The benefits of graph augmentation were more pronounced for smaller-scale models. Qwen-8B showed percentage RMSE improvements ranging from 16.4% to 22.4% across RAG settings, compared to 8.7% to 16.5% for GPT-5. This suggests smaller models rely more heavily on external structured guidance.
• Subgraph Effectiveness: The MP-centric subgraph yielded the best results across all scenarios. The Pursuit-centric subgraph performed worse, likely due to its larger size and complexity, which the LLMs struggled to leverage effectively.
• Party-Specific Challenges: While the model improved overall coherence, it struggled specifically with Social Democrats (SP). The model tended to predict SP members as more right-leaning than their voting behavior indicated and failed to capture within-party diversity as effectively as it did for other blocs.

Significance and Claims
The paper claims that integrating domain-specific relational information via a graph-augmented RAG pipeline enhances the modeling of political behavior. Specifically:

1. Value of Relational Data: The results highlight that encoding information about different entities and relations (beyond raw text) improves prediction performance, particularly for smaller models.
2. Beyond Text: The approach demonstrates that capturing inter-MP relationships and institutional contexts allows LLMs to better understand the complex web defining legislative behavior.
3. Future Directions: The authors modestly note that while the method improves general performance, the persistent difficulty in predicting left-leaning parties (specifically Social Democrats) suggests a need for further investigation into whether this stems from heterogeneous MP actions or inherent biases in LLMs regarding political rhetoric.

The work establishes a foundation for future research into extending these methods to broader datasets and refining the modeling of specific ideological nuances within parliamentary systems.