Technical Summary: Diagnosing Korean-Language LLM Political Bias via Census-Grounded Agent Simulation

1. Problem Statement

Large Language Models (LLMs) are increasingly utilized to simulate human political behavior, yet they exhibit systematic political biases that compromise their utility in electoral forecasting. While prior research (e.g., Feng et al. [4]) has documented a liberal-left lean in English-language LLMs using intrinsic probing, it remains unclear whether these biases generalize to non-English electorates with distinct political structures, or what specific mechanisms drive the divergence between simulation and reality.

Korean politics presents a unique challenge for simulation due to its persistent regionalism (conservative strongholds in Gyeongsang vs. progressive in Jeolla), a large moderate bloc (approx. 48% of voters), frequent party renaming, and generational cleavages. Existing census-grounded simulation frameworks have been developed primarily for the US, leaving a gap in understanding how LLMs handle non-English political dynamics. This paper addresses two core questions: (i) Do Korean-language LLMs exhibit the same directional bias patterns as English-language models? and (ii) What specific failure mechanisms drive the gap between simulation and actual election outcomes?

2. Methodology: The Dynamo-K Framework

The authors introduce Dynamo-K, a census-grounded LLM agent-simulation framework designed as a diagnostic instrument. The pipeline operates through six layers:

1. Census Grounding: Agents are synthesized via weighted proportional sampling from the MDIS 2020 Census 2% microdata (794,342 records), covering 5,790 non-empty demographic cells across age, sex, province, education, and marital status.
2. Belief Seeding: Political orientations are assigned based on conditional probabilities derived from the Korean General Social Survey (KGSS) cumulative data (2003–2023), mapping demographics to progressive, moderate, or conservative orientations.
3. Gallup Calibration: To correct for KGSS's methodological tendency to push ambiguous respondents toward poles, a calibration adapter redistributes agent orientations to match Gallup Korea's long-term benchmark (26% conservative, 48% moderate, 26% progressive). This involves reassigning "borderline" agents and nudging their belief scores.
4. LLM Simulation (ORC): Each agent receives an election scenario and reasons through a vote decision using an Observation–Reasoning–Conclusion (ORC) pipeline. The system prompt explicitly encodes the agent's persona, including demographics and a behavioral description designed to mitigate bias (e.g., instructing moderate voters to reason as a profiled voter, not an AI, and emphasizing equal probability of voting for either camp).
5. Aggregation: Results are aggregated at national and sub-national (17 provinces) levels, with bootstrap confidence intervals generated from 1,000 resamples.
6. Evaluation: Performance is measured against six certified Korean elections (2017–2025) using Mean Absolute Error (MAE), winner prediction accuracy, and Wasserstein similarity.

The study evaluates four independently trained models: Qwen3-30B-A3B, EXAONE-4.0-32B, Llama-3.1-8B, and DeepSeek-R1-8B.

3. Key Contributions and Findings

A. Systematic Failure Modes

The authors identify three reproducible failure modes and one model-dependent finding:

1. Progressive Bias in Moderate Agents: Without mitigation, moderate agents vote for progressive candidates 97% of the time. Even with demographic conditioning alone, this remains at 82%. Explicit prompt mitigation (balanced descriptions, anti-AI perspective instructions) reduces this to 59%, significantly lowering the overall MAE from 36.8%p to 7.1%p. This confirms a progressive lean consistent with English-language findings but highlights its catastrophic impact on simulating a large moderate bloc.
2. Third-Party Salience Collapse: LLMs struggle to distribute votes among viable third-party candidates. In the 2017 election, the simulation collapsed the field, predicting the eventual winner (Moon Jae-in) with 71.3% share while under-predicting the centrist candidate (Ahn Cheol-soo) at 0.9% (actual: 21.4%).
   • Mechanism Decomposition: The collapse is driven by two distinct mechanisms: Salience Failure (models like EXAONE fail to mention the candidate in reasoning text) and Decision Bias (models like Qwen3 mention the candidate but fail to vote for them).
   • Reframing Solution: Scenario reframing (reordering candidates, expanding pledges) alone recovered Ahn's predicted share from 0.9% to 18.8%, reducing MAE by 62% without architectural changes or retraining.
3. Regional Polarization Collapse: What appears as a uniform conservative bias in general elections is actually a bidirectional under-prediction of both conservative and progressive strongholds. The error is concentrated in swing regions where moderate voters dominate.
4. Opposite-Valence Model Bias: On the 2025 presidential race, Qwen3 exhibited a +11.8%p progressive bias, while EXAONE exhibited a −17.2%p conservative bias. This demonstrates that directional bias is highly model-dependent, even among similarly sized models.

B. Calibration and Validation

• OSLR Adapter: To address model-dependent bias, the authors propose the Orientation × Sido Logistic Reweighting (OSLR) adapter. This learned reweighting mechanism uses no candidate names at train or test time, relying only on stable features (party orientation, region).
• Cold Held-Out Test: The OSLR adapter was evaluated on the 2022 local election (held out entirely). While it did not perfectly predict the winner due to the split in model predictions, it reduced the national MAE from 2.29%p (simple ensemble) to 1.57%p, demonstrating that the learned reweighting structure transfers to structurally different elections.
• Performance: The system correctly predicted the winner in all three presidential elections (3/3), with a best-case MAE of 2.1%p on the 0.73%p-margin 2022 race. However, general election winner prediction was 0/2 correct using a party-proxy abstraction, though a constituency-level proof-of-concept for 2024 flipped the result to correct.

C. Cost and Scalability

The pipeline operates at approximately $0.25 per 5,000-agent run, roughly two orders of magnitude cheaper than professional polling ($50,000+). The authors frame this not as a polling replacement but as a tool for "diagnosis at scale," enabling rapid scenario exploration (e.g., "what-if" candidate dropouts) and sensitivity analysis that is economically infeasible with traditional methods.

4. Significance and Claims

The paper claims that Dynamo-K serves as a diagnostic instrument that surfaces specific, reproducible failure modes in LLM political behavior which are otherwise hidden behind aggregate accuracy summaries.

• Diagnostic Validity: The framework successfully localizes failures to specific mechanisms (e.g., distinguishing between salience failure and decision bias), allowing for targeted interventions like prompt reframing or reweighting.
• Non-English Generalization: The study provides the first empirical characterization of how the "liberal lean" documented for English LLMs surfaces in Korean voter-decision simulation, confirming that the bias is present but interacts with local political structures (e.g., the large moderate bloc) in unique ways.
• Limitations: The authors are modest about their claims, noting that the sample size (six elections) is small for robust statistical conclusions. They acknowledge limitations including census temporal mismatch (using 2020 data for 2017/2025 elections), static agents (no dynamic belief updating), and the lack of social interaction effects. They explicitly state that the system is not a replacement for polling but a complementary tool for rapid triage and mechanism diagnosis.

The work concludes that while LLMs can achieve competitive accuracy in winner prediction for presidential elections, significant challenges remain in modeling general elections and third-party dynamics, necessitating constituency-level modeling and further cross-lingual research.