📈 economics

Model Selection as Policy Choice: Open-Weight LLMs Converge on Institutions but Diverge on Economic Policy

This study demonstrates that open-weight large language models are not interchangeable tools for policy analysis, as model selection—rather than sampling temperature—drives stable, structured divergences in economic policy preferences while maintaining consensus on institutional and environmental issues, thereby making model choice a critical governance decision.

Original authors: Tamas Olah, Marianna Abuczki, Nayna Babar, Tibor Tokes

Published 2026-07-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tamas Olah, Marianna Abuczki, Nayna Babar, Tibor Tokes

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Not All AI Chefs Cook the Same Dish

Imagine you are a government official trying to decide on a new economic policy. You ask an AI for advice. Most people assume that if they ask five different AI models the same question, they will get five slightly different versions of the same answer, like asking five different people to recite the same poem.

This paper argues that this assumption is wrong.

Instead of being interchangeable tools, the authors found that different AI models are like different chefs with distinct personalities and political philosophies. If you ask them the same question about the economy, they don't just give different words; they often choose completely different recipes based on their own hidden "tastes."

The Experiment: The AI Taste Test

The researchers set up a massive taste test.

The Pantry: They gathered 40 different "open-weight" AI models (these are models anyone can download and run on their own computers, unlike the secret ones owned by big tech companies).
The Menu: They gave every single model the exact same questionnaire. The questions covered tough topics like inflation, unemployment, housing, and international relations.
The Rules: For every question, the models had to rate four different possible answers (A, B, C, or D) on a scale of 0 to 100.
The Repetition: To make sure the results weren't just random glitches, they asked each model the same questions 100 times each.

What They Discovered

1. The "Personality" is Real, Not Random

When you flip a coin, the result is random. When you ask an AI a question, you might expect some randomness too. The researchers found that randomness (noise) was actually very small.

The Analogy: Imagine a choir. If the singers were just making random noises, the sound would be a mess. But here, the researchers found that the "voice" of each singer (each AI model) was very distinct and stable.
The Result: About 57% of the differences in answers came from which model was answering. Changing the "temperature" (a setting that makes the AI more or less random) barely changed the answer (less than 2%). This means the models have stable, built-in biases that don't go away just because you tweak a setting.

2. They Agree on "Who's in Charge," But Fight Over "How to Spend Money"

The researchers used a statistical map (like a GPS for ideas) to see where the models stood. They found two main "fault lines" where the models differed:

The "Trust in Institutions" Line (Where they mostly agree): Almost all models agreed that international organizations, rules, and formal institutions are important. They all generally liked the idea of following the rules and trusting experts.
The "Economic Philosophy" Line (Where they fight): This is where the models split up.
- Group A (The Market Lovers): These models prefer letting the free market decide, cutting regulations, and prioritizing growth.
- Group B (The Interventionists): These models prefer the government stepping in to fix inequality, control prices, and protect the environment, even if it slows down growth.

The "Stagflation" Test:
The researchers asked a classic hard question: "The country has high inflation AND high unemployment. What do we do?"

Some models said: "Crush inflation, even if people lose jobs."
Others said: "Save the jobs, even if prices go up."
Others said: "Make strict rules for both."
Others said: "Use a mix of weird, experimental controls."
The Shock: Every single one of these four answers was the "top choice" for at least some of the models. There was no single "AI answer."

3. The "Confidence" Trap

One of the most interesting findings was about confidence.

Some models were very shy and said, "I'm not sure" (low confidence scores).
Others were very loud and said, "I know the answer!" (high confidence scores).
The Twist: A model's confidence didn't depend on how hard the question was. It depended entirely on which model it was.
The Analogy: It's like a student who always gets a C but says "I'm 100% sure I'm right," while another student gets an A but says "I'm only 50% sure." If you are a policy maker, you might be tricked into trusting the loud, confident model even if it's just as biased as the quiet one.

4. Size Doesn't Mean "Better" Politics

You might think a bigger, smarter AI (with more "brain power") would have a more balanced or "correct" political view.

The Finding: Bigger models were slightly more consistent (less noisy), but they did not have a different political opinion than smaller models.
The Analogy: A giant, super-smart robot and a small, simple robot might both be "Market Lovers" or both be "Interventionists." You cannot guess a model's political bias just by looking at its size or brand name.

The Bottom Line: Choosing a Model is a Political Choice

The paper concludes that choosing which AI to use is not a technical detail; it is a policy decision.

The Metaphor: If you hire a consultant to write a report on the economy, you are hiring a person with a specific worldview. If you hire AI Model X, you are hiring a "Market Conservative." If you hire AI Model Y, you are hiring a "Government Interventionist."
The Danger: If you don't know this, you might think the AI is giving you a neutral, objective fact. But in reality, the AI is quietly injecting its own hidden values into your policy decisions.

The Takeaway:
Before using AI to help make government decisions, we need to treat it like a human expert: Check their background, know their biases, and don't just trust one person's opinion. We need to ask multiple different AIs and compare their answers to see the full picture, rather than letting one "default" model decide our future for us.

Technical Summary: Model Selection as Policy Choice

Problem Statement
Large Language Models (LLMs) are increasingly integrated into policy analysis, strategic planning, and decision support. However, applied work frequently treats individual models as interchangeable instruments, an assumption rarely tested empirically. The paper posits that this assumption is flawed: when tasks and prompts are held constant, LLMs may still differ systematically in how they frame economic and institutional dilemmas. These differences are not merely decoding noise but reflect stable orientations embedded in training data, architecture, and alignment objectives. Consequently, the choice of a specific model in policy contexts is not a neutral technical detail but a substantive governance decision that influences the narrative, normative emphases, and trade-offs foregrounded in policy proposals.

Methodology
The study employs a controlled, reproducible experimental design to map the behavioral regularities of 40 open-weight LLMs executed locally via Ollama.

Data Generation: A fixed multiple-choice questionnaire was administered to all models. The instrument spans macro-fiscal trade-offs, regulatory dilemmas, international institutions, perceptions of public figures, and political-economic interpretive frameworks.
Experimental Protocol:
- Replication: Each model answered the full questionnaire 100 times at two sampling temperatures ( $T=0.2$ and $T=0.8$ ) to distinguish stable orientations from stochastic noise.
- Scoring: For each item, models assigned a score (0–100) to four predefined answer frames (A–D).
- Normalization: Responses were centered relative to the within-question mean to remove level effects, focusing on relative preference ordering.
Analytical Pipeline:
1. Variance Decomposition: The total variance in option support was decomposed into between-model variance, temperature effects, and replicate noise.
2. Dimensionality Reduction: Weighted Principal Component Analysis (PCA) was applied to the pooled-by-model response profiles to identify latent axes of variation.
3. Clustering: Models were clustered in PCA-score space (using the first 16 components) to derive a discrete typology of model "types."
4. Validation: The data-driven PCA axes were validated against an independently hand-coded taxonomy of ideological axes (e.g., institutional trust, economic state–market) to ensure construct validity.
5. Topical Decomposition: The study localized where heterogeneity is concentrated by comparing model "leans" across specific topical domains.

Key Results

Signal vs. Noise: Model identity accounts for 57% of the variation in option support, while sampling temperature explains less than 2%. The remaining variance is replicate noise, which varies significantly by model (ranging from 3% to 79%). Within-model response profiles are highly stable across temperatures (median correlation $r=0.99$ ), confirming that differences are stable orientations rather than decoding artifacts.
Latent Dimensions: PCA recovered two dominant, interpretable axes:
1. Institutional-Trust Dimension (PC1): Contrasts institutional, coordinated, and technocratic framings (positive pole) against sovereignty-first, executive-centric, or ad hoc bargaining framings (negative pole). This axis correlates strongly ( $r=0.92$ ) with hand-coded institutional trust.
2. Economic–Environmental Dimension (PC2): Contrasts market-growth/efficiency-first framings against regulated-green/redistributive sustainability framings. This axis correlates with the economic state–market axis ( $r=0.48$ ) and climate ambition ( $|r|=0.60$ ).
Typology: Clustering yielded eight distinct model types, ranging from "literalist forecasting" and "pragmatic-interventionist" to "risk-and-sovereignty salient" and "polarized, state-leaning."
Topic-Specific Heterogeneity:
- Consensus: Models are near-unanimous on institutional trust, openness, and climate framings (e.g., 98% lean toward climate ambition and institutional trust).
- Divergence: Models split sharply on economic questions. On a single stagflation item, all four policy frames were the top choice for some subset of models. Only about one in five models leans "market" on the economic state–market axis, while the rest lean "state."
- Stability: Economic orientation is not a fixed global trait; a model's stance on one economic question weakly predicts its stance on another (Cronbach's $\alpha = 0.58$ ).
Confidence and Scale: Self-reported confidence is largely a fixed property of the model (55% of variance) rather than a reflection of question difficulty or contestation. Larger model scale predicts greater consistency (lower noise) but does not predict a specific political orientation or policy preference.

Key Contributions

Empirical Evidence of Non-Interchangeability: The study provides robust evidence that open-weight LLMs are not interchangeable in policy contexts. Their differences are structured, stable, and significant enough to alter policy recommendations and justifications.
Methodological Framework: It introduces a reproducible pipeline for auditing model behavior, separating persistent orientations from sampling noise, and mapping models onto interpretable ideological dimensions.
Typology of Model Behavior: It establishes a compact typology of eight model "types" based on their response profiles, offering a way to categorize models by their underlying policy logic rather than just performance benchmarks.
Reframing Model Selection: The paper argues that model selection is a governance-relevant design decision. It demonstrates that the "values" embedded in models are measurable and contestable, challenging the notion of the model as a neutral information retrieval system.

Significance and Claims
The paper claims that the societal stakes of model selection are central to the technical comparison of models. By demonstrating that models converge on institutional and cosmopolitan framings but diverge on economic and distributive questions, the authors argue that model choice conditions both the recommended policy frame and its justification.

The significance lies in shifting the perspective from "finding the correct model" to "making model orientations visible." The authors contend that treating model selection as a technical detail obscures the value choices inherent in policy automation. Instead, they propose that:

Model selection should be treated as a governance decision, akin to robustness checks in empirical economics.
Multi-model stress-testing is necessary to reveal the sensitivity of conclusions to latent framing tendencies.
Making the values embedded in models visible and contestable is a precondition for their responsible use in policy work.

The study concludes that while models are not neutral, their specific orientations are not arbitrary; they are structured, measurable, and consequential. Therefore, the choice of instrument must remain a deliberate, accountable act to ensure that society shapes these systems rather than being silently shaped by them.