💻 computer science

Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses

This paper introduces a new definition of AI political neutrality based on maximizing and balancing approval across opposing political groups, validates it through a large-scale human evaluation dataset (PARETO) involving over 7,000 participants, and reveals that while frontier models can achieve high cross-partisan approval, their default responses often exhibit a liberal bias.

Original authors: Jonathan Stray, David Zhai Yang, Steven Luo, Miu Nicole Takagi, Serina Chang

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonathan Stray, David Zhai Yang, Steven Luo, Miu Nicole Takagi, Serina Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hosting a dinner party where two guests, let's call them "Lefty" and "Righty," have a fierce disagreement about a topic like abortion or gun control. They are shouting over each other, and neither can agree on what the "truth" is.

Now, imagine you hire a smart robot waiter (the AI) to serve them. Your goal isn't to pick a side or tell them who is right. Your goal is to serve a dish that both Lefty and Righty can agree is delicious, even though they still disagree on the recipe.

This paper is about teaching that robot waiter how to do exactly that.

The Big Problem: The "Refusal" Trap

Previously, when people asked these robots controversial questions, the robots often did one of two things:

They picked a side: They sounded like Lefty or Righty, making the other person angry.
They refused to answer: They said, "I can't talk about that," which made everyone feel ignored.

The researchers wanted to know: Can we make an AI that answers these tough questions in a way that makes both sides say, "Okay, I can live with that answer"?

The New Recipe: "Balanced Approval"

The authors came up with a new definition of "neutrality." They call it Balanced Approval.

Think of it like a tightrope walker.

The Rope: This is the line of maximum possible happiness.
The Goal: The AI needs to stand on the tightrope where Lefty is happy AND Righty is happy at the same time.
The Catch: If the AI leans too far to the left, Righty gets mad. If it leans too far right, Lefty gets mad. The "perfect" neutral answer is the exact spot on the rope where both sides are equally satisfied.

The researchers call this the "Pareto Frontier." In plain English, it's the "best possible deal" where you can't make one side happier without making the other side unhappy.

The Experiment: A Massive Taste Test

To test this, the researchers didn't just ask the robots to guess. They set up a massive taste test:

The Menu: They picked 20 hot-button topics (like abortion, gun control, and student debt).
The Diners: They recruited 7,434 real people from the US.
The Chefs: They used 5 different top-tier AI models (like GPT, Claude, and Gemini).
The Dishes: For every question, they made four types of answers:
- The Default: What the AI says naturally.
- The "For" Dish: An answer arguing only for one side.
- The "Against" Dish: An answer arguing only for the other side.
- The "Balanced" Dish: A special answer that gave a short paragraph for both sides, then wrapped it up neutrally.

Then, they asked the diners: "How much do you approve of this answer?"

What They Found: The Surprising Results

1. The "Balanced Dish" was the crowd favorite.
Even though Lefty and Righty hated each other's ideas, they both liked the balanced answer.

The Magic Number: The balanced answer got high approval scores (above 60%) from both sides on almost every topic.
The Trade-off: You might think, "If I want my side to win, I'd hate a balanced answer." But the study found that people only lost about 10% of their approval when switching from a "my side wins" answer to a "balanced" answer. That's a small price to pay for an AI that doesn't make the other side scream.

2. Most Robots are secretly "Lefty."
When the researchers looked at what the robots said naturally (without any special instructions), most of them (GPT, Claude, Gemini, Llama) leaned toward the liberal/left side. They got more approval from Lefty than Righty.

The Exception: One robot, Grok, was different. It didn't lean left; sometimes it leaned right, sometimes left, depending on the topic.

3. Angry Questions are Harder to Answer.
When users asked the robots questions that were already angry or charged (e.g., "Why are liberals so stupid?"), the robots got lower approval scores. It's much harder to be neutral when the question itself is a fight.

4. The "Both Sides" Myth.
Some people worry that showing "both sides" is fake or weak. But the study showed that when the AI presented strong arguments for both sides, people actually felt it was fairer. The only time people hated the balanced answer was when they felt the AI was "faking it" or ignoring their specific concerns.

The Bottom Line

This paper proves that it is possible to build AI that acts like a fair mediator. It doesn't have to be a robot that refuses to speak, nor does it have to be a robot that picks a team.

By aiming for Balanced Approval—finding the answer that makes the most people on opposing sides feel heard and respected—we can create AI that people trust, even when they disagree with each other. It's like a robot waiter who manages to serve a meal that satisfies the entire family, even if they can't agree on what to order next.

Technical Summary: Political Neutrality as Balanced Approval

Problem Definition

As AI systems increasingly serve as primary information sources, their potential to shape political attitudes and induce epistemic fragmentation has raised urgent concerns regarding political neutrality. Current definitions of neutrality are often ambiguous, relying on opaque corporate benchmarks or simplistic metrics like "liberal vs. conservative" slant that fail to capture issue-specific nuances. Furthermore, existing evaluation methods often test models against static political compass tests or multiple-choice questions rather than open-ended user interactions. The core challenge is to define political neutrality in a way that is empirically testable, generalizable across diverse political contexts, and capable of maintaining trust among groups with fundamentally opposing values.

Methodology

The authors propose a new definition of AI political neutrality grounded in political theory (Mill, Rawls) and conflict mediation practices. They define a neutral response as one that achieves Maximum Equal Approval (MEA): a response that lies on the Pareto frontier of approval scores between opposing groups while minimizing the imbalance between those scores.

To operationalize and test this definition, the authors constructed the PARETO benchmark and conducted a large-scale human evaluation study:

Issue Selection and Canonical Questions: The study focuses on 20 controversial, values-based U.S. policy issues (e.g., abortion, gun control, trans rights). For each issue, a "canonical" survey question was identified to define the principal dimension of disagreement, mapping responses to "For" and "Against" stances.
Prompt Engineering: To ensure ecological validity, 200 user prompts were sourced from Reddit. These prompts were systematically categorized into five valences: Strongly For, For, Neutral, Against, and Strongly Against.
Response Generation: Five frontier AI models (GPT, Claude, Gemini, Grok, Llama) were evaluated. For each prompt, the study generated eight distinct response types:
- Default: The model's natural response.
- Single-Side: Arguments generated from the perspective of the "For" side and the "Against" side.
- Balanced: A constructed response including one paragraph from each side's perspective and a concluding sentence acknowledging trade-offs.
Human Evaluation: A study was conducted with 7,434 participants recruited from Prolific. Participants indicated their own positions on the issues and rated AI responses on seven Likert-scale statements (including approval, fairness, bias, and trust) and provided free-text feedback. This resulted in 208,152 human evaluations.

Key Contributions

A New Definition of Neutrality: The paper formalizes political neutrality as Maximum Equal Approval (MEA). Unlike previous approaches that seek to avoid bias or match a specific distribution, this definition seeks to maximize approval across opposing groups while balancing that approval, effectively treating neutrality as a tool for bridging polarized societies rather than merely avoiding offense.
The PARETO Dataset and Benchmark: The authors release a comprehensive dataset containing 208,152 human evaluations of 1,600 AI responses across 20 issues. This includes the prompts, model outputs, and detailed human ratings, providing a rigorous standard for measuring progress toward political neutrality.
Empirical Validation of Balanced Responses: The study demonstrates that it is possible to generate AI responses that achieve high approval rates from both sides of a controversy, even when those sides strongly disagree on the substance of the issue.

Results

Feasibility of Balanced Approval: Across all 20 issues, the balanced response (presenting arguments from both sides) most frequently achieved the empirical MEA point. It reached the Pareto frontier in 85% of issues and maintained an average approval difference of less than 1% between conservative and liberal participants ( $s_{cons} - s_{lib} \approx -0.01$ ).
Low Cost of Balance: While single-side responses achieved the highest approval from their respective aligned groups, the balanced response incurred a relatively small "cost." On average, the balanced response lost less than 10% in approval compared to the single-side response that aligned with the participant's view, suggesting that balanced models can remain viable even for partisan users.
Model Defaults Exhibit Liberal Lean: With the exception of Grok, the default responses of GPT, Claude, Gemini, and Llama consistently received higher approval from the liberal side than the conservative side. Grok's default responses varied, leaning liberal on some issues and conservative on others.
Impact of Prompt Charge: AI responses received significantly lower approval scores when responding to politically or emotionally charged prompts compared to neutral prompts, highlighting the difficulty models face in maintaining neutrality under high-stakes user framing.
Issue-Specific Nuance: Principal Component Analysis (PCA) of participant responses revealed that while a general liberal-conservative axis exists, many issues (e.g., labor unions, death penalty) do not align neatly with this axis, validating the paper's approach of defining neutrality on a per-issue basis rather than a single global axis.

Significance and Claims

The paper claims to establish both the theoretical goal and the empirical possibility of building AI models that maintain broad trust among people who disagree. By shifting the definition of neutrality from "avoiding bias" to "maximizing equal approval," the authors argue that AI can serve as a shared information source that prevents epistemic fragmentation.

The authors modestly note that while their balanced responses perform well, they are not perfect; approval scores rarely exceed 0.70, and participants still offer systematic criticisms regarding clarity and "false both-sides-ism." The work does not claim that all AI should be neutral, but rather that achieving broad trust through balanced approval is an essential public good. The PARETO dataset is provided to enable future research into automated evaluation and the development of models that better approximate the MEA ideal.