💬 NLP

When Names Change Verdicts: Intervention Consistency Reveals Systematic Bias in LLM Decision-Making

This paper introduces ICE-Guard, a framework that reveals LLMs exhibit significant and domain-specific biases toward authority and framing—often exceeding demographic bias—and demonstrates that structured decomposition combined with an iterative detect-diagnose-mitigate-verify loop can substantially reduce these inconsistencies in high-stakes decision-making.

Original authors: Abhinaba Basu, Pavan Chakraborty

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhinaba Basu, Pavan Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot judge named "AI." You put it in charge of making life-changing decisions: who gets a job, who gets a loan, who goes to jail, or which medical treatment to prioritize. You expect this robot to be perfectly fair, looking only at the facts and ignoring everything else.

But what if the robot is secretly judging people based on their name, the university they went to, or how the story is told?

This paper, titled "When Names Change Verdicts," introduces a new way to test these robots and fix their hidden biases. Here is the breakdown in simple terms:

1. The Problem: The "Magic 8-Ball" Effect

The researchers realized that while we worry a lot about robots being racist or sexist (demographic bias), we are ignoring two other huge problems:

The "Name Drop" Bias (Authority Bias): The robot cares too much about who says something. If a study comes from "Harvard," the robot loves it. If the exact same study comes from a "Community College," the robot hates it.
The "Spin Doctor" Bias (Framing Bias): The robot cares too much about how something is said. If you say "95% of people survive," the robot is happy. If you say "5% of people die" (which is the exact same math), the robot gets scared and makes a different decision.

The Analogy: Imagine a hiring manager who ignores your resume entirely. Instead, they hire you because your name sounds "fancy," or they reject you because the letter of recommendation was written on "fancy paper" instead of "plain paper." That's what these AI models are doing.

2. The Solution: ICE-GUARD (The "Spot the Difference" Game)

The authors created a framework called ICE-GUARD. Think of it as a rigorous "Spot the Difference" game for AI.

How it works: They feed the AI two almost identical stories.
- Story A: A job applicant named "James Smith" with a degree from "State University."
- Story B: The exact same applicant, but named "Jamal Washington" with a degree from "Harvard."
The Test: If the AI gives different answers to these two stories, it has failed the test. It means the AI is relying on "spurious features" (irrelevant details) instead of the actual facts.

They tested this on 3,000 different scenarios across 10 serious fields (like finance, law, and medicine) using 11 different AI models.

3. The Shocking Results

The researchers found some surprising things:

The "Name" isn't the biggest problem: Everyone thought the AI would be most biased by race or gender names. But actually, the AI was 2 to 3 times more likely to change its mind based on authority (where the info came from) or framing (how the info was phrased).
Finance is the worst offender: In the world of money, the AI was incredibly fickle. About 1 in 5 investment recommendations changed just because the analyst's name was different or the numbers were phrased differently.
Some AIs are better than others: The top-tier models (like GLM-5 and Claude Opus) were very fair, making mistakes less than 1.3% of the time. But others were making mistakes up to 7.7% of the time.

4. The Fix: The "Chef and the Recipe" (Structured Decomposition)

The paper doesn't just point out the problem; it offers a clever fix called Structured Decomposition.

The Old Way (Free-Form): You ask the AI, "Should we hire this person?" The AI writes a paragraph, uses its "gut feeling," and decides. This is where the bias hides.

The New Way (Structured): You split the job into two steps:

The Chef (The AI): The AI is only allowed to do one thing: extract the facts into a strict list (e.g., "Years of experience: 5", "Salary: $50k"). It cannot make a decision yet.
The Recipe (The Computer Code): A simple, unchangeable computer program (a "rubric") takes that list and makes the final decision based on hard rules.

Why this works:

The AI (the Chef) can't sneak in bias because it's only allowed to list facts.
The Code (the Recipe) is a robot that doesn't have feelings, so it doesn't care if the name is "James" or "Jamal." It just follows the math.

The Result: For most models, this fix reduced bias by nearly 50% to 100%. In some cases, it made the AI perfectly fair.

5. The Big Takeaway

We need to stop worrying only about whether AI is racist or sexist. We also need to worry about whether it is easily tricked by fancy names or clever wording.

The paper suggests that to make AI truly fair in high-stakes situations (like hiring or lending), we shouldn't just ask the AI to "be nice." Instead, we should force it to list the facts first and let a simple, unfeeling rulebook make the final call. This stops the AI from being a "prejudiced judge" and turns it into a "fair fact-checker."

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making scenarios (e.g., hiring, criminal sentencing, lending, medical triage). While existing bias benchmarks (e.g., StereoSet, BBQ) focus on stereotyping in cloze tasks or generation, they fail to measure whether these biases actually alter final decisions.

The paper identifies a critical gap: current research disproportionately focuses on demographic bias (names, race, gender), often overlooking other forms of spurious feature reliance. The authors argue that LLMs may exhibit significant sensitivity to:

Authority/Prestige Bias: Decisions changing based on the source of credentials (e.g., "Harvard study" vs. "community college study").
Framing Bias: Decisions changing based on semantic phrasing (e.g., "95% survival" vs. "5% mortality").

The core problem is that models might pass demographic fairness tests while still making inconsistent, biased decisions when irrelevant features like authority or framing are swapped.

2. Methodology: The ICE-GUARD Framework

The authors introduce ICE-GUARD (Intervention Consistency Evaluation for Guardrails), a framework adapted from explanation faithfulness literature to detect bias in decision-making.

A. Core Principle: Intervention Consistency

The framework relies on the Invariance Principle: If a feature is irrelevant to a decision, swapping it should not change the model's output.

Input: A decision vignette $x$ with relevant features $S$ (facts, evidence) and irrelevant features $Z$ (names, credentials, phrasing).
Intervention: An operator $O_z$ swaps the irrelevant features (e.g., changing a name from "James Smith" to "Jamal Washington").
Metric: The Flip Rate (FR) measures how often $f(x) \neq f(O_z(x))$ . A high flip rate indicates systematic spurious reliance.

B. Three Bias Intervention Types

The framework tests three specific categories of interventions across 10 high-stakes domains:

Demographic Intervention ( $O_{dem}$ ): Swaps identity markers (names, race, gender).
Authority Intervention ( $O_{auth}$ ): Swaps prestige markers (institutional affiliations, journal rankings, expert titles).
Framing Intervention ( $O_{frame}$ ): Swaps semantically equivalent but valence-different phrasing (e.g., gain vs. loss framing).

C. Benchmark Construction

Scale: 3,000 decision vignettes across 10 domains (Criminal Justice, Hiring, Healthcare, Lending, Education, Insurance, Legal, Content Moderation, Finance, Customer Service).
Validation: Vignettes were validated against real-world systems (e.g., COMPAS, Resume Matcher) showing $\ge$ 80% feature overlap. A subset was derived from real ProPublica COMPAS data to ensure ecological validity.
Models Evaluated: 11 instruction-tuned LLMs from 8 families (including GPT-5.2, Claude Opus/Sonnet 4.6, Gemini 2.5/3, Kimi-K2.5, etc.).

D. Mitigation Strategy: Structured Decomposition

To address bias, the authors propose an architectural shift:

Extract: The LLM extracts only relevant features into a structured JSON schema, filtering out irrelevant noise.
Decide: A deterministic Python rubric (rule-based system) makes the final decision based only on the extracted features.
This separates the "reasoning" (LLM) from the "logic" (deterministic code), theoretically eliminating bias in the decision step.

3. Key Results

A. Authority and Framing Bias Dominate Demographics

Contrary to the community's focus on demographic bias, the study found:

Authority Bias: Mean flip rate of 5.8%.
Framing Bias: Mean flip rate of 5.0%.
Demographic Bias: Mean flip rate of 2.2%.
Insight: Authority and framing biases are 2.5x more prevalent than demographic bias. Modern RLHF alignment appears to have successfully mitigated overt demographic sensitivity, leaving cognitive biases (authority/framing) as the primary residual risk.

B. Domain-Specific Vulnerability

Bias is not uniform across domains:

Finance: The most vulnerable domain, exhibiting 22.6% authority bias and 20.4% framing bias. This implies roughly 1 in 5 investment recommendations could change based solely on the analyst's title or the phrasing of the report.
Hiring: Shows significant authority bias (10.9%).
Criminal Justice, Education, Customer Service: Show near-zero flip rates, suggesting these domains have received targeted alignment attention.
Demographic Baseline: All domains showed demographic flip rates within the 5% noise baseline, indicating demographic guardrails are broadly effective.

C. Model Performance

Top Performers: GLM-5, Claude Opus 4.6, and GPT-OSS-120B achieved overall flip rates $\le$ 1.3% (approaching the noise floor).
Worst Performers: Qwen3-32B (7.7%) and Claude Sonnet 4.5 (6.0%).
Pattern: For almost all models, the highest bias dimension was Authority or Framing, never Demographics.

D. Effectiveness of Structured Decomposition

Applying the "Extract then Decide" architecture significantly reduced bias:

Median Reduction: 49% across 9 models.
Best Case: Kimi-K2.5 reduced flip rates from 5.5% to 0.0% (100% reduction).
Failure Cases: GPT-5.2 saw an increase in bias (3.7% $\to$ 6.2%) due to inconsistent feature extraction, highlighting that structured decomposition relies on the LLM's ability to reliably extract data.
Iterative Loop: An ICE-guided "detect $\to$ diagnose $\to$ mitigate $\to$ verify" loop achieved a cumulative 78% bias reduction for Claude Sonnet 4.5 by patching extraction prompts.

4. Significance and Contributions

Paradigm Shift in Bias Detection: The paper challenges the narrow focus on demographic bias, demonstrating that authority and framing biases are more pervasive and consequential in high-stakes decision-making.
New Benchmark (ICE-GUARD): Provides a comprehensive, publicly available benchmark of 3,000 vignettes covering 10 domains and 3 bias types, moving beyond simple association tests to decision flip rates.
Architectural Mitigation: Proposes Structured Decomposition as a practical, effective method to reduce bias, showing that separating feature extraction from decision logic can eliminate spurious reliance for most models.
Real-World Relevance: The validation against real COMPAS data suggests that synthetic benchmarks may actually provide conservative estimates of real-world bias, as real-world flip rates were higher than synthetic ones.
Policy Implications: The findings suggest that current alignment strategies (RLHF) are effective for demographic fairness but insufficient for cognitive biases. Future safety protocols must explicitly target authority and framing susceptibility, particularly in domains like finance and lending.

Conclusion

The paper concludes that LLMs are not merely "stereotyping" but are making systematically inconsistent decisions based on irrelevant cues like prestige and phrasing. The ICE-GUARD framework offers a rigorous method to detect these flaws, and structured decomposition offers a viable path toward more robust, fair, and reliable AI decision systems.