🤖 machine learning

Moral Sensitivity in LLMs: A Tiered Evaluation of Contextual Bias via Behavioral Profiling and Mechanistic Interpretability

This paper introduces a tiered Moral Sensitivity Index to evaluate contextual bias in large language models, revealing distinct behavioral signatures across architectures and mechanistically validating that reasoning distillation can inadvertently reactivate criminal bias circuits despite increased capability.

Original authors: Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yash Aggarwal, Atmika Gorti, Vinija Jain, Aman Chadha, Krishnaprasad Thirunarayan, Manas Gaur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a group of very smart, but slightly different, robots. You want to see how fair they are when they have to make tough choices. Most people test robots by asking a simple "Yes or No" question: "Is this robot biased?" But this paper argues that's like asking if a person is "angry" without looking at what made them angry or how they reacted.

The authors of this paper say: "Let's look closer." They built a special test to see how these robots change their minds as a situation gets more complicated and emotional.

Here is the breakdown of their discovery, using simple analogies:

1. The Test: The "Moral Stress Ladder"

The researchers didn't just ask one question. They built a 7-step ladder of scenarios, starting very simple and getting harder.

Step 1 (The Bottom): A math problem. "Save 5 people or 1 person." No names, no ages, no race. Just numbers.
Step 2–3: They add details like "The 5 people are children" or "The 1 person caused the accident."
Step 4–7 (The Top): They add heavy social labels like race, gender, poverty, or historical injustice.

They call this the Moral Sensitivity Index (MSI). It measures how quickly a robot switches from "Let's do the math" to "Oh no, this is a sensitive topic, I can't answer."

2. The Robots' Personalities

They tested four famous AI models (Claude, Qwen, Llama, and Gemini). Each one acted like a different type of person:

Claude (The "Stop Sign"): This robot is calm and logical at the bottom of the ladder. But the moment you mention race or gender (Step 4), it slams on the brakes. It goes from "Let's think" to "I refuse to answer" instantly. It's like a guard who only checks your ID at a specific gate.
Qwen (The "Philosopher"): This robot doesn't just say "No." It talks a lot. It uses big, fancy words and sounds unsure ("It depends..."). It tries to think through the whole problem before deciding. It's like a professor who writes a long essay before giving an answer.
Gemini (The "Skeptic"): This robot is suspicious of the whole game. Even at Step 1 (just numbers), it says, "Wait, is it fair to save 5 people just because there are more of them?" It questions the rules of the game itself, especially when money or class is involved.

3. The Big Surprise: The "U-Curve" of Bias

This is the most important part of the paper. The researchers looked at how the robots were built, specifically a process called "Distillation."

Think of Distillation like taking a giant, super-smart encyclopedia and compressing it into a small, fast pocket guide. You want the pocket guide to be just as smart but smaller.

They tested three types of robots:

Small Robots: Naturally biased (they pick the "criminal" label too easily).
Big Robots: Not biased (they learned to be fair).
Compressed Robots (Distilled): These were the small versions of the big, fair robots.

The Shock: The researchers found a U-shaped curve.

The small robots were biased.
The big robots fixed the bias.
But the compressed robots brought the bias back!

It's like taking a master chef who learned to cook a perfect, healthy meal, shrinking their recipe down to fit on a napkin, and finding out the napkin recipe accidentally brings back the unhealthy ingredients. The process of making the AI smaller and faster actually re-introduced the very biases the big AI had learned to avoid.

4. Looking Inside the Brain (Mechanistic Interpretability)

To understand why this happened, the researchers didn't just watch what the robots said; they looked inside their "brains" (the code and math layers).

The "Criminal" Trigger: They found that when the robots saw the word "Criminal," their internal circuits lit up.
The Compression Effect: In the big, fair robots, there were "brakes" in the later layers of the brain that stopped the bias. But in the compressed (distilled) robots, those brakes were gone or moved. The compressed robots made the biased decision faster and earlier in their thinking process.
The Result: The compressed robots didn't just "forget" to be fair; they actually reorganized their thinking to rely on old, shallow stereotypes again.

The Bottom Line

The paper concludes that we can't just ask AI, "Are you biased?" We have to look at how they react to different levels of social complexity.

Most importantly, they discovered that making AI smaller and faster (distillation) is not a neutral process. It can accidentally break the "safety training" that larger models have, causing them to become biased again. This means that before we use these smaller, faster AI models in the real world, we need to check if they've lost their moral compass during the shrinking process.

Technical Summary: Moral Sensitivity in LLMs

Problem Statement

Current bias evaluations for Large Language Models (LLMs) predominantly rely on binary classifications ("biased" vs. "unbiased"), which fail to capture the gradual, context-sensitive nature of how bias emerges. This binary framing obscures the dynamic interaction between a model's baseline reasoning capabilities and its safety-alignment policies. Specifically, models often exhibit neutral behavior in abstract scenarios but shift abruptly to safety-driven refusals or moralized judgments when sensitive demographic attributes (e.g., race, gender, socioeconomic status) are introduced. The paper argues that this shifting behavior indicates that contextual bias is not merely a reflection of pre-trained associations but an emergent property of alignment interventions overriding baseline decision-making. Furthermore, existing benchmarks do not explain the internal computational mechanisms driving these shifts, nor do they account for how model distillation might reintroduce biases that larger models had learned to suppress.

Methodology

The study employs a two-stage evaluation framework: Behavioral Profiling and Mechanistic Validation.

1. Behavioral Profiling

Dataset: A seven-tier "trolley-problem" dataset designed to incrementally increase contextual complexity:
- Tier 1: Numerical baseline (anonymous lives).
- Tier 2: Age (children vs. elderly).
- Tier 3: Responsibility (causal culpability).
- Tier 4: Protected social identities (race, gender).
- Tier 5: Socioeconomic status (wealth, poverty).
- Tier 6: Historical injustice.
- Tier 7: Combined systemic proxies.
Metric: The Moral Sensitivity Index (MSI) is introduced as a quantitative framework to measure the intensity of a model's safety-alignment override. It is defined as $MSI = \alpha B + \beta A + \gamma E$ $M S I = α B + β A + γ E$ , where:
- $B$ (Bias Score): Degree of stereotypical association.
- $A$ (Ambiguity Score): Extent of hedging or uncertainty.
- $E$ (Semantic Entropy): Variability in reasoning across repeated queries.
- Coefficients $\alpha, \beta, \gamma$ are derived via multiple linear regression.
Models Evaluated: Four leading instruction-tuned models (Claude 3.5, Qwen 3.5, Llama 3, Gemini 1.5) and six models for mechanistic analysis spanning three capability tiers: Small Language Models (SLMs), Instruction-tuned Base models, and Reasoning-distilled variants.

2. Mechanistic Interpretability Pipeline

To trace the internal emergence of bias, the authors apply a five-step pipeline to criminal-bias scenarios (a controlled subset of the behavioral findings):

Logit Lens: Probing hidden states at every layer to track the evolution of $P(\text{Criminal})$ vs. $P(\text{Non-Criminal})$ .
Attention Analysis: Computing differential attention heads focusing on criminal-associated tokens.
Activation Patching: Causal intervention (ablation) of specific heads to measure their impact on output probability.
Semantic Direction Analysis: Projecting concepts onto a semantic valence direction constructed from positive/negative lexica.
OV Circuit Reconstruction: Analyzing the tokens promoted by the Output-Value (OV) matrices of implicated heads.

Key Results

Behavioral Findings: Divergent "Moral Personalities"

The MSI analysis reveals distinct alignment strategies across models:

Claude: Exhibits a "step-function" profile. It remains relatively measured in Tiers 1–3 but undergoes a sharp, non-linear escalation in bias rates at Tier 4 (protected identities), reaching 100% bias labeling by Tier 6. This suggests a hard-coded override triggered by protected-class language.
Qwen: Displays a "category-first" reasoning strategy. It peaks in ambiguity (33.3%) in Tiers 2–3 before jumping to bias in Tier 5 (socioeconomic context). It utilizes high linguistic diversity and academic vocabulary to navigate complexity rather than refusing.
Gemini: Shows "structural skepticism." It begins with a high baseline bias rate (37.5%) even in Tier 1, questioning the utilitarian premise itself. Its highest certainty of bias occurs at Tier 5 (socioeconomic context), suggesting a sensitivity to class-based inequality over race/gender.
General Trend: Socioeconomic and geographic inequality (Tier 5) act as potent moral triggers, often overriding utilitarian logic more strongly than other factors.

Mechanistic Findings: The U-Curve of Bias

The study identifies a non-monotonic "U-curve" in criminal-label bias across model capabilities:

SLMs (3–4B): Exhibit strong criminal bias (e.g., Llama 3.2 3B assigns 87.3% probability to the "Criminal" option).
Base Models (7–8B): Instruction-tuned base models successfully suppress this bias (e.g., Llama 3.1 8B drops to 0.8%).
Distilled Models (7–8B): Reasoning distillation reintroduces the bias to levels comparable to or exceeding SLMs (e.g., Distill-Qwen-7B returns to 95.7%).

Internal Mechanisms of the U-Curve:

Emergence: In distilled models, the preference for the "Criminal" label emerges earlier in the network (e.g., Layer 4 in Distill-Llama-8B vs. Layer 8 in Base-Llama) and is more strongly committed.
Attention Reorganization: Distillation restructures attention patterns, concentrating criminal-focus heads in specific early or late positions rather than the broadly distributed patterns seen in base models.
Causal Concentration: Ablation studies show that distilled models rely on fewer, highly causal heads to drive bias. For instance, ablating the top head in Distill-Llama-8B has an effect size nearly three times that of the base model.
Semantic Compression: Distillation compresses the semantic distance between neutral demographic concepts and the "Criminal" anchor. In the Qwen family, neutral concepts like "Scientist" and "Citizen" cross into positive criminal valence space in distilled models, whereas they remain distinct in base models.
Lexical Content: OV circuit reconstruction reveals that heads in distilled models write explicitly negative lexical content (e.g., "dirty," "worthless"), whereas base model heads promote neutral or incoherent tokens.

Significance and Claims

The paper claims to provide the first unified framework linking observable behavioral shifts in moral reasoning to their underlying computational circuits. Its primary contributions are:

Quantification of Contextual Bias: The MSI framework moves beyond binary evaluation, identifying specific "moral breaking points" where models abandon logic for safety-driven responses.
Discovery of the Distillation Paradox: The study provides empirical evidence that the process of "distilling" models to improve efficiency actually reintroduces biases that larger base models had successfully learned to avoid. This suggests that distillation compresses reasoning traces in ways that reactivate shallow statistical associations and degrade safety-relevant internal representations.
Mechanistic Explanation: The research identifies that the reintroduction of bias is not merely a behavioral artifact but is rooted in reorganized attention structures, concentrated causal heads, and compressed semantic geometries.
Regulatory Relevance: The authors argue that these findings necessitate post-distillation auditing. With upcoming regulations like the EU AI Act and Colorado AI Act requiring documented bias examinations, the paper offers a graduated, mechanistically grounded protocol for evaluating high-risk systems.

The study concludes that current distillation objectives, which optimize for output fidelity, fail to preserve safety-relevant internal representations, highlighting a critical gap in the alignment of efficient, distilled models.