💬 NLP

Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

This paper demonstrates that the Knobe effect, a moral bias in intentionality judgments, emerges in finetuned large language models but can be localized to specific layers and mitigated through targeted layer-patching interventions without retraining.

Original authors: Bianca Raimondi, Daniela Dalbagno, Maurizio Gabbrielli

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bianca Raimondi, Daniela Dalbagno, Maurizio Gabbrielli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to be a good judge. You don't just give it a rulebook; you show it thousands of stories about people making choices and ask, "Was that person being good or bad?" Over time, the robot learns patterns. But here's the tricky part: humans aren't perfect judges. We often have hidden biases. For example, if a person accidentally causes a bad result, we are much more likely to say, "They meant to do it!" than if they accidentally caused a good result. This is a famous psychological quirk called the "Knobe effect." It's like our brains have a built-in filter that makes us think bad accidents were actually on purpose, while good accidents were just luck.

Now, scientists are worried that if we teach robots to act like us, they might pick up these same weird filters. If a robot makes moral decisions for us, we need to know: does it think like a human, or is it just a giant calculator? This is where "mechanistic interpretability" comes in. Think of a giant robot brain as a massive city with millions of tiny workers (neurons) passing notes to each other. Mechanistic interpretability is like putting on X-ray glasses to see exactly which workers are passing which notes. Instead of just guessing why the robot gave a certain answer, we can peek inside the machine to see the specific gears turning. This matters because if we can find exactly where a bias lives inside the robot, maybe we can fix just that one gear without having to rebuild the whole city.

The researchers in this paper decided to put this idea to the test. They wanted to see if popular AI models (specifically Llama, Mistral, and Gemma) had picked up the "Knobe effect" and, if they had, whether they could find the exact spot in the AI's brain where this bias was hiding. They started by testing the AI models on the same moral stories used with human volunteers. They found that the "raw" AI models, which hadn't been taught to talk like humans yet, didn't really show this bias. Their answers were pretty balanced. However, once the researchers "finetuned" the models—training them to follow instructions and align with human preferences—the bias suddenly appeared. The AI started acting just like us: it was much more likely to say a bad outcome was intentional than a good one.

But the real magic happened when they looked inside the machine. Using a technique called "Layer-Patching," the researchers treated the AI like a set of stacked layers, like floors in a skyscraper. They suspected that the bias wasn't spread out everywhere, but was stuck in a specific set of floors. To test this, they took the "thoughts" (activations) from the unbiased, raw model and swapped them into the biased, finetuned model, one floor at a time. It was like taking a clean, unbiased blueprint and pasting it over a specific floor of a corrupted building to see if the whole structure became honest again.

The results were surprisingly precise. They discovered that the bias wasn't scattered randomly; it was localized in the middle-to-upper layers of the AI's brain. Even more exciting, they found that swapping just one specific layer from the unbiased version into the biased model was enough to almost completely erase the Knobe effect. For some models, the bias dropped from a strong 1.6 or 3.8 difference down to nearly zero (0.00 or 0.03). And the best part? They did this without retraining the model or changing its weights. It was a surgical fix. They also checked to make sure this "surgery" didn't break the robot's ability to do other things, like answering general questions, and found that the models stayed just as smart as before.

The paper suggests that these social biases aren't some mysterious, unfixable magic that emerges from the whole system. Instead, they are specific, localized patterns that get written into the AI's brain during the training process where we try to make it act like a human. By using these X-ray glasses and surgical swaps, the researchers showed that we can spot these biases and remove them with targeted interventions, keeping the AI helpful without the unwanted human quirks.

Technical Summary: Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

Problem Statement

Large Language Models (LLMs) have demonstrated the capacity to internalize human-like biases during the finetuning process, yet the specific mechanisms through which these biases manifest within the model architecture remain unclear. While behavioral studies confirm that LLMs can reproduce cognitive biases such as the Knobe effect—a phenomenon where individuals are more likely to judge an action as intentional when it leads to a negative side effect compared to a positive one, even with identical intentions—the internal representation of this bias is not well understood.

The paper addresses three primary research questions:

RQ1: Do LLMs exhibit the Knobe effect, and to what extent is this a consequence of finetuning versus pretraining?
RQ2: Are specific Transformer layers responsible for encoding this moral bias?
RQ3: Can targeted interventions, specifically Layer-Patching, mitigate or eliminate this bias without retraining the model?

Methodology

Models and Data

The study evaluated three open-weights LLM families: Llama-3 (specifically 8B and 1B variants), Mistral (7B), and Gemma (9B, 2B, and 27B variants). Both pretrained ( $M_p$ ) and instruction-tuned/finetuned ( $M_f$ ) versions were analyzed.

The dataset consisted of 80 moral scenarios adapted from Ngo et al. [27], derived from the original Knobe effect studies. Each scenario presented an agent performing an action with a known side effect (either positive or negative) but without caring about the outcome. The model was prompted to assign an intentionality score from 0 (non-intentional) to 10 (fully intentional).

To simulate human response variability and avoid deterministic outputs, the authors employed stochastic sampling with a fixed seed but randomized temperature ( $T \sim U[0.85, 1.15]$ ) across 283 generations per model per scenario.

Experimental Design

Behavioral Analysis (RQ1): The authors calculated the Knobe effect size ( $\Delta_{Knobe}$ ) as the difference between mean intentionality scores for negative ( $\mu_{neg}$ ) and positive ( $\mu_{pos}$ ) conditions. Statistical significance was assessed using a repeated-measures ANOVA.
Mechanistic Localization (RQ2): Using the TransformerLens library, the authors extracted residual stream activations from every layer. They computed the absolute difference in mean activations between negative and positive scenarios ( $\delta_l$ ) for each layer $l$ . By comparing these differences between pretrained and finetuned models, they identified layers where the bias was encoded.
Intervention via Layer-Patching (RQ3): The authors implemented a Layer-Patching algorithm. For a specific layer $l$ in the finetuned model, the residual activations were replaced with the corresponding activations from the pretrained model during inference. The forward pass continued using these patched activations. This process was repeated for every layer to determine which specific layer, when "unbiased," most effectively reduced the $\Delta_{Knobe}$ .

Key Results

1. Emergence of Bias via Finetuning (RQ1)

Pretrained Models: Showed negligible or no Knobe effect. The distributions of intentionality scores for negative and positive scenarios largely overlapped, indicating that the base models do not inherently possess this specific moral asymmetry.
Finetuned Models: Exhibited a significant and systematic Knobe effect. All finetuned models assigned higher intentionality scores to negative side effects than positive ones.
Magnitude: The effect was most pronounced in Gemma-2-9B ( $\Delta_{Knobe} = 3.83$ ), followed by Mistral-7B ($1.67$) and Llama-3.1-8B ($1.60$).
Comparison to Humans: The magnitude of the bias in finetuned models approached or exceeded human behavioral baselines (human $\Delta_{Knobe} \approx 3.15$ ), suggesting that alignment procedures can amplify human cognitive biases.

2. Localization of Bias (RQ2)

Layer Specificity: The bias was not diffusely distributed. Activation differences between moral conditions were minimal in pretrained models but became highly localized in mid-to-late Transformer layers in finetuned models.
Interpretation: This suggests that while pretraining captures statistical co-occurrences, the finetuning process (specifically instruction tuning) introduces a structured, evaluative asymmetry that is encoded in the semantically rich upper layers of the architecture.

3. Mitigation via Layer-Patching (RQ3)

Effectiveness: Replacing activations in a single critical layer with those from the pretrained model was sufficient to nearly eliminate the Knobe effect.
- Llama-3.1-8B: $\Delta_{Knobe}$ reduced from 1.60 to 0.00.
- Mistral-7B: $\Delta_{Knobe}$ reduced from 1.67 to 0.00.
- Gemma-2-9B: $\Delta_{Knobe}$ reduced from 3.83 to 0.03.
Preservation of Utility: The intervention did not require retraining. Evaluation on standard benchmarks (ARC-Easy, HellaSwag, MMLU, TruthfulQA) showed that the patched models maintained performance levels comparable to the finetuned baselines, with only minor regressions (~1%).

4. Ablation Study: Model Scale

The study examined smaller (1B, 2B) and larger (27B) models.

Consistency: The pattern held across all scales: pretrained models showed no significant bias, while finetuned models exhibited the Knobe effect.
Scaling: The absolute magnitude of the bias increased with model size (e.g., Gemma-2-27B showed a $\Delta_{Knobe}$ of 6.07), suggesting that larger models internalize and amplify these abstract moral constructs more strongly when aligned.

Significance and Claims

The paper claims that social biases in LLMs are not merely emergent, diffuse properties of the entire network but can be mechanistically localized to specific functional subcomponents (specific layers).

Key contributions include:

Causal Attribution: Demonstrating that the Knobe effect is primarily induced by the finetuning/alignment process rather than pretraining data alone.
Modularity of Bias: Showing that moral biases operate as modular computations within the Transformer architecture, specifically in mid-to-late layers.
Practical Mitigation: Proving that Layer-Patching offers a surgical, post-hoc method to remove specific moral biases without the computational cost of retraining and without catastrophic degradation of general capabilities.

The authors conclude that this work bridges interpretability and fairness, suggesting that targeted interventions can control the emergence of human-like cognitive biases in LLMs, offering a pathway for more responsible deployment of these models in ethical reasoning tasks.