🤖 AI

Building Interpretable Models for Moral Decision-Making

This paper presents a compact, 2-layer transformer model that achieves 77% accuracy on Moral Machine data while utilizing interpretability techniques to reveal how moral reasoning and biases are distributed across distinct computational stages in neural networks.

Original authors: Mayank Goel, Aritra Das, Paras Chopra

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mayank Goel, Aritra Das, Paras Chopra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a computer makes a tough choice, like deciding who to save in a classic "trolley problem" (where a runaway train must be diverted to hit one group of people instead of another). Usually, we use giant, complex AI brains (Large Language Models) to do this, but they are like black boxes: we see the answer, but we have no idea how they got there.

This paper is about building a tiny, transparent AI brain specifically designed to solve these moral puzzles so we can actually watch its gears turn.

Here is a simple breakdown of what they did and what they found:

1. Building a "Moral Calculator" Instead of a Black Box

The researchers didn't just feed a giant AI a bunch of stories. Instead, they built a custom, small model (only 104,000 parameters—tiny for an AI) from scratch.

Think of their model like a structured spreadsheet rather than a chaotic library. Every time the AI looks at a scenario, it breaks the situation down into three specific facts for every person involved:

Who are they? (e.g., a doctor, a criminal, a baby).
How many are there? (e.g., 1 person, 5 people).
Which side are they on? (Team A or Team B).

The model's "hypothesis" is that making a moral decision is just a math problem of identifying these three things and weighing them against each other.

2. How Smart is It?

They trained this tiny model on 3 million real human decisions from the "Moral Machine" dataset.

The Result: It got 77% accuracy.
The Takeaway: You don't need a massive, opaque super-computer to make moral choices. A small, simple, and transparent model can learn human moral principles just as well as the big ones, but with the added benefit that we can actually see how it thinks.

3. Peeking Inside the Machine (The "X-Ray" Vision)

Because the model is small and custom-built, the researchers could use special tools to look inside its "brain" and see where specific biases live. They found three fascinating things:

A. The "Moral Hierarchy" (Who matters most?)

They tested the model by swapping characters in the scenarios to see who influenced the decision the most.

The Findings: The model learned a clear hierarchy of "worth," just like humans do.
- High Value: Pregnant women and babies in strollers had the biggest positive impact (the model really wanted to save them).
- Low Value: Criminals had the strongest negative impact (the model was very willing to sacrifice them).
- Neutral: Regular "Men" and "Women" were treated as the baseline, with almost no special weight.
The Metaphor: Imagine the model has a scale. It doesn't just count heads; it puts heavy weights on "babies" and light weights on "criminals."

B. Where the Bias Lives (The Factory Floor)

The model has two layers of processing (think of them as two assembly stations). The researchers found that different types of bias happen at different stations:

Station 1 (Layer 0): This is where the model decides if someone is a criminal. It spots the "bad guy" label immediately.
Station 2 (Layer 1): This is where the model decides between humans and animals. It takes a second step to figure out that a human is more important than a cat.
The Metaphor: It's like a factory where one worker checks for "dangerous items" (criminals) right at the door, while a second worker further down the line checks if the items are "people" or "pets."

C. The "Secret Circuit"

They found a tiny, sparse group of neurons (only 45 out of 256) that act as the final decision-makers. If you turned these specific neurons off, the model's ability to make the right moral choice dropped slightly. It's like finding the specific fuse in a house that controls the front door light.

4. Why This Matters

The paper argues that by building small, transparent models, we can stop guessing why AI makes bad moral choices.

The Problem: If a giant AI is biased, it's hard to fix because we don't know where the bias is hiding.
The Solution: With this small model, we can see that the "criminal bias" happens in Layer 0. This means we could potentially fix it by tweaking just that specific part of the code, rather than trying to retrain the whole system or scrub the data.

Summary

The researchers built a small, see-through AI to solve moral dilemmas. They proved that:

Small models can learn human moral rules effectively.
These models learn a specific "hierarchy" of who is valuable (babies > criminals).
Different biases (like age or species) happen at different stages of the AI's thinking process.
Because the model is simple, we can pinpoint exactly where these biases live and potentially fix them surgically.

The code for this "moral microscope" is available for anyone to study, proving that understanding AI ethics doesn't require a black box—it requires a clear window.

Technical Summary: Building Interpretable Models for Moral Decision-Making

Problem Statement

As AI systems increasingly encounter moral dilemmas (e.g., autonomous vehicles, content moderation), there is an urgent need to understand not only what decisions these systems make but how they reason and where moral biases originate within their architectures. Current approaches often rely on probing opaque, large-scale Language Models (LLMs) post-hoc, which limits mechanistic understanding. This paper addresses the challenge of building a transparent, small-scale model capable of learning general moral principles from the Moral Machine dataset (Awad et al., 2018) to facilitate detailed interpretability analysis.

Methodology

Architecture Design

The authors propose a custom, 2-layer transformer model specifically optimized for structured moral reasoning tasks. Unlike general-purpose LLMs, this architecture exploits the compositional nature of trolley-problem scenarios.

Input Representation: Each moral dilemma is represented as a pair of outcomes ( $O_0, O_1$ ). Each outcome is encoded as a sequence of tokens corresponding to 23 character types (e.g., Man, Woman, Criminal, Doctor). For each character $c$ with cardinality $n_c$ in outcome $t$ , a compositional embedding is constructed:
$e^{(t)}_c = [E_{char}(c) ; E_{card}(n_c) ; E_{team}(t)]$
This design encodes the hypothesis that moral reasoning decomposes into identifying stakeholders, quantifying impact, and resolving competing interests.
Model Configuration: The model uses an embedding dimension $d=64$ , 2 attention heads, and 2 transformer layers ( $L=2$ ). It processes a sequence of 46 tokens (23 per outcome) preceded by a learnable [CLS] token.
Symmetry Enforcement: To ensure side-invariance (where the preference for Group A over B is the complement of B over A), the inference process averages predictions from both input orderings:
$p(O_1 \succ O_0) = \frac{1}{2} [\sigma(f(O_0, O_1)) + (1 - \sigma(f(O_1, O_0)))]$
Classification: The final [CLS] representation passes through a two-layer MLP to produce a scalar logit, converted to a probability via sigmoid. This probabilistic output allows the model to learn uncertainty when faced with contradictory training preferences.

Interpretability Techniques

The paper employs three complementary methods to analyze the model's internal mechanisms:

Causal Intervention: Using the DoWhy framework, the authors estimate the Average Treatment Effect (ATE) of specific character types on decision outcomes, controlling for group size confounders.
Layer-wise Attribution: Attention weights are extracted from each layer and correlated with bias scores across five dimensions (legality, gender, social role, age, species) to localize where biases emerge.
Circuit Probing: Sparse binary masks are trained over a frozen model to identify specific neurons (in MLP blocks and attention heads) that causally implement the moral hierarchy. This is validated through targeted ablation.
Local Relevance: Gradient-weighted attention relevance is computed to explain individual token contributions to specific decisions.

Key Results

Performance

The 2-layer model achieves 77.1% accuracy on held-out Moral Machine scenarios (using the configuration $d=64, H=2, L=2$ ).
While a larger configuration ( $d=64, H=4, L=3$ ) achieved 77.5% accuracy, the authors selected the smaller model to prioritize interpretability and computational efficiency.
The model demonstrates that transparency and predictive capability in moral choices are not mutually exclusive.

Interpretability Findings

Moral Hierarchy (Causal Intervention): The model learns an implicit hierarchy of moral worth independent of utilitarian counts. "Pregnant" (ATE +0.12) and "Stroller" (ATE +0.11) show the strongest positive influence, while "Criminal" (ATE -0.10) shows the strongest negative influence. Generic categories like "Man" and "Woman" cluster near zero, serving as moral baselines.
Bias Localization:
- Legality Bias: Localizes almost entirely to Layer 0, suggesting criminal status is identified early via character embeddings.
- Species Bias: Emerges predominantly in Layer 1, indicating that human-animal distinctions require compositional reasoning over multiple characters.
- Functional Specialization: Specific attention heads specialize in different biases (e.g., Layer 0 Head 1 for legality; Layer 1 Head 1 for species discrimination).
Circuit Discovery: Circuit probing identified a highly sparse subnetwork (45 out of 256 neurons, 17.6% sparsity) in Layer 1's MLP block that causally implements the moral scoring signal. Ablating this circuit reduced agreement with model-score labels by 1.2 percentage points, accounting for approximately 8.3% of the model's margin above chance.
Token Relevance: In specific scenarios (e.g., 3 Men vs. 3 Criminals), the decision is primarily driven by the "Criminal" token (relevance score 0.27), while demographic tokens like "Man" contribute weakly.

Significance and Claims

The paper claims that moral competence does not require large, pretrained models; a simple, structurally constrained model trained on the intuitive structure of trolley problems is sufficient to learn and encode human moral biases.

The primary significance lies in transparency. By building a small, custom architecture, the authors demonstrate that:

Moral biases are not monolithic but localize to distinct computational stages (e.g., early layers for identity-based biases, later layers for compositional reasoning).
This granularity enables targeted intervention strategies. For instance, knowing that criminal bias localizes to Layer 0 Head 1 allows for specific debiasing techniques (such as orthogonalizing representations or clamping attention weights) rather than coarse dataset rebalancing.
The work establishes a foundation for exploring moral reasoning in larger LLMs, using this interpretable model as a base for understanding how moral principles are encoded in more complex systems.

The authors acknowledge that training on aggregate human preferences inherently inherits cultural biases, but argue that the transparency of their approach is a necessary step toward understanding and potentially mitigating these biases in AI systems.