Are Language Models Sensitive to Morally Irrelevant Distractors?

Technical Summary: Are Language Models Sensitive to Morally Irrelevant Distractors?

Problem Statement

Current AI alignment research relies heavily on the assumption that Large Language Models (LLMs) possess stable moral preferences. Existing benchmarks evaluate models using value statements or moral scenarios, implicitly assuming that if a model is "aligned," its moral judgments will remain consistent across similar scenarios.

However, this paper challenges that assumption by applying the "situationist" view from moral psychology. In humans, moral judgments are often sensitive to "morally irrelevant situational factors" (e.g., ambient noise or pleasant smells). The researchers investigate whether LLMs exhibit similar cognitive biases—specifically, whether emotionally-valenced but morally irrelevant context can steer an LLM toward pro-social or anti-social judgments.

Methodology

The authors developed a novel framework to test for "situationist" moral bias using the following components:

1. Moral Distractors Dataset: They curated a multimodal dataset of 60 "moral distractors" (10 positive, 10 neutral, and 10 negative for both text and images).
   • Textual distractors were sourced from the IDEST dataset, filtered to ensure they were sensory, mild, and lacked moral lessons.
   • Visual distractors were sourced from the OASIS dataset, excluding people and animals to maintain moral irrelevance.
2. Benchmarks: The distractors were injected into two distinct moral benchmarks:
   • MORALCHOICE: A dataset of forced-choice scenarios (low and high ambiguity) based on the "10 rules of common morality."
   • r/AITA (Am I The Asshole): A dataset of real-world interpersonal dilemmas where models must provide a verdict (e.g., YTA, NTA, ESH).
3. Models Evaluated: The study tested four model families: GEMMA-3-4B-IT, LLAMA-3.2-4B-INSTRUCT, GPT-4.1, and QWEN3-4B.
4. Metrics:
   • MMAP (Marginal Moral Action Probability): Used in MORALCHOICE to measure the probability of selecting a rule-following action.
   • Verdict Distribution: Used in r/AITA to track shifts in social judgments.
   • Moral Foundations Theory (MFT): Used to check if distractors changed the underlying moral reasoning/framing.

Key Contributions

• Introduction of "Moral Distractors": A new method for evaluating the robustness of LLM moral reasoning against irrelevant affective context.
• Discovery of Cognitive Moral Biases: Evidence that LLMs do not have stable moral "characters" but are highly susceptible to situational "leakage."
• Multimodal Evaluation: Demonstrating that both text and images can trigger these moral shifts.
• Philosophical Re-framing: Shifting the discourse from "model character" to "situational management" in AI alignment.

Results

• Significant Moral Shifts: Moral distractors can shift LLM moral judgments by over 30%, even in low-ambiguity scenarios.
• Valence-Dependent Behavior:
  • Negative distractors consistently decrease pro-social behavior and increase "immoral" choices.
  • Positive distractors tend to increase pro-social behavior (though the effect is sometimes muted by existing alignment).
  • In r/AITA tasks, negative distractors significantly increased "ESH" (Everyone Sucks Here) verdicts, with LLAMA-3.2-3B seeing a 9.5% increase.
• Visual vs. Textual: Visual distractors induced similar shifts to textual ones, proving the bias is not limited to language.
• Ablation Insights:
  • Reasoning: Enabling "reasoning" (Chain-of-Thought) is effective at mitigating these biases in low-ambiguity scenarios.
  • Model Size/Tuning: There was no consistent relationship between model size and sensitivity, though instruction-tuning affected models differently across families.
• Stability of Reasoning: Interestingly, while verdicts changed, the Moral Foundation scores remained flat, suggesting the models change their judgment without necessarily changing their underlying moral framework.

Significance and Implications

1. Challenge to Alignment Assumptions: The study proves that "alignment" is not a static property. A model that passes a moral benchmark in a "neutral" environment may fail catastrophically in a "negative" environment.
2. Shift in Moral Responsibility: The authors argue that since LLMs are context-responsive tools rather than stable moral agents, moral responsibility should shift to the developers and deployers. Designers must ensure that the "situations" (prompts and interfaces) in which LLMs operate are carefully managed to avoid inducing bias.
3. Practical Safety Recommendations:
   • Contextual Truncation: Separating fact-gathering from decision-making to limit irrelevant affective context.
   • Stress-Testing: Safety guardrails must be tested against systematically varied distractors (especially negative ones) to ensure robustness.
   • Contextual Alignment: Moving toward "Realistic Use and Tangible Effect" (RUTEd) evaluations rather than static, isolated benchmarks.