💬 NLP

CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

The paper introduces CLASH, a dataset of high-stakes dilemmas with diverse character perspectives, to reveal that even advanced language models struggle with value-based decision-making, exhibiting specific failure patterns like early commitment and limited understanding of value shifts despite reasonable predictions of psychological discomfort.

Original authors: Ayoung Lee, Ryan Sungmo Kwon, Peter Railton, Lu Wang

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ayoung Lee, Ryan Sungmo Kwon, Peter Railton, Lu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make tough life choices. Most previous tests asked the robot simple questions like, "Is it okay to jaywalk if you're late for work?" But in the real world, the hardest choices aren't simple; they are high-stakes dilemmas where every option hurts someone, and values clash like two storms colliding.

This paper introduces CLASH (Character perspective-based LLM Assessments in Situations with High-stakes), a new "exam" designed to see if AI can handle these messy, high-pressure situations.

Here is a breakdown of what they did and what they found, using simple analogies:

1. The Exam: CLASH

Think of previous AI tests as a multiple-choice quiz with short, clean sentences. CLASH is more like a drama series.

The Stories: Instead of one-liners, the dataset contains 345 long, detailed stories about life-or-death or life-changing situations (like a doctor deciding on a treatment, or a bank teller making a costly mistake).
The Characters: For every story, the AI has to answer from the perspective of different "characters." Some characters care only about safety; others care only about fairness. Some characters change their minds halfway through the story.
The Goal: The exam asks: "If you were Character A, would you do this? Would it make you feel sick to your stomach? Did your mind change from yesterday to today?"

2. The Big Findings: Where the AI Stumbles

The researchers tested 14 different AI models (including the newest, smartest ones like GPT-5 and Claude-4) on this exam. Here is what happened:

A. The "Can't Decide" Problem (Ambivalence)

The Metaphor: Imagine standing in front of two doors. One leads to a reward, the other to a trap. A human might say, "I'm torn; I can't pick."
The Result: The AI models are terrible at admitting they are torn. Even the smartest models forced a "Yes" or "No" answer when the situation clearly required a "Maybe." They struggled to say, "I don't know," even when the right answer was "I'm conflicted." In fact, the best model only got this right about 51% of the time.

B. The "Gut Feeling" Problem (Discomfort)

The Metaphor: Sometimes you know what you should do, but it feels wrong in your gut. Like a parent who has to fire a friend to save the company.
The Result: The AI is actually pretty good at spotting when a character should feel uncomfortable. If the story says, "Character A values honesty but has to lie," the AI correctly guesses, "Yes, Character A feels bad about this."

C. The "Memory" Problem (Value Shifts)

The Metaphor: Imagine a character who loves pizza, but then gets a stomach ache and decides they hate pizza now.
The Result: The AI models are terrible at updating their "minds." When a story says, "Character A used to value X, but now values Y," the AI often forgets the change and answers based on the old belief. It's like a student who studied for a test last year but forgot they learned a new subject this year. The accuracy dropped by nearly 43 points when values shifted!

3. How the AI "Thinks" (Reasoning Chains)

The researchers peeked inside the AI's "brain" to see how it solved these problems.

The Old Trick: In math or chess, smart AIs use a strategy called "backtracking" (checking their work, looking back at the rules). This works great for math.
The New Failure: In moral dilemmas, this backtracking didn't help. Instead, the AI developed two bad habits:
1. Early Commitment: The AI picks a side too quickly, like a judge banging a gavel before hearing the full story.
2. Overcommitment: Once it picks a side, it stubbornly sticks to it, ignoring evidence that suggests the other side might be right. It's like a lawyer who refuses to listen to the opposing argument once they've started their closing statement.

4. The "Steering" Test

The researchers also tested how easily they could "steer" the AI toward a specific value (like Safety vs. Self-Esteem).

The Finding: If an AI already really likes "Safety," it is very hard to make it choose "Self-Esteem" instead. The more the AI naturally prefers one value, the harder it is to steer it toward the opposite one.
The Perspective Trick: The AI listened better when asked, "What would Character A do?" (Third-person) rather than "What would you do?" (First-person). However, for the value of "Safety," the AI actually listened better when asked in the first person, likely because safety is a deeply ingrained instinct for the model.

Summary

The paper concludes that while AI is getting smarter at math and games, it is still very clumsy at navigating the messy, emotional, and changing world of human values. It struggles to admit when it's confused, it forgets when people change their minds, and it often gets stuck in its own stubborn logic.

Important Note: The authors emphasize that this is a diagnostic tool. They are not saying these AI models are ready to be doctors or judges. In fact, they warn that a high score on this test doesn't mean an AI is safe to use in real life; it just means we now have a better way to see where the AI is failing.

Technical Summary: CLASH – Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

Problem Statement

While Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as medicine, law, and finance, their ability to navigate value-laden dilemmas remains underexplored, particularly in contexts involving conflicting values and significant consequences. Prior research has largely focused on everyday, low-stakes scenarios or relied on synthetic, short-form situations that fail to capture the nuance of human value systems. Furthermore, existing benchmarks often constrain models to binary choices, conflating genuine decision ambivalence with scenario complexity, and overlook critical psychological aspects of decision-making such as discomfort and dynamic value shifts. There is a lack of rigorous evaluation frameworks that assess how LLMs reason from specific character perspectives, handle value conflicts, and adapt to changing value systems in high-stakes contexts.

Methodology

To address these gaps, the authors introduce CLASH (Character perspective-based LLM Assessments in Situations with High-stakes), a meticulously curated dataset and benchmark.

Dataset Construction

CLASH consists of 345 human-written, long-form dilemmas drawn from public web sources in medical, business, journalism/media, and government/politics domains. Each dilemma is enriched with:

Action: A specific, controversial action derived from the situation.
Value-Related Rationales: Pairs of supporting and opposing rationales generated and refined by human inspectors to ensure relevance and clarity.
Character Descriptions: 11 distinct character profiles per situation, categorized into Static (Straightforward, Simple Contrast, Swayed Contrast) and Dynamic (Shift, Half-Shift, False-Shift) types. These descriptions operationalize how characters prioritize values, experience ambivalence, or undergo value shifts over time.

The dataset includes 3,795 individual perspectives, designed to test models on three specific dimensions:

Ambivalence: The ability to recognize indecision when competing values are equally weighted or when a clear choice is not possible.
Psychological Discomfort: The capacity to detect the internal unease a character feels when making a difficult decision, grounded in cognitive dissonance theory.
Value Shifts: The ability to reason about how a character's acceptance of an action changes when their underlying value preferences evolve (e.g., from "previous" to "current" perspectives).

Experimental Setup

The authors benchmarked 14 models, including non-thinking LLMs (e.g., Qwen2.5, Llama3.1/3.3, Mistral, GPT-4o) and "thinking" Reasoning Language Models (RLMs) (e.g., GPT-5, Claude-4-Sonnet, Deepseek-3.1). Models were prompted to generate reasoning chains followed by answers to ternary (Yes/No/Ambiguous) or binary (Yes/No) questions.

Additional Analyses

Reasoning Chain Analysis: The authors analyzed successful vs. unsuccessful reasoning chains, examining cognitive behaviors (backward chaining, verification, early commitment, overcommitment) and the usage of ethical theories (e.g., Pragmatic Ethics, Deontology).
Conditional Steerability: A new metric was proposed to evaluate how effectively a model can be guided toward one value when it conflicts with another, contrasting with prior work on absolute steerability.
Framing Effects: The study compared first-person vs. third-person question framing to assess their impact on model performance and steerability.

Key Contributions

CLASH Dataset: A novel benchmark featuring 345 high-stakes, long-form dilemmas with 3,795 diverse character perspectives, moving beyond short, synthetic scenarios to capture real-world complexity.
Novel Evaluation Dimensions: The first systematic evaluation of LLMs on decision ambivalence, psychological discomfort, and dynamic value shifts, revealing specific failure modes not captured by standard accuracy metrics.
Conditional Steerability: The introduction of a method to measure steerability in conflicting value scenarios, revealing a significant negative correlation between a model's inherent value preferences and its ability to be steered toward opposing values.
Reasoning Insights: Identification of new failure patterns in value reasoning, specifically early commitment and overcommitment, and the finding that cognitive strategies effective in math (e.g., verification) are less effective in value reasoning.

Key Results

Struggle with Ambivalence: Even state-of-the-art proprietary models struggle with ambivalent decisions. For instance, GPT-5 achieved only 24.06% accuracy, and Claude-4-Sonnet achieved 51.01% in ambivalent settings (Simple Contrast), indicating a failure to recognize genuine indecision.
Discomfort vs. Value Shifts: While models reasonably predict psychological discomfort (e.g., GPT-5 reached 98.61% in Swayed Contrast), they fail significantly when reasoning about value shifts. GPT-4o-mini showed an average accuracy drop of 42.95 points when asked to reason from a "current" perspective compared to a "previous" one, with a maximum drop of 66.43 points.
Reasoning Failures: Successful reasoning chains in value tasks exhibited less backward chaining and verification than in math tasks. Instead, failures were characterized by early commitment (prematurely favoring one side) and overcommitment (persisting with an initial judgment despite conflicting evidence).
Ethical Theories: Successful chains were more likely to utilize Pragmatic Ethics and Rights Ethics, suggesting that frameworks focusing on real-world adaptation are more effective for value reasoning than rigid deontological or utilitarian approaches.
Steerability Correlation: There is a significant negative correlation ( $r = -0.243, p < 0.005$ ) between a model's inherent preference for a value and its steerability toward the opposing value. Stronger inherent preferences hinder the ability to be steered away.
Framing Effects: Third-person framing generally improved steerability, though first-person framing was uniquely effective for steering toward Safety relative to Self-Esteem, likely due to training data emphasis.

Significance and Claims

The paper claims that CLASH reveals fundamental limitations in current frontier LLMs regarding human-like judgment in high-stakes environments. The authors assert that:

High-stakes reasoning is distinct: Value-based reasoning does not benefit from the same cognitive behaviors (like verification) that aid math or game reasoning; it requires new strategies to handle ambiguity and dynamic values.
Current models are insufficient: Even the strongest models (GPT-5, Claude-4-Sonnet) exhibit significant gaps in handling ambivalence and value shifts, suggesting they are not yet reliable for autonomous decision-making in sensitive domains.
Methodological advancement: The introduction of "conditional steerability" and the focus on character perspectives provide a more nuanced view of model alignment than previous single-value or absolute alignment metrics.

The authors emphasize that their work is methodological and diagnostic. They explicitly caution that high performance on CLASH should not be interpreted as evidence of safety or reliability for real-world deployment in high-stakes domains. Instead, the benchmark serves to expose limitations and guide future research toward training models that better align with complex human reasoning processes.