🤖 AI

How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models

This study finds that while enabling reasoning modes in frontier LLMs does not significantly alter their aggregate binary moral verdicts, it notably reduces cross-model disagreement and demographic inconsistencies in specific contentious scenarios while more frequently shifting the models' self-labeled ethical frameworks.

Original authors: Sai Sourabh Madur

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sai Sourabh Madur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have five different super-smart robots (AI models) from five different tech companies. You ask them a series of 100 tricky moral questions, like "Is it okay to push one person off a bridge to save five others?" or "Should a doctor prioritize a patient based on their job or their age?"

The researchers wanted to see what happens when they tell these robots to "think before they speak."

Normally, the robots answer instantly (like a reflex). In "Thinking Mode," the robots pause, generate a long internal reasoning process, and then give their answer. The paper asks: Does taking that extra time to think actually change their moral compass, or do they just say the same thing but with a longer explanation?

Here is the breakdown of what they found, using simple analogies:

1. The Big Picture: The Crowd Still Agrees

When you look at the robots' answers as a whole group, taking time to think didn't really change the final verdict.

The Analogy: Imagine a jury of five people. Whether they shout their answer immediately or whisper it to each other for a minute first, they still end up agreeing on the verdict about 78% of the time. The "Thinking Mode" didn't make them suddenly agree or disagree more often as a group.

2. The "Hard" Cases: Thinking Helps a Little Bit

However, the story changes when you look at the really difficult, controversial questions (like complex trolley problems or modern ethical dilemmas).

The Analogy: In the "Instant Mode," the five robots were basically guessing on these hard questions, agreeing with each other only about as often as if they were flipping coins.
The Result: When they switched to "Thinking Mode," they started to agree with each other a bit more. It wasn't a magic fix, but it was like they stopped shouting random guesses and started finding a little more common ground.
The Catch: The researchers warn that this improvement is "directional" (it went the right way) but not statistically rock-solid yet. It's a hint, not a guarantee.

3. The "Reasoning" vs. The "Verdict"

Here is the most interesting twist: The robots changed their reasons much more often than they changed their answers.

The Analogy: Imagine two people both saying, "No, you can't have that cookie."
- Person A (Instant) says: "Because I said so."
- Person B (Thinking) says: "No, because cookies are bad for your teeth and you promised to eat veggies first."
- They both said "No," but their internal logic shifted.
The Finding: The robots frequently changed the ethical framework they claimed to use (e.g., switching from "Utilitarian" to "Deontological") even when their final Yes/No answer stayed the same. It's like a lawyer changing their legal argument in the middle of a trial but still asking for the same verdict.

4. The "Demographic" Test: Thinking Makes Them Fairer

The researchers tested if the robots judged people differently based on their race, gender, or nationality.

The Analogy: Imagine a robot judging a crime. In "Instant Mode," it might give a harsher sentence if the criminal is described as an "immigrant" versus a "citizen."
The Result: When three of the five robots switched to "Thinking Mode," they became much more consistent. They stopped letting the person's background sway their judgment as much. It's like the "thinking" process acted as a filter that removed the noise of bias.

5. The "Apples vs. Oranges" Problem (A Major Caveat)

The paper has a huge warning label: We didn't actually compare the same amount of "thinking."

The Analogy: Imagine asking five students to solve a math problem.
- Student A (Claude) spends 1 minute thinking.
- Student B (Qwen) spends 80 minutes thinking.
- Student C (GPT) spends 2 minutes.
The Reality: The researchers couldn't make them all "think" for the exact same amount of time because the companies control the buttons differently. One robot might be doing a quick mental check, while another is writing a whole essay in its head. So, when we say "Thinking Mode," we are comparing a light stretch to a heavy workout depending on which robot you use.

Summary of the 5 Hypotheses

The researchers had five guesses before they started. Here is what happened:

Do different robots have different moral defaults? Yes. (Some lean more on "greatest good," others on "rules").
Does changing the wording of a question change the answer? No. (The robots were surprisingly good at ignoring tricky wording).
Does the robot's answer change based on the person's background? Sometimes. (Instant mode was inconsistent; thinking mode fixed this for some robots).
Do robots agree on easy questions but fight on hard ones? Yes. (They agreed on simple stuff, but were confused on the hard stuff until they thought).
Does "Thinking Mode" change the moral judgment? Sort of. (It rarely changed the final Yes/No, but it often changed why they said it, and it helped them agree slightly more on the hardest questions).

The Bottom Line

Turning on "Thinking Mode" doesn't turn the robots into completely new moral beings. They generally give the same "Yes" or "No" answers. However, thinking does seem to help them:

Be slightly more consistent with each other on the hardest questions.
Be less biased against specific groups of people.
Change their internal logic even if the final answer stays the same.

The researchers released all their data (the questions, the answers, and the robots' "thought processes") so other scientists can double-check this work.

Technical Summary: How Does Thinking Mode Change LLM Moral Judgments?

Problem Statement
As reasoning-trained Large Language Models (LLMs) are increasingly deployed in high-stakes moral domains (e.g., triage, policy summarization, content moderation), providers have introduced "thinking modes" that allocate additional inference-time computation to reasoning. While prior research has established that different models encode varying ethical preferences, a critical gap remains: does enabling explicit reasoning change moral preferences within a single model checkpoint? Furthermore, is "thinking mode" a comparable construct across different providers, given their distinct API implementations and computational budgets?

Methodology
The authors conducted a controlled, single-checkpoint comparison across five frontier model families: Claude Sonnet 4.6, GPT-5.5, Gemini 3 Flash, DeepSeek-V3.1, and Qwen3.5-397B.

Experimental Design: For each model family, the model weights were held constant. The only variable was the provider's canonical reasoning-mode parameter (e.g., toggling reasoning.enabled from False to True or adjusting reasoning.effort). This isolates the effect of the reasoning mechanism from changes in model architecture or training data.
Benchmark: A 100-scenario moral-reasoning benchmark was constructed, covering five categories:
1. Trolley Problems (15 scenarios).
2. Moral Foundations Theory (25 scenarios across five foundations).
3. Paraphrase Consistency (20 scenarios testing surface-form robustness).
4. Demographic Sensitivity (30 scenarios varying demographics like race, gender, and nationality).
5. Contemporary Dilemmas (10 scenarios on AI safety, bioethics, etc.).
Elicitation: Models were prompted to provide a binary verdict, a confidence rating, a self-labeled ethical framework (from a fixed set of six), and a reasoning trace. Three responses were sampled per configuration per scenario ( $N=3$ ).
Metrics: The study measured cross-model agreement (Krippendorff's $\alpha$ ), paraphrase consistency, demographic-judgment inconsistency, and the rate of verdict flips versus framework shifts between instant and thinking modes.

Key Contributions

Controlled Instant-vs-Thinking Comparison: The study provides the first systematic evaluation of reasoning effects within single checkpoints across five diverse model families, removing capability confounds associated with comparing different model versions.
Benchmark and Pipeline: The authors released a reproducible open-source pipeline, the 100-scenario benchmark, and 2,963 raw responses (including provider-exposed reasoning artifacts) for future analysis.
Empirical Findings on Reasoning Effects: The paper quantifies how reasoning impacts moral judgments, specifically regarding verdict stability, framework self-labeling, and demographic consistency.

Results

Aggregate Agreement: Enabling thinking mode did not significantly change aggregate cross-model agreement on binary moral verdicts. Krippendorff's $\alpha$ remained statistically indistinguishable between instant ($0.781$) and thinking ($0.789$) modes.
Hard vs. Easy Cases (Stratification):
- Easy Cases: On 79 scenarios where models initially agreed (instant mode), agreement remained high, though reasoning introduced minor dissent in four cases (reducing $\alpha$ from $1.00$ to $0.95$).
- Hard Cases: On 21 "model-disputed" scenarios (where instant-mode agreement was near chance, $\alpha \approx 0.08$ ), reasoning directionally narrowed cross-model disagreement. Mean pairwise agreement rose from $5.4/10$ to $6.7/10$ . However, the authors note this improvement is suggestive; the paired bootstrap 95% confidence interval for the change in $\alpha$ ( $[-0.05, 0.36]$ ) includes zero, and the result does not survive multiple-comparisons correction.
Verdicts vs. Frameworks: Reasoning changed the self-labeled ethical framework more often than it changed the binary verdict. Across all models, the framework-shift rate (9–19%) exceeded the verdict-flip rate (1–9%). For example, DeepSeek-V3.1 showed an 8% verdict flip rate but a 19% framework shift rate.
Demographic Consistency: Reasoning reduced demographic-judgment inconsistency (variance in verdicts based on demographic cues) in three of five models (DeepSeek, GPT-5.5, Qwen). It did not increase inconsistency in any model.
Paraphrase Robustness: All models demonstrated high robustness to paraphrasing (consistency rates of 0.90–1.00), contradicting the hypothesis that surface-form variations would cause non-trivial verdict flips in reasoning-trained models.
Construct Validity Caveat: The study highlights a significant limitation: "lightweight thinking" is not computationally comparable across providers. Mean reasoning-token spend per call ranged from 33 tokens (Claude) to 2,639 tokens (Qwen3.5), an 80-fold spread. Consequently, cross-family comparisons conflate "reasoning enabled" with "amount of reasoning spent."

Significance and Claims
The paper modestly claims that enabling reasoning mode does not materially alter aggregate moral verdicts but directionally improves agreement on difficult, contested scenarios where instant-mode models diverge. It further establishes that reasoning acts as a corrective for demographic inconsistency in several models without increasing bias.

The authors emphasize that while reasoning often leads models to re-label their ethical rationale (shifting frameworks), it rarely overturns the final binary decision. The study concludes that "thinking mode" is a provider-specific construct with varying computational depths, and while it offers benefits for consistency in hard cases and demographic sensitivity, it is not a monolithic solution for moral alignment. The results suggest that for fairness-sensitive applications, enabling reasoning is a defensible default, but self-labeled frameworks should not be treated as reliable signals of a model's underlying ethical logic without independent validation against reasoning traces.