🤖 machine learning

The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions

This paper reveals a "judgment-consequence gap" in large language models, showing that while they agree with humans that patients are responsible for health-harming behaviors, they systematically refuse to let this responsibility influence scarce resource allocation, unlike humans who consistently favor less-culpable patients.

Original authors: Hadi Hosseini, Samarth Khanna, Leona Pierce

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hadi Hosseini, Samarth Khanna, Leona Pierce

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the referee at a very high-stakes game where the prize is a life-saving medical treatment, like a new kidney or a special surgery. You have two players: Player A, who has always been healthy, and Player B, who got sick after doing something risky, like smoking heavily or drinking too much. In the real world, when people have to decide who gets the prize, they often look at who "deserves" it. If Player B made their own bad choices, many people feel Player B should pay the price and lose the chance. This idea is called moral responsibility: the belief that if you cause your own trouble, you can't expect a free pass.

Now, imagine you ask a super-smart robot, a Large Language Model (LLM), to be the referee. These robots are like digital brains that have read almost everything on the internet. They are great at giving advice, writing stories, and even helping doctors diagnose illnesses. But here is the big question: When these robots have to make a tough choice about who gets a scarce medical resource, do they think like humans? Do they look at Player B's bad choices and say, "Sorry, you made your bed, now lie in it"? Or do they have a different set of rules? This paper dives into that exact mystery, testing whether these AI referees follow the same moral playbook as us or if they are playing a completely different game.

The researchers set up a series of digital scenarios, like a video game simulation, where an AI had to choose between two patients for a kidney transplant. One patient had a history of bad habits (like smoking or drug use), and the other didn't. The AI was asked a few questions in a row: First, "Is the patient responsible for their bad habit?" Second, "Is the patient responsible for getting sick because of that habit?" And finally, the big one: "Who should get the kidney?"

Here is where the plot twist happens. When the AI was asked if the patient was responsible for their bad habits, it agreed with humans almost perfectly. It said, "Yes, that patient made a bad choice." It even agreed that the patient was responsible for getting sick. But then, when asked to actually give the kidney to the "good" patient, the AI completely changed its mind. Instead of picking the patient who didn't smoke, the AI overwhelmingly chose to flip a coin.

The researchers call this the "Judgment-Consequence Gap." It's like a robot saying, "I know you broke the rules, and I know you are the one who broke them," but then adding, "But I'm not going to punish you for it because that would be unfair." While humans usually connect the dots—thinking, "You caused the problem, so you shouldn't get the prize"—the AI breaks the chain. It admits the patient is at fault but refuses to let that fault decide who gets the life-saving treatment.

The study also found some other interesting quirks. If the patient didn't know their bad habit was dangerous (maybe they were never told smoking causes kidney failure), the AI was much more forgiving, almost letting them off the hook. Humans, on the other hand, didn't care as much about whether the patient knew the risks; they still felt the patient was responsible. Also, when the researchers turned on the AI's "thinking mode"—making it pause and reason longer before answering—the gap actually got wider. The AI became even more convinced that the patient was responsible, but even more stubborn about refusing to use that responsibility to decide who gets the kidney.

In short, this paper suggests that while AI can understand the concept of "you did it, you own it," it doesn't seem to believe that "owning it" means you should lose out on a life-saving resource. It treats the decision to give a kidney as a matter of strict fairness where everyone gets an equal chance, regardless of past mistakes. This isn't just a glitch; it seems to be a deep, built-in rule in how these models think. So, if we ever let AI help doctors decide who gets a transplant, we need to know that the robot might agree with us on who is to blame, but it will likely refuse to let that blame change the outcome.

Technical Summary: The Judgment-Consequence Gap in LLM Moral Reasoning

Problem Statement

As Large Language Models (LLMs) are increasingly integrated into high-stakes healthcare domains for decision support and resource allocation, understanding their moral reasoning capabilities is critical. A specific ethical challenge arises when scarce medical resources (e.g., organs, specialized treatments) must be allocated among patients, some of whom have contributed to their own illness through health-harming behaviors (e.g., smoking, drug use).

Human moral reasoning, grounded in attribution theory, typically links perceived responsibility for a negative outcome to downstream consequences, often deeming those responsible for their misfortune less deserving of scarce resources. However, it remains unclear whether LLMs replicate this linkage or if they diverge in how they assess responsibility versus how they act upon that assessment in consequential decisions. Prior work has largely treated moral reasoning as a single-step evaluation, leaving a gap in understanding where in the reasoning chain human and LLM behaviors diverge.

Methodology

The authors conducted a comprehensive empirical study evaluating 19 distinct LLM configurations (spanning 12 models from families including Claude, GPT, DeepSeek, Gemini, and Llama, with both reasoning-enabled and non-reasoning modes) against human baselines derived from prior studies.

Experimental Design:
The study utilized clinical vignettes adapted from human studies on kidney transplant allocation (Chan et al., 2024), lung cancer treatment (Björk, Lynøe, and Juth, 2015), and hip replacement surgery (Björk, Juth, and Lynøe, 2018). The experimental design traced the moral reasoning process across successive levels:

Behavioral Responsibility: Is the patient responsible for the harmful behavior?
Disease Responsibility: Is the patient responsible for developing the illness?
Deprivation Responsibility: Is the patient responsible for being denied the scarce resource?
Allocation Decision: Given scarcity, should the resource go to the non-culpable patient, the culpable patient, or be decided randomly?

Variables Manipulated:

Behavioral Change: Whether the patient continued or stopped the harmful behavior post-diagnosis.
Epistemic State (Knowledge Level): Whether the patient knew about health risks, had access to information but didn't know, or had no access to information. This included conditions where patients were actively misled by third parties, humans, or AI agents.
Reasoning Capability: Comparison of models with "thinking" (extended reasoning) enabled versus disabled.

Procedure:
Models were prompted as simulated study participants in multi-turn conversations to replicate the sequential structure of human studies. Responses were collected via structured JSON outputs containing Likert scale ratings and allocation choices. Statistical comparisons were made using Welch's t-tests, Mann-Whitney U tests, and $\chi^2$ tests, aggregating results across model configurations to identify general patterns.

Key Contributions

Identification of the Judgment-Consequence Gap: The paper defines and empirically demonstrates a systematic disconnect where LLMs agree with humans on moral responsibility judgments but refuse to let those judgments influence consequential allocation decisions.
Granular Tracing of Reasoning Chains: Unlike prior work that treats moral reasoning as a monolithic output, this study isolates specific stages (behavior $\to$ disease $\to$ deprivation $\to$ allocation) to pinpoint exactly where the divergence occurs.
Analysis of Epistemic Sensitivity: The study reveals that LLMs are significantly more sensitive to a patient's access to information (informed consent) than humans, sharply reducing responsibility attributions when information is unavailable, a pattern not observed in human baselines.
Impact of Extended Reasoning: The authors investigate whether enabling "thinking" modes bridges the gap, finding instead that extended reasoning often amplifies responsibility attribution while simultaneously pushing allocation decisions further toward randomization, widening the gap.

Key Results

1. Agreement on Responsibility, Divergence on Consequence

Responsibility Attribution: LLMs and humans show near-complete agreement that patients are responsible for their harmful behaviors (mean scores ~4.4/5). However, LLMs (particularly non-reasoning models) attribute significantly less responsibility for the resulting disease and deprivation compared to humans.
Allocation Decisions: While a majority of human participants (67.6%) allocated the scarce resource to the less-culpable patient, LLMs overwhelmingly defaulted to random allocation (deciding by chance).
- Reasoning models chose the non-culpable patient only 9.4% of the time.
- Non-reasoning models chose the non-culpable patient 24.7% of the time.
- In contrast, humans chose the non-culpable patient 67.6% of the time.
Normative Stance: LLMs actively endorsed the view that behavior-based allocation is unfair (mean 5.6/7 on unfairness scale) compared to humans (3.9/7). Reasoning traces confirm that models acknowledge responsibility but explicitly set it aside based on fairness principles rather than ignoring the judgment.

2. Sensitivity to Knowledge and Epistemic State

Human Baseline: Human responsibility judgments and allocation decisions remained largely flat across different knowledge conditions (known, accessible, or unknown risks).
LLM Sensitivity: LLMs showed dramatic sensitivity. When patients lacked access to information about health risks, LLMs significantly reduced responsibility and fault attributions. This sensitivity extended to allocation decisions; for non-reasoning models, the rate of choosing the non-culpable patient dropped from 37.4% (knowledge condition) to 12.6% (no-access condition).
Misinformation: LLMs treated patients who were actively misled (by AI, humans, or third parties) as morally equivalent to those who had no access to information, further reducing responsibility attributions.

3. Effect of Extended Reasoning

Enabling "thinking" modes increased responsibility attribution scores for most model families, bringing them closer to human assessments on the judgment side.
However, this did not translate to human-like allocation behavior. Instead, thinking-enabled models often increased their preference for randomization, thereby widening the judgment-consequence gap. This suggests the gap is a stable normative commitment rather than a processing limitation.

4. Generalization

These patterns held across three medical domains (kidney, lung cancer, hip replacement) and four types of health-harming behaviors (alcohol, drugs, smoking, poor diet), indicating the divergence is a general property of current LLMs.

Significance and Claims

The paper claims that the judgment-consequence gap represents a fundamental misalignment in how LLMs connect moral assessment to action. The authors argue this is not a failure of knowledge (as LLMs correctly identify responsibility) but a failure of composition: LLMs apply a different normative framework (likely contractualist or egalitarian) that prioritizes procedural fairness and equal treatment over desert-based reasoning when making consequential decisions.

Implications for Healthcare:
The authors highlight a risk of "partial alignment." Clinicians consulting LLMs may see the model correctly identify a patient's responsibility (matching human intuition) and falsely assume the model will recommend a corresponding allocation. Instead, the model may refuse to deprioritize the patient, framing such an action as unjust. This creates a risk of false confidence in AI advice, particularly in scenarios involving patients misled by AI agents, where the model may absolve responsibility based on the source of misinformation.

Implications for Alignment Research:
The findings suggest that current alignment methods (like RLHF) may produce "weak alignment"—a surface-level correspondence with human values that breaks down when models must compose moral assessments into decisions. The authors posit that the gap reflects a stable normative commitment that deepens with reasoning capabilities, suggesting that interventions focusing solely on behavioral consistency or surface-level value preferences may be insufficient to bridge the gap between judgment and action.