Technical Summary: Can LLMs Imagine Moral Alternatives Beyond Binary Dilemmas?

Problem Statement

Existing research on Large Language Models (LLMs) and moral reasoning predominantly frames ethical decision-making as a selection task among pre-specified, competing options (e.g., Option A vs. Option B). This binary framing overlooks a central aspect of human moral cognition: moral imagination, defined as the capacity to identify novel courses of action, anticipate their consequences, and move beyond forced either/or choices. Current evaluations fail to assess whether LLMs can (1) shift their judgments when presented with non-binary alternatives (compromises or reframed options) and (2) generate such nuanced alternatives with quality comparable to human-authored ones. This gap limits the ability of AI systems to function as sophisticated moral agents in high-stakes settings where value conflicts require reconciliation rather than simple selection.

Methodology

Dataset Construction: MoralAltDataset

The authors introduce MoralAltDataset, a corpus of 307 moral dilemmas designed to evaluate moral imagination. The dataset is divided into two complementary subsets:

1. Advisor Dilemmas (156 items): Derived from movie plot synopses (MPST dataset), these provide contextually rich, narrative-driven human conflicts.
2. Agent Dilemmas (151 items): Adapted from the Litmus-Testing AI Values framework, these cover realistic, high-stakes scenarios AI systems might encounter (e.g., healthcare, waste management).

Each dilemma is augmented with two distinct types of alternatives beyond the original binary options (A and B):

• Compromise Alternative (Option C): Balances the competing aims of A and B through a concrete trade-off, preserving at least one core moral aim from each side.
• Reframed Alternative (Option D): Restructures the dilemma itself by introducing new principles, surfacing absent stakeholders, or redefining the conflict's scope/temporal frame.

Data Sources:

• Human Baseline: 164 dilemmas feature human-written alternatives (75 Advisor, 89 Agent) created by graduate-level annotators.
• LLM Generation: 143 dilemmas feature author-filtered alternatives generated by GPT-5.
• Total: The final four-option choice dataset includes 307 dilemmas, with human judgment data collected from 1,535 responses (5 participants per dilemma) under controlled conditions.

Experimental Design

The study addresses two primary research questions through two experimental setups:

1. Selection Sensitivity (Q1): Do LLMs shift their judgments when alternatives are introduced?

   • Setup: A four-option choice task (A, B, C, D) presented to 15 LLMs (including GPT, Claude, Gemini, Llama, Qwen, and Mistral families) and human participants.
   • Metrics: Selection rates, value shifts (using the LITMUS VALUES framework), and human-LLM agreement (measured via 4-way F1 and group-level agreement).

2. Generation Quality (Q2): Can LLMs generate alternatives comparable to humans?

   • Setup: LLMs generated compromise and reframed alternatives for the 164 human-written dilemmas.
   • Evaluation:
     • Pairwise Preference: Evaluators compared Human vs. LLM outputs head-to-head based on criteria like Feasibility, Balancing, Reframing, and Moral Acceptability.
     • Expert-Based Intrinsic Evaluation: Three philosophy specialists assessed 100 instances against fine-grained checklists for generation quality (e.g., Scenario Validity, Trade-off Mechanism) and ethical frameworks (Deontology, Utilitarianism, Virtue Ethics).

Key Results

1. Alternatives Reshape Moral Judgment

• Preference for Compromise: Across both humans and 15 LLMs, the introduction of alternatives significantly altered choice distributions. Compromise alternatives (Option C) were the most frequently selected option, often receiving a higher share of choices than the original binary options.
• Model Behavior: Closed-weight models (e.g., GPT-5, Claude) showed a strong, consistent preference for compromise over reframing. Open-weight models exhibited more heterogeneity.
• Value Shifts: The presence of alternatives systematically shifted the ethical values prioritized by LLMs. In Advisor dilemmas, models shifted from autonomy/respect toward Wisdom and prudence. In Agent dilemmas, there was a shift toward Cooperation and Adaptability, moving away from rigid rule-based values like Justice and Professionalism.
• Human-LLM Agreement: While exact 4-way agreement was modest, agreement increased significantly when humans selected alternatives. LLMs aligned with human choices in the "alternative" group (C/D) much more strongly than in the "binary" group (A/B), particularly for compromise solutions.

2. LLMs Generate High-Quality Alternatives

• Preference Over Humans: In pairwise preference evaluations, LLM-generated alternatives (specifically GPT-5, Claude Sonnet 4.5, and Qwen 3.5) were generally preferred over human-authored ones. They were perceived as clearer, better structured, and more aligned with the intended function of the alternative type.
• Structural vs. Feasibility Trade-off: A critical trade-off emerged. Models like GPT-5 excelled in structural criteria (e.g., explicit trade-off rules, moral novelty) but scored lower on Feasibility compared to models like Claude Sonnet 4.5 and Qwen 3.5. This suggests that emphasizing the defining features of an alternative (e.g., perfect balance) can sometimes reduce its practical groundedness.
• Ethical Alignment: GPT-5 achieved the highest scores across all three ethical dimensions (Deontology, Utilitarianism, Virtue Ethics) in expert evaluations, and its high preference scores correlated with expert judgments of normative defensibility.

Significance and Claims

The paper claims that current LLMs possess a meaningful but uneven capacity to engage in moral imagination.

• Beyond Binary: The study demonstrates that moral reasoning is not static; the availability of alternatives causally reshapes moral choices for both humans and machines, with compromise emerging as a robust conflict-resolution strategy.
• Generation Capability: LLMs are not merely selecting from a fixed set but can generate alternatives that often surpass human-authored ones in structural quality and ethical alignment, though they struggle to simultaneously optimize for both structural elegance and practical feasibility.
• Implications for AI Agency: These findings suggest that for LLMs to act as sophisticated moral advisors, they must be evaluated not just on their ability to choose between given options, but on their ability to imagine and propose "third ways" that reconcile competing values.

Limitations

The authors note that the dataset, while spanning narrative and agent scenarios, does not exhaust the diversity of real-world cultural, linguistic, or institutional constraints. The use of standardized zero-shot prompting and fixed option sets ensures reproducibility but does not capture interactive deliberation or stakeholder negotiation. Furthermore, the ethical evaluation relies on checklist-based proxies for normative frameworks, which are practical tools but not definitive philosophical measurements. The authors emphasize that the dataset is a diagnostic tool for studying LLM capabilities, not a prescriptive tool for high-stakes decision-making without human oversight.