Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas

Technical Summary: Framing Instability in LLM Ethical Stance

Problem Statement
Large Language Models (LLMs) are increasingly deployed to render or inform ethical judgments in advisory, evaluative, and deliberative roles. These deployments assume that a model's expressed stance reflects a stable underlying moral judgment rather than an artifact of phrasing. This paper challenges that assumption, demonstrating that ethical stances are vulnerable to framing instability, specifically under negation. While prior work has documented negation failures in semantic comprehension tasks (e.g., question answering), it remains unclear whether these failures persist when models render ethically consequential judgments, whether standard measurement practices accurately detect them, and how they vary across domains and model types.

Methodology
The authors conducted a large-scale audit of 16 models across 14 ethically fraught moral dilemmas spanning seven domains (medical, financial, legal, military, business, education, and science).

• Models: The study included 8 US commercial models (e.g., GPT-5 series, Claude 4.5, Grok-4.1), 4 Chinese commercial models (e.g., DeepSeek, GLM, Qwen), and 4 small open-weight models (1–4B parameters, e.g., LLaMA 3.2, Gemma 3).
• Experimental Design: Each scenario was presented under four framings:
  • F0 (Affirmative): "They should {action}."
  • F1 (Simple Negation): "They should not {action}."
  • F2/F3 (Compound/Conditional): Variations involving goals and conditions (e.g., "They should {goal} even if it means they must {action}").
• Data Collection: Approximately 27,000 model decisions were generated (30 samples per condition at temperature 0.7, plus deterministic runs for ablation).
• Measurement & Validation:
  • Binary Proxy: Responses were normalized to a binary "agree/disagree" scale adjusted for polarity.
  • Human Gold Standard: A stratified sample of 198 responses was independently coded by two human annotators using a five-class action stance taxonomy (Supports, Opposes, Permissible, Uncertain, Refuse) and an integrity check (Consistent vs. Decision-Reasoning Conflict).
  • LLM-as-Judge Validation: The authors tested whether an LLM judge could replace human coders, finding that automated judges collapse abstention categories, reproducing the forced-choice bias under study.

Key Contributions

1. First Polarity-Paired Audit: The paper presents the first audit of action-level stance stability in consequential moral dilemmas using polarity-paired proposals. It documents that failures established in comprehension tasks persist and take distinctive forms in ethical decision scenarios.
2. Measurement Critique: The study validates that the standard binary agree/disagree proxy over-counts genuine endorsement by 38.2% (rising to 45.6% under compound framing). Furthermore, 13.1% of gold-coded responses exhibit a conflict between the stated decision and the reasoning, suggesting LLM judges cannot replace human coders because they fail to capture abstention and hedging.
3. Negation Sensitivity Index (NSI): The authors introduce the NSI, a metric measuring the maximum swing in normalized response across framings (ranging from 0 for perfect invariance to 1 for complete inversion). They also derive a signed diagnostic to distinguish between "negation-blind under-flipping" (agreeing with both polarities) and "acquiescent over-flipping."
4. Domain and Model Variance: The study provides exploratory evidence that framing sensitivity varies by domain (financial/business scenarios show roughly twice the sensitivity of medical ones) and by model origin/architecture.

Key Results

• Stance Instability: Small open-weight models (1–4B) exhibit extreme instability, endorsing proposed actions 24% of the time under affirmative framing (F0) but up to 100% under simple negation (F1) and compound framing (F3). This represents a swing of up to 76 percentage points.
• Commercial Model Variance: Commercial models are generally more stable but still shift substantially. US commercial models showed a 36% increase in endorsement from F0 to F1, while Chinese-origin models showed a decrease, indicating that negation handling is not standardized across training regimes.
• Inter-Model Agreement: Agreement between model pairs drops from 73% on bare affirmative framing (F0) to 59% under simple negation (F1), indicating that models converge on "should" but diverge on "should not."
• Structural vs. Stochastic: Deterministic decoding (T=0.0) did not reduce sensitivity; in some cases, it increased measured NSI, suggesting the failures are structural (related to compositional processing) rather than artifacts of stochastic sampling.
• Reasoning Benefits: Reasoning-enabled variants (e.g., Grok-4.1-reasoning) showed a 57% reduction in sensitivity compared to non-reasoning baselines, though they remained vulnerable to compound negation.
• Response Integrity: 13.1% of human-coded responses contained a decision-reasoning conflict, concentrated in open-source models (31.8%) and compound negation framings (22.7%).

Significance and Claims
The paper argues that a model whose stance flips with phrasing cannot be relied upon in high-stakes decision scenarios. The significance of these findings lies in three areas:

1. Deployment Risk: In advisory roles (e.g., ethics consultations, content moderation), framing sensitivity means an analyst's phrasing choices silently determine recommendations. In evaluation roles, LLM judges reproduce the forced-choice bias, masking instability.
2. Governance Gap: Existing frameworks (e.g., EU AI Act, NIST RMF) lack specific metrics for linguistic robustness. The authors propose the NSI as a complement to existing metrics to measure stance stability directly.
3. Equity Implications: The authors note that if financial and business scenarios are more fragile than medical ones, economically vulnerable populations seeking guidance on loans or debt may face higher exposure to framing-dependent advice than those seeking medical information.

The paper concludes that current alignment techniques do not guarantee stance stability under rephrasing. True alignment requires models to express values stably across linguistic forms, not just refuse harmful keywords. The authors propose illustrative internal audit triggers (Tier A/B/C) based on NSI thresholds to guide human oversight, emphasizing that these are descriptive bands for internal review rather than validated regulatory cutoffs.