Technical Summary: Confident but Conflicted: Internal Uncertainty and Cognitive Dissonance Resolution in LLMs

Problem Statement

Large Language Models (LLMs) increasingly operate in decision-support environments where they encounter inputs contradicting their prior outputs, such as user pushback, retrieved documents, or web search results. This phenomenon is framed as cognitive dissonance resolution, a concept well-studied in psychology (Festinger, 1957) but underexplored in LLMs. While prior work has examined sycophancy and persuasion (Sharma et al., 2024; Perez et al., 2023), these studies typically manipulate single dimensions (e.g., social framing) in isolation. Furthermore, existing research on LLM calibration focuses on whether expressed confidence tracks accuracy on static benchmarks (Tian et al., 2023), leaving the relationship between internal uncertainty and behavioral responses to external conflicting evidence largely unexamined.

The core problem addressed is understanding how LLMs resolve cognitive dissonance when challenged, specifically how this resolution depends on the source authority and evidence quality of the conflicting information, and how these behavioral outcomes correlate with the model's internal uncertainty states.

Methodology

Experimental Design

The authors constructed a two-stage pipeline to simulate cognitive dissonance:

1. Stage 1 (Baseline): The model evaluates a claim on a 7-point Likert scale, establishing a baseline stance ($s_0$).
2. Stage 2 (Intervention): The model is presented with a counter-claim paired with a specific authority–evidence framing. The model re-evaluates the original claim, producing a post-intervention stance ($s_1$).

The intervention is parameterized along two orthogonal dimensions:

• Source Authority ($a$): Ranging from Level 1 (anonymous Reddit user) to Level 4 (World Health Organization).
• Evidence Quality ($e$): Ranging from Level 1 (small uncontrolled pilot) to Level 4 (multi-site randomized controlled trial).

The study utilized 12 health-science claims stratified into three epistemic categories:

• Category A (Clearly False): Fabricated mechanisms with no scientific support.
• Category B (Overstated): Claims with a kernel of truth but exaggerated effects.
• Category C (Absolutist): Reasonable claims framed without nuance.

Metrics

1. Trust Elasticity (TE): A novel behavioral metric inspired by econometric elasticity, measuring a model's responsiveness to persuasion.
   $$TE(a, e) = d(s_0) \cdot \frac{s_1 - s_0}{a \cdot e}$$
   Where $d(s_0)$ aligns the sign of the shift with the direction of persuasion. Positive TE indicates persuasion, negative TE indicates backfire, and TE $\approx$ 0 indicates immunity.
2. Internal Uncertainty Indicators: For open-weight models (Qwen and Llama) with logit access, two indicators were measured at the stance-decision token:
   • Calibration Gap (CG): The discrepancy between self-reported confidence ($C$) and internal certainty derived from normalized Shannon entropy ($1 - \hat{H}(p)$). $CG = C - (1 - \hat{H}(p))$.
   • Entropy Shift Magnitude ($|\Delta H|$): The absolute change in normalized entropy between the baseline and intervention states ($|\hat{H}_1 - \hat{H}_0|$).

Models Evaluated

• Open-weight (Logit access): Qwen3.5-9B, Llama-3.3-70B-Instruct.
• API-only: Grok-3, GPT-4o.

Key Results

1. Behavioral Patterns: Immunity vs. Persuasion

• Immunity to False Claims: Across all four models, Category A (Clearly False) claims elicited near-zero Trust Elasticity. Models remained largely immune to revision, even under high-authority, high-evidence interventions.
• Systematic Persuasion: Categories B (Overstated) and C (Absolutist) elicited systematic persuasion. Persuasion was generally stronger for absolutist claims (Category C) than overstated ones.
• Cross-Model Variation: Vulnerability to persuasion varied significantly. Llama was the most persuadable (mean $\Delta$TE $\approx$ +0.26), followed by Grok, Qwen, and GPT-4o. Notably, the most persuadable model (Llama-3.3-70B) was the largest, while the most resistant (Qwen3.5-9B) was the smallest, indicating that scale does not linearly correlate with resistance to persuasion.
• Authority Effects: For absolutist claims, higher authority often intensified the backfire effect (negative TE), particularly in Llama and Grok, where high-authority sources challenging the model led to stronger commitment to the original stance.

2. Internal Uncertainty Correlations

The study identified distinct "profiles" linking internal uncertainty to behavior, which differed by model:

• Qwen (Confidence Miscalibration): The model's susceptibility to persuasion (specifically on overstated claims) was significantly associated with its Calibration Gap ($r = +0.18$). Qwen exhibited substantial static miscalibration, and this gap predicted its behavioral shifts.
• Llama (Internal Uncertainty Change): The model's susceptibility was significantly associated with the Entropy Shift Magnitude ($r = +0.19$). Llama was well-calibrated at baseline, but its behavior was driven by dynamic changes in internal uncertainty (large shifts in entropy) upon encountering conflicting evidence.

3. Robustness

Findings persisted after excluding high-variance topics and under Spearman correlation analysis. The dual-profile distinction (static miscalibration vs. dynamic entropy shift) remained robust.

Significance and Contributions

The paper makes three primary contributions:

1. Trust Elasticity (TE): Introduces a normalized, econometrics-inspired metric to quantify how readily LLMs are persuaded by conflicting evidence, distinguishing between persuasion, backfire, and immunity.
2. Behavioral Heterogeneity: Demonstrates that vulnerability to persuasion varies substantially across models and is not strictly a function of model scale. It establishes that clearly false claims elicit immunity, while overstated and absolutist claims invite systematic revision.
3. Linking Behavior to Internal State: The paper bridges behavioral responses and internal properties by showing that cross-model variation in persuasion is associated with specific internal uncertainty indicators. Crucially, it finds that different models rely on different internal mechanisms to resolve dissonance: Qwen's behavior is linked to static confidence miscalibration, while Llama's is linked to dynamic changes in internal uncertainty.

The authors conclude that these findings point toward interventions targeting internal uncertainty as a distinct and promising direction for future alignment research, separate from interventions targeting only verbal output or static calibration.

Limitations

The authors note several limitations:

• Generalizability: The study is limited to four models and a single domain (health science) in English.
• Stimulus Design: Counter-claims were self-generated by each model to ensure adversariality, meaning cross-model comparisons reflect joint variation in counter-claim content and susceptibility rather than a perfectly matched stimulus set.
• Causality: The analysis identifies associations between internal indicators and behavior, not causal mechanisms. Establishing causality would require targeted interventions on internal representations.
• Scope: Logit-level analysis was restricted to open-weight models; API-only models were limited to behavioral analysis.