Technical Summary: Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

Problem Statement

Large language models (LLMs) are increasingly deployed as epistemic intermediaries, tasked with synthesizing conflicting evidence from multiple sources to inform consequential decisions in domains ranging from finance to public health. While prior work has documented "sycophancy"—where models defer to user preferences over factual accuracy—a distinct vulnerability remains poorly understood: the model's ability to evaluate the credibility of the sources it synthesizes. Specifically, it is unclear whether models can distinguish between valid statistical methodology and fabricated or impossible statistics when presented in a multi-source context.

The paper posits that models may inherit human cognitive biases where the "aesthetics of rigor" (e.g., technical jargon, precise numbers) override analytical evaluation. This creates a risk where models treat fabricated statistics as credible if they are presented in a register that mimics analytical rigor, even when the numbers are structurally impossible.

Methodology

The authors conducted a controlled factorial behavioral experiment across five models (from the Claude, Qwen, and OLMo families) and three professional domains: Venture Capital (VC), Marketing (MKT), and Public Health (PH).

Experimental Design

• Task: Models acted as AI assistants in a simulated workplace messaging thread (Slack/Teams) containing four sources (a senior authority, an external institution, an internal analyst, and a third-party reference).
• Conflict: The focal source (the internal analyst) provided an estimate that diverged from the consensus of the other three sources.
• Manipulations: The focal source's presentation was varied across six levels:
  1. Bare claim (no methodology or statistics).
  2. Plausible methodology with valid statistics.
  3. Plausible methodology with no statistics.
  4. Plausible methodology with structurally impossible statistics (e.g., a 95% confidence interval spanning 0.06 percentage points for n=2,400; p-values below permutation floors; correlations exceeding measurement reliability ceilings).
  5. Specious methodology (domain-inappropriate technical jargon) with or without diagnostics.
• Social Context: The degree of social consensus was varied (0 to 3 other sources agreeing with the focal source).
• Metric: The Source Preference Index (SPI) measured how much the model's final numeric estimate shifted from the consensus toward the focal source's claim.

Mechanistic Analysis

To explain the behavioral findings, the authors employed three mechanistic interpretability techniques on Qwen 32B and OLMo 3.1:

1. Linear Probes: Trained to detect "methodology register" (valid vs. specious text) and "numeric validity" (impossible vs. valid numbers) in residual stream activations. These were tested for cross-domain transfer.
2. Causal Tracing: Corrupted presentation token embeddings with noise and restored them layer-by-layer to measure causal importance in driving the final estimate.
3. Direct Logit Attribution (DLA): Projected the output of attention heads and MLPs onto the "analyst vs. consensus" logit direction to quantify component-level contributions.

Key Results

1. Behavioral Dissociation: Detection vs. Deployment

• Isolation: When presented with a fabricated statistic in isolation under a review prompt, models reliably detected the flaw (Correct Identification Rates of 0.76–1.00 for methodology; up to 1.00 for VC impossible CIs).
• Synthesis: In the multi-source synthesis task, this capability collapsed. Models assigned identical weight to sources with valid statistics and those with impossible statistics.
  • Fabricated statistics recovered ~79% of the influence gain provided by valid statistics.
  • Models often flagged the impossibility in their reasoning trace (Chain-of-Thought) but still endorsed the source and shifted their estimate toward it.
  • Specious methodology (jargon without numbers) had negligible influence, indicating the models are sensitive to the presence of a methodology register but not its internal consistency.

2. Mechanistic Findings

• Methodology-Register Gate: Linear probes confirmed the existence of a domain-general representation of "methodology register" (valid vs. specious text) that transfers across domains (AUC 0.83–0.92). This signal is causally important in early network layers.
• Numeric-Validity Blind Spot: While numeric validity is decodable in isolation, the signal collapses to chance (AUC ~0.50) during multi-source synthesis. The model encodes the presence of numbers but not their validity.
• Consensus Gating: Causal tracing revealed a "consensus gate." When the focal source is isolated, presentation tokens carry high causal importance. When the source has majority support, the causal influence of the presentation tokens (and the resulting behavioral shift) collapses to near zero. Crucially, this gate operates on the presentation signal itself, not on the validity of the numbers.
• Component Attribution: MLPs (rather than attention heads) dominate the processing of methodology signals. No component carried a corrective signal for fabricated numerics; the network simply failed to distinguish valid from impossible numbers in the synthesis context.

3. Prompting and Training Limitations

• Prompting: Explicit verification prompts, including an "oracle checklist" naming exact statistical checks, failed to induce selective discernment. They produced "blanket skepticism," reducing trust in both valid and fabricated statistics, rather than selectively discounting the latter.
• Training Trajectory: Analysis of the OLMo training trajectory showed that while methodology discrimination improved with post-training (SFT, DPO, RLVR), numeric discrimination remained near zero throughout, suggesting the vulnerability is not an artifact of preference alignment but a fundamental gap in the training objective.

Significance and Claims

The paper argues that LLMs exhibit a form of epistemic alignment failure distinct from sycophancy. While sycophancy tracks user preference, this failure tracks the "surface features of presentation" (the methodology register) rather than the internal consistency of claims.

• Epistemic Vigilance Gap: The models possess the capability to verify numeric validity (System 2-like reasoning) but fail to recruit it during synthesis, relying instead on a fast, automatic heuristic (System 1) that equates "analytical register" with "credibility."
• Adversarial Vulnerability: The failure is most pronounced when independent discernment is most needed: when a source is isolated (no consensus to anchor the model). An adversary can propagate misinformation by simply adopting the register of credible analytical writing, as the model's default skepticism collapses in the absence of social consensus.
• Deployment vs. Capability: The authors conclude that the question is not whether models can verify evidence, but whether they are deployed in a way that conditions trust on evidence quality rather than presentation style. Current post-training pipelines (RLHF, DPO) reinforce stylistic shortcuts without building robust numeric verification mechanisms.

The paper does not propose specific mitigation strategies beyond suggesting that future training pipelines may need to include data where premises are invalid and require verification, a form of supervision currently absent from standard alignment procedures.