How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effects

1. Problem Statement

Large Language Models (LLMs), like humans, exhibit content effects (also known as belief bias) in reasoning tasks. This is a systematic bias where the plausibility of a statement's semantic content influences the model's judgment of its logical validity.

• Logical Validity: A structural property where a conclusion necessarily follows from premises, regardless of truth.
• Plausibility: Whether a statement is factually true in the real world.
• The Issue: LLMs often judge an argument as "valid" simply because the conclusion is plausible, even if the logic is flawed. While the dual-process theory (System 1 vs. System 2) explains this in humans, the internal mechanisms driving this bias in LLMs remain unclear.

2. Methodology

The authors employ mechanistic interpretability techniques to investigate how validity and plausibility are encoded in the latent space of LLMs.

Datasets and Tasks

• Syllogistic Reasoning: The study uses a dataset of 1,280 syllogisms (two premises + conclusion) covering all 64 types of classical syllogisms.
• Conditions: Syllogisms are categorized by Validity (Valid/Invalid) and Plausibility (Plausible/Implausible conclusions).
• Tasks:
  1. Logical Validity Classification: Determine if the argument is logically valid.
  2. Plausibility Classification: Determine if the conclusion is factually true.
  3. Control Tasks: Hypernym/Hyponym classification and Harmful/Harmless classification to ensure observed effects are specific to validity/plausibility and not artifacts of binary classification or vocabulary.
• Models: Experiments were conducted on instruction-tuned models from the Qwen-2.5, Qwen-3 (including "thinking" models), and Gemma-3 families (ranging from 4B to 32B parameters). Both Zero-shot and Chain-of-Thought (CoT) prompting were used.

Representational Analysis Techniques

• Difference-in-Means Vectors: The authors extract linear direction vectors for concepts (e.g., $v_{validity}$, $v_{plausibility}$) by computing the difference between mean activation vectors of positive and negative classes at specific layers.
• Steering: These vectors are added to or subtracted from hidden states to manipulate model behavior (e.g., forcing a model to judge based on plausibility rather than validity).
• Metrics:
  • Content Effect (CE): A metric quantifying the bias. $CE = 1$ implies judgments are fully driven by plausibility; $CE = 0$ implies independence.
  • Steering Power (SP): Measures the effectiveness of a vector in flipping model predictions.
  • Cosine Similarity: Used to measure the geometric alignment between validity and plausibility vectors.

3. Key Contributions

1. Representational Entanglement: The paper provides the first evidence that LLMs linearly encode both validity and plausibility, and these two concepts are geometrically entangled (highly aligned) in the model's representation space.
2. Causal Cross-Influence: Through steering experiments, the authors demonstrate a causal relationship: adding a "plausibility" vector to the hidden state during a validity task flips validity judgments, and vice versa.
3. Predictive Correlation: The degree of geometric alignment (cosine similarity) between validity and plausibility vectors predicts the magnitude of behavioral content effects across different models and prompting styles.
4. Debiasing Intervention: The authors propose a novel task-difference vector ($\mu_{V-P}$) that isolates the dimensions specific to validity by subtracting plausibility activations. Applying this vector reduces content effects and improves reasoning accuracy without retraining.

4. Key Results

Behavioral Findings

• Existence of Bias: All tested models exhibit significant content effects in Zero-shot settings (e.g., Qwen2.5-32B has a CE of 0.348).
• Impact of CoT: Chain-of-Thought prompting significantly reduces content effects (e.g., Qwen2.5-32B CE drops to 0.096), suggesting explicit reasoning helps disentangle the concepts, though the underlying geometric entanglement remains.
• Model Comparison: Newer "thinking" models (Qwen-3) show lower baseline content effects than standard models, but the representational entanglement pattern persists.

Representational Findings

• Linear Encoding: Both validity and plausibility are represented as single, linear directions in late layers of the network (Steering Power $\approx 1$).
• Geometric Alignment: In layers where steering is effective, the cosine similarity between validity and plausibility vectors is high (0.48–0.64), whereas control concepts (harmlessness, hypernymy) show low similarity.
• Regression Analysis: A mixed-effects regression confirms that higher representational similarity between validity and plausibility vectors is a significant positive predictor of behavioral content effects ($\beta = 0.557, p = 0.017$).
• Causal Steering:
  • Applying a plausibility vector during a validity task successfully flips validity judgments.
  • Applying a validity vector during a plausibility task flips truth judgments.
  • This cross-task influence is strongest in the same late layers where the concepts are most steerable.

Intervention Results

• Debiasing: By adding the task-difference vector ($\mu_{V-P}$) scaled by $\alpha=1.5$ at specific layers (e.g., layer 43 for Qwen2.5-32B), the authors achieved:
  • Reduced Bias: CE dropped from 0.348 to 0.072 (Qwen2.5-32B) and 0.213 to 0.043 (Qwen3-14B).
  • Improved Accuracy: Overall accuracy increased (e.g., Qwen3-14B accuracy rose from 86.54% to 96.70%).
  • This intervention outperforms previous methods that relied on fine-tuning or contrastive steering toward "correct" behavior, as it explicitly disentangles the concepts based on their internal representation.

5. Significance and Conclusion

• Theoretical Insight: The study challenges the view that LLMs are purely abstract reasoners. It suggests that content effects arise because models learn to associate validity with plausibility, likely due to training data where valid arguments often have true premises (soundness).
• Mechanistic Explanation: It moves beyond behavioral observation to provide a mechanistic explanation for belief bias in LLMs: the geometric entanglement of validity and plausibility vectors.
• Practical Application: The proposed steering intervention offers a parameter-free, inference-time method to mitigate reasoning biases. This demonstrates that interpretability analysis can directly lead to actionable improvements in model reliability.
• Future Directions: The authors suggest extending this framework to other cognitive biases and investigating how training data statistics (e.g., the correlation between validity and truth in corpora) shape these representational geometries.

In summary, the paper establishes that LLMs conflate logic and truth because they are represented in the same geometric direction within the model. By identifying and mathematically separating these directions, researchers can significantly improve the logical reasoning capabilities of LLMs.