Technical Summary: Bias by Necessity

Problem Statement

The paper addresses a fundamental question in the intersection of artificial intelligence and cognitive science: Are certain cognitive biases (specifically primacy effects, anchoring, and order-dependence) mathematically inevitable consequences of sequential information processing, or are they merely artifacts of current model training and architecture? While extensive empirical work has documented these biases in both humans and Large Language Models (LLMs), and various debiasing techniques (e.g., Chain-of-Thought, reflection prompting) have been proposed, these methods reduce but never eliminate the biases. The authors posit that these biases may be "architecturally necessary" due to the causal masking constraints inherent in autoregressive processing.

Methodology

The research employs a hybrid methodology combining formal theoretical proofs, large-scale empirical validation across frontier models, and pre-registered human behavioral experiments.

1. Theoretical Framework: The authors formalize the constraints of autoregressive models using the concept of "causal masking," where a token $x_i$ can only attend to preceding tokens $x_1, \dots, x_{i-1}$. They define Positional Privilege ($\Phi(j)$) as the total expected attention weight a position receives from all subsequent positions.
2. Impossibility Theorems: Three theorems are derived to prove that under standard assumptions (non-trivial attention, causal masking, and content-position interaction), specific biases cannot be eliminated without violating architectural constraints or incurring prohibitive computational costs.
3. LLM Empirical Validation: The framework was tested across 12 frontier LLMs (including GPT-4, Claude-3, Gemini-1.5, Llama-3, and others). The study utilized a classic anchoring paradigm with 200 trials per model, manipulating the position of irrelevant anchors within an 8-sentence context to trace the positional privilege curve. Model comparison was performed using Bayesian Information Criterion (BIC).
4. Human Behavioral Experiments: Two pre-registered studies ($N=464$ analyzed) tested quantitative predictions derived from the formal model:
   • Study 1: Tested if anchor position (before vs. after target information) modulates anchoring magnitude.
   • Study 2: Tested if working memory (WM) load amplifies primacy bias and if individual WM capacity predicts bias reduction.

Key Contributions

The paper makes five primary contributions:

1. Impossibility Theorems: Proving that primacy bias, anchoring, and order-dependence are mathematically inevitable in autoregressive models due to causal masking.
2. Quantitative Bounds: Establishing that exact debiasing via permutation marginalization requires $\Omega(n!)$ (factorial-time) computation, rendering practical elimination computationally infeasible.
3. Empirical Validation: Validating the theoretical framework across 12 diverse LLMs, demonstrating that an exponential decay model of positional privilege fits the data significantly better than linear, power-law, or null models ($R^2 = 0.89$; $\Delta BIC = 16.6$ vs. next-best).
4. Cross-Domain Prediction: Deriving quantitative predictions for human cognition based on the formal model and confirming them in human experiments.
5. Resource-Rational Reframing: Framing cognitive biases not as errors, but as optimal, resource-rational responses to sequential processing constraints shared by both humans and machines.

Results

Theoretical Results

• Theorem 1 (Primacy Bias Inevitability): For any autoregressive model satisfying causal masking, there exist input permutations where the output distribution differs. The first token possesses a structural advantage (positional privilege) that cannot be nullified without violating causal constraints.
• Theorem 2 (Anchoring Emergence): Anchoring bias emerges inevitably because the hidden state at the query position must mathematically contain a non-zero quantity of information from the preceding anchor, regardless of the anchor's normative relevance.
• Theorem 3 (Debiasing Cost): Exact elimination of bias requires evaluating all $n!$ permutations of the input. Monte Carlo approximation is feasible but incurs a constant per-tolerance overhead, meaning perfect debiasing is computationally intractable.

LLM Empirical Results

• Model Fit: The exponential decay model of positional privilege strongly predicted anchoring bias across 12 models ($R^2 = 0.89$).
• Positional Effect: Anchoring magnitude decreased as the anchor position moved later in the sequence, following the predicted harmonic/exponential decay of positional privilege.
• Generalization: The model generalized across model families (leave-one-out cross-validation $R^2 = 0.84$) and successfully predicted out-of-sample results for Mistral-Large.

Human Experimental Results

• Study 1 (Anchor Position): Confirmed that anchors presented before target information produced significantly stronger anchoring than those presented after ($d = 0.52$, $BF_{10} = 847$). This aligns with the predicted range of $d \in [0.35, 0.55]$.
• Study 2 (WM Load): High working memory load amplified the primacy effect ($d = 0.41$). Individual differences analysis showed that higher WM capacity predicted smaller position effects ($r = -0.38$), supporting the hypothesis that primacy bias is a resource-rational response to limited sequential processing capacity.

Significance and Claims

The paper claims to reframe cognitive biases as necessary consequences of sequential processing rather than correctable defects.

• Computational-Level Parallel: The authors argue that causal masking in transformers and serial processing constraints in human working memory represent analogous constraints at Marr's computational level. Both systems process information sequentially with limited access to future inputs, leading to similar bias structures.
• Resource Rationality: Drawing on resource-rational analysis, the paper suggests that both human and machine biases are optimal strategies given finite computational resources. The "bias" emerges from the structure of the problem (sequential processing), not the structure of the solver.
• Implications for AI Design: The findings suggest that pursuing exact debiasing is computationally futile. Instead, the paper advocates for human-AI collaboration strategies that leverage complementary architectural constraints (e.g., pairing autoregressive models with bidirectional encoders or retrieval systems) to achieve robust joint decisions.
• Evaluation Methodology: The paper asserts that position bias in "LLM-as-judge" evaluations is a fundamental architectural property, not a bug. Evaluation protocols should therefore randomize presentation orders and aggregate across permutations rather than expecting models to learn to ignore position.

The authors maintain modesty regarding mechanistic equivalence, noting that while the computational constraints are parallel, the underlying implementations (neural vs. artificial) differ. They acknowledge limitations regarding the WEIRD nature of human samples and the need to test alternative architectures like state-space models (e.g., Mamba).