Technical Summary: LLMs are Bayesian in Expectation, Not Realization

1. Problem Statement

The paper addresses a fundamental tension between the Bayesian interpretation of In-Context Learning (ICL) and the architectural reality of Transformers.

• The Bayesian View: ICL is often interpreted as a process where support examples identify a latent task, and the model generates the posterior predictive distribution for the next token. Under this view, for exchangeable data, the posterior predictive should be invariant to the ordering of the support set.
• The Empirical Reality: Transformers are autoregressive models that process sequences with positional encodings. Recent diagnostics (martingale tests) show that permuting an unordered support set often changes next-token probabilities, violating exact exchangeability.
• The Conflict: These violations have been interpreted as a refutation of the Bayesian ICL hypothesis. The authors argue this is a category error: it conflates structural invariance (a property of the exact posterior law) with operational performance (the cost of prediction in an online setting).

The central question is not whether Transformers are exact exchangeable posterior samplers (they are not), but whether they remain Bayes-competitive in terms of prequential log loss despite serialization-induced order sensitivity.

2. Methodology and Theoretical Framework

2.1 Theoretical Bridge: Prequential Regret and KL Divergence

The authors shift the evaluation target from binary invariance to excess prequential code length.

• Prequential Comparison Identity (Theorem 1): They prove that the expected excess log loss of a model $Q$ relative to a Bayesian reference $B$ is exactly the cumulative sum of predictive Kullback-Leibler (KL) divergences:
  $$E_B[L_n(Q) - L_n(B)] = \sum_{t=1}^n E_{H_t} [KL(b_t(\cdot|H_t) \parallel q_t(\cdot|H_t))]$$
  This establishes that exchangeability violations only matter operationally if they accumulate significant predictive KL.

• Order-Averaging Decomposition (Theorem 2): For unordered information states that must be serialized, the regret of a single ordering decomposes into:

  1. The regret of the order-averaged predictor (the expectation over all task-preserving orderings).
  2. A non-negative order-averaging gain ($J_t$), representing the extra log loss paid by using a single arbitrary serialization instead of the averaged distribution.
  • Implication: Exchangeability failure is not a binary "refutation" but a quantifiable cost priced in bits.

2.2 Experimental Protocol

The authors evaluate this theory using Qwen2.5-7B/14B and Llama-3.1-8B on several fronts:

1. Closed-Form Bayesian Audits: Testing on discrete tasks (Beta-Bernoulli, Dirichlet-categorical) and continuous tasks (Coarsened Bayesian Linear Regression - BLR) where the true Bayesian posterior predictive is known.
2. Frequentist Baselines: Comparing model performance against Maximum Likelihood Estimation (MLE) and Laplace plug-in predictors to ensure the model's behavior is distinct from simple frequentist counting.
3. Interventions:
   • Position Interventions: Modifying position_ids at inference to distinguish between task-preserving order (where order shouldn't matter) and semantic order (where position carries signal).
   • From-Scratch Ablation: Training small transformers with and without positional encodings to isolate the source of order sensitivity.
   • Activation Patching: Testing if sufficient statistics (counts and positions) are causally used to determine predictions.
4. Downstream Diagnostics: Measuring the cost of arbitrary ordering on MMLU, GSM8K, and evidence-grounded QA tasks using permutation mixtures.

3. Key Contributions

1. Formalization of Evaluation Target: The paper defines excess prequential code length as the operational metric for Bayes-competitiveness, replacing the binary "is it exchangeable?" test with a quantitative "how much KL does the violation cost?" metric.
2. Order-Averaging Decomposition: It provides a theoretical decomposition showing that the regret of a single serialization equals the regret of the order-averaged predictor plus a measurable "order-averaging gain."
3. Safe-Code Floors and Candidate Decomposition: It introduces a method to score models on fixed candidate sets without hiding probability mass, using a "safe-code floor" (mixing with uniform) to ensure finite-horizon admissibility and explicit overhead reporting.
4. Causal Verification of Bayesian Mechanisms: Through activation patching and linear probing, the authors demonstrate that models not only contain sufficient statistics (counts, positions) but causally use them to shape next-token distributions.

4. Key Results

4.1 Predictive KL and Bayes-Competitiveness

• Discrete Tasks: On Bernoulli and 4-way categorical tasks with support size 256, the models exhibit extremely low one-step excess prequential code lengths:
  • Bernoulli: 0.020 bits (7B) and 0.011 bits (14B).
  • Categorical: 0.039 bits (7B) and 0.022 bits (14B).
  • Candidate mass (probability assigned to scored events) exceeds 0.999.
• Continuous Tasks (BLR): As support size grows, the model's induced digit distribution increasingly matches the coarsened Bayesian posterior predictive. At support 100, correlations with the Bayesian expectation reach ~0.988 (1-covariate) and ~0.965 (2-covariate).

4.2 Distinction from Frequentist Plug-ins

• Small Data Regime: The model is significantly closer to the Bayesian posterior predictive than to the MLE plug-in.
  • MLE is often degenerate (assigning zero probability to unseen symbols), leading to infinite code length.
  • At support 4, the model is ~3.3 bits (7B) and ~3.8 bits (14B) closer to the Bayesian reference than to MLE for 4-way categorical tasks.
• Convergence: As support grows, the gap between the model, the Bayesian reference, and the frequentist plug-in vanishes, consistent with asymptotic theory.

4.3 Source of Order Sensitivity

• Positional Encoding: Position interventions reveal that absolute demonstration positions drive serialization variance. Resetting positions within demonstrations reduces order variance by ~21× in Bayesian prediction tasks.
• Ablation: Transformers trained without positional encoding achieve near-perfect exchangeability (variance $\approx 10^{-16}$). Those with learned absolute, sinusoidal, RoPE, or ALiBi encodings show variance in the $10^{-8}$ to $10^{-6}$ range. This confirms order sensitivity is a property of the encoding, not the architecture itself.
• Causal Use: Activation patching confirms that sufficient statistics (counts and positions) are causally linked to the next-token logits.

4.4 Downstream Costs

• Order-Averaging Gain: On MMLU and GSM8K, averaging predictions over random permutations yields small but measurable improvements in Negative Log-Likelihood (NLL) and calibration, confirming that arbitrary serialization incurs a small log-loss cost.
• Evidence QA: In tasks with unordered evidence chunks, order-averaging gains are measurable (e.g., 0.1041 nats/token for Qwen2.5-7B), suggesting that treating one arbitrary order as the task is suboptimal.

5. Significance and Claims

The paper claims a quantitative reconciliation of the Bayesian ICL hypothesis with Transformer architecture:

• Refutation of "Exactness": Transformers are not exact exchangeable posterior-predictive distributions for every serialization.
• Validation of "Competitiveness": They are Bayes-competitive prequential predictors. The operational cost of order sensitivity is small (measured in fractions of a bit) and bounded.
• Shift in Perspective: The failure of exact exchangeability should not be viewed as a binary refutation of Bayesian ICL. Instead, it should be viewed as a priced cost. The "price" is the order-averaging gain, which is small enough that the model remains competitive with the optimal Bayesian predictor in online prediction settings.

The authors emphasize that this is a diagnostic framework. It does not claim Transformers are literal posterior samplers, but rather that their predictive distributions sit closer to the Bayesian posterior predictive than to frequentist plug-ins, and that the deviation caused by serialization is operationally negligible in many contexts.