🤖 machine learning

From Drift to Coherence: Stabilizing Beliefs in LLMs

This paper demonstrates that while large language models initially exhibit belief drift violating the martingale property during autoregressive answer resampling, they eventually self-stabilize into coherent predictive distributions, a phenomenon leveraged to propose prompting and fine-tuning strategies that reduce drift and improve coherence without sacrificing accuracy.

Original authors: SongEun Kim, Seungyoo Lee, Edwin Fong, Hyungi Lee, Juho Lee

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: SongEun Kim, Seungyoo Lee, Edwin Fong, Hyungi Lee, Juho Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot (a Large Language Model or LLM) that you ask a question. Usually, you expect it to give you an answer and stick to its reasoning. But this paper discovers something strange: if you ask the robot to answer the same question over and over again in a row, its confidence wobbles.

Here is the story of how the authors fixed that wobble, explained simply.

The Problem: The Robot's "Wobbly Brain"

Think of the robot's brain like a person trying to guess the weather.

The Ideal: If the robot were a perfect Bayesian thinker (a fancy term for a perfect probabilistic guesser), its confidence should be stable. If it thinks there's a 70% chance of rain, that 70% shouldn't change just because it thought about it a second time.
The Reality: The authors found that when they asked the robot to generate a sequence of answers to the same question, its internal confidence drifted.
- Analogy: Imagine you ask a friend, "Is it going to rain?" They say, "I'm 70% sure." Then, without looking outside again, you ask them again immediately. They might suddenly say, "Actually, I'm only 50% sure." Then you ask again, and they say, "Wait, maybe 80%!"
- This "wobbling" means the robot's beliefs are unstable in the short term. It's like a compass that spins wildly before finally pointing North.

The Discovery: It Eventually Settles Down

The researchers noticed something interesting: if you kept asking the robot to answer the same question many times (like 20 or 30 times in a row), the wobble eventually stopped. The robot's confidence would stabilize and settle into a consistent number.

The Metaphor: It's like shaking a jar of mixed-up marbles. At first, the colors are swirling chaotically. But if you let the jar sit, the marbles settle, and you can finally see the true ratio of colors. The robot needs a "settling period" (called a burn-in phase) to find its true belief.

The Solution: Two Tricks to Stop the Wobble

The problem is that waiting for the robot to settle down takes a long time and uses up a lot of computer power. The authors came up with two ways to make the robot stable immediately.

1. The "Warm-Up" Trick (Answer Seeding)

Instead of asking the robot to start cold, the authors give it a "warm-up" list of answers first.

How it works: They ask the robot to generate a few random answers to the question once, and then they feed those answers back to the robot as part of the prompt before asking it to start the real sequence.
The Analogy: It's like a musician tuning their instrument before a concert. Instead of starting the song with a screechy, out-of-tune note, they play a few quick notes to get the pitch right. The robot uses these "seed" answers to jump straight into the stable zone, skipping the chaotic wobble at the start.

2. The "Training" Trick (Self-Consistency Loss)

The authors also taught the robot to be stable by changing how it learns.

How it works: They created a special math lesson (a "loss function") where they showed the robot its own future answers. They told the robot: "Your answer right now should match what you will say in 5 steps."
The Analogy: Imagine a student who usually gets nervous and changes their mind during a test. The teacher gives them a rule: "Whatever you write in question #1, you must be able to stand by when you get to question #10." By practicing this rule, the student learns to be consistent from the very first second, so they don't need to "wobble" to find their confidence later.

The Results

When they tested these tricks on standard multiple-choice questions (like common sense or science quizzes):

Less Wobble: The robot's confidence stopped drifting and stayed consistent.
Better Guessing: The robot became better at knowing how sure it was about its answers (calibration).
Same Accuracy: Crucially, making the robot more stable didn't make it dumber. It still got the right answers just as often as before, but now it was more reliable about when it was right.

The Bottom Line

The paper shows that while AI models often start with a "wobbly" belief system when generating sequences, they naturally want to be consistent. By using a little bit of "warm-up" data or by training them to match their future selves, we can make them stable and reliable immediately, without needing to wait for them to settle down on their own.

Technical Summary: From Drift to Coherence: Stabilizing Beliefs in LLMs

Problem Formulation
Large Language Models (LLMs) are frequently hypothesized to perform implicit Bayesian inference, particularly in the context of in-context learning (ICL). A critical condition for such Bayesian coherence is the martingale property of predictive beliefs: the expectation of future beliefs, conditioned on current history, should equal the current belief. Previous work by Falck et al. (2024) demonstrated that LLMs violate this property in controlled synthetic settings (e.g., random number generation), suggesting they are not generally Bayesian. However, these synthetic tasks obscure the model's actual predictive distribution and may not reflect typical usage regimes.

This paper investigates whether LLMs exhibit martingale violations in a more practical setting: generic multiple-choice question answering (MCQA). The authors define the predictive process as a sequence of answers generated autoregressively for a fixed query. They observe that while LLMs eventually stabilize, they exhibit a significant initial belief drift where the predictive distribution shifts systematically during early sampling steps, violating the martingale property. This drift implies that the model's internal belief state is not immediately coherent, requiring a "burn-in" period to reach a stable limiting distribution.

Methodology
The authors propose a framework to analyze and correct this drift using the discrete nature of MCQA answer spaces, which allows for the exact computation of predictive distributions without sampling approximations.

Prompted Predictive Resampling (PPR): The core experimental protocol involves instructing an LLM to generate a sequence of answers to the same question repeatedly. This creates a history-dependent predictive process $\theta_n(x_Q)$ , where the model conditions on its own previous outputs.
Answer Seeding: To mitigate early-stage drift, the authors introduce a prompting strategy where the model is first provided with a short sequence of "seed" answers sampled i.i.d. from the standard model distribution (without resampling instructions). These seeds act as a "mean-field" summary of the model's uncertainty, steering the generative process into a stable regime more rapidly.
Martingale Amortization (Self-Consistency Loss): To eliminate the computational cost of long resampling sequences or external seeding at inference time, the authors propose a fine-tuning objective called Self-Consistency (SC) loss. This loss minimizes the divergence between the model's short-horizon predictions and its long-horizon predictions (approximated via Monte Carlo rollouts). By training the model to match its own future marginal statistics, the SC loss "amortizes" the stabilization process, enabling the model to output a coherent belief distribution immediately (zero-shot) without requiring a burn-in phase.

Key Contributions

Empirical Characterization of Drift: The paper provides empirical evidence that while LLMs violate the martingale property in the initial steps of predictive resampling, they spontaneously stabilize after a sufficient number of steps, converging to a coherent predictive distribution (a martingale posterior).
Stabilization Mechanisms: The authors introduce two methods to achieve coherence:
- Inference-time: Answer seeding, which reduces the burn-in period by providing stochastic context.
- Training-time: The SC loss, which teaches the model to internalize stability, effectively collapsing the convergence horizon.
Theoretical Justification: The paper establishes that minimizing the SC loss ensures the predictive process approaches an approximate martingale, provided the model class is sufficiently expressive. Theoretical bounds show that the residual martingale violation is proportional to the SC loss value.

Results
Experiments were conducted on standard MCQA benchmarks (CSQA, AI2-ARC, TinyMMLU) using instruction-tuned models (LLAMA-3.1-8B and OLMO-3-7B).

Drift Reduction: Both answer seeding and SC loss training significantly reduced the Expected Martingale Drift (EMD). The SC loss (AM-PPR) consistently outperformed standard PPR, achieving lower drift values across all datasets and models.
Calibration and Accuracy: The stabilized predictive distributions demonstrated improved uncertainty calibration (lower Expected Calibration Error and Brier Score) compared to standard direct-query baselines and un-stabilized PPR. Crucially, these improvements in coherence and calibration were achieved without sacrificing predictive accuracy.
Efficiency: The AM-PPR method allowed the model to access its stable belief state immediately (using the first token's distribution), eliminating the need for computationally expensive long rollouts or auxiliary seed generation during inference.

Significance and Claims
The paper claims that practical LLMs exhibit a transient belief drift that can be substantially reduced through specific prompting and training strategies. By operationalizing martingale-style coherence, the authors demonstrate that it is possible to improve the reliability and uncertainty calibration of LLMs in high-stakes reasoning tasks without compromising accuracy.

The authors modestly note that reducing martingale violations is a necessary but not sufficient condition for fully Bayesian, well-calibrated inference; the stabilized beliefs remain internally consistent but do not guarantee factual correctness or alignment with ground truth. Furthermore, the method currently relies on prompt engineering to elicit sequential resampling behavior. The paper concludes that while the framework was validated on discrete MCQA, it is inherently extensible to richer reasoning trajectories (e.g., Chain-of-Thought), suggesting a promising avenue for future research into stabilizing stochastic thinking paths in complex reasoning scenarios.