📈 economics

Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

This paper analyzes the performance of Thompson Sampling under model misspecification by classifying posterior evolution into distinct regimes within a two-armed Gaussian bandit and establishing a unified stochastic stability framework for general finite model classes to characterize limiting beliefs and asymptotic regret.

Original authors: Xinyu Dai, Daniel Chen, Yian Qian

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinyu Dai, Daniel Chen, Yian Qian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to navigate a maze, but you've given it a map that is slightly wrong. Maybe the map says a wall is made of glass when it's actually brick, or it thinks a shortcut leads to the exit when it actually leads to a dead end. This is the world of "misspecified learning." In science and economics, we often assume that if we give a smart system enough data, it will eventually figure out the truth and stop making mistakes. This idea relies on the concept of "Bayesian learning," where a system updates its beliefs based on new evidence, like a detective gathering clues to solve a case. Usually, we expect that with enough clues, the detective will point to the one true suspect. But what happens if the detective is using a flawed theory about how the world works? Does the robot eventually learn the right path, or does it get stuck in a loop of confusion? This question matters because today, everything from online shopping algorithms to government policy decisions relies on these learning systems. If they get stuck in a loop, the consequences could be expensive prices, bad recommendations, or ineffective laws.

This paper, written by researchers Xinyu Dai, Daniel Chen, and Yian Qian, dives deep into what happens when a learning system uses a specific, popular strategy called "Thompson Sampling" while its internal map is wrong. Thompson Sampling is a clever way for a robot to learn: instead of just picking the option it thinks is best right now, it occasionally tries a different option just to see what happens. It's like a chef who usually cooks their favorite dish but occasionally tries a new recipe just to keep their skills sharp. The authors wanted to know: if the chef's recipe book is full of errors, does this "tasting" behavior help them eventually find the truth, or does it trap them in a weird, endless cycle?

The researchers found that the answer depends entirely on how the wrong recipes interact with the real ingredients. They discovered three main scenarios. First, there is the "Self-Confirming" trap. Imagine the robot believes a high price is best, and it keeps charging high prices. If the real world happens to look good at high prices (even for the wrong reason), the robot gets confident and never changes its mind. It locks into a single strategy forever, which might be the right one, or it might be a permanent mistake. Second, there is the "Uniform Dominance" scenario, where one of the robot's wrong models is just clearly better than the others at explaining everything. In this case, the robot eventually figures out which model is the "least wrong" and sticks with it, converging to a stable decision.

But the most surprising discovery is the third scenario: the "Self-Defeating" loop. This happens when the robot's wrong models are so tricky that every time it tries to prove one is right, the results actually prove it wrong. For example, if the robot thinks a high price is best, it charges high prices. But the data from those high prices makes the robot think, "Wait, maybe a low price is better!" So it switches to a low price. But then the data from the low price makes it think, "No, high price was better!" The robot ends up oscillating back and forth forever. The authors show that in this specific setup, the robot never settles down. Even with infinite amounts of data, it never stops guessing. Instead of finding a single answer, its beliefs settle into a permanent, rhythmic dance of uncertainty.

The paper proves mathematically that this "Self-Defeating" behavior isn't just a glitch; it's a stable state where the system keeps exploring forever. This is a big deal because it challenges the old idea that "more data always leads to certainty." The authors show that if the learning algorithm is designed to keep experimenting (like Thompson Sampling does), and the world is misunderstood in a specific way, the system will never stop fluctuating. It will keep changing its mind, leading to constant changes in behavior—like a company that keeps changing its prices up and down forever, not because the market is changing, but because its learning algorithm is stuck in a loop of self-doubt. The researchers also extended this idea to situations with many more models, showing that while these loops can happen, they often get "pruned" down to simpler loops or single choices as the system gets more complex, unless the conditions are just right to keep the chaos alive.

In short, this paper tells us that being "smart" and "curious" isn't always enough to find the truth. If your starting assumptions are wrong in a specific way, your curiosity can actually prevent you from ever settling on a decision. The robot might never stop trying new things, not because it's learning, but because the very act of learning keeps pushing it away from the answer. This suggests that for systems making real-world decisions, we need to be careful about how we design their learning rules, because sometimes, the best way to learn might be to stop guessing and start trusting a simpler, more stable approach.

Technical Summary: Dynamic Decision-Making under Model Misspecification

Problem Statement
This paper investigates the long-run consequences of dynamic decision-making when agents rely on online learning algorithms (specifically Thompson Sampling) within misspecified model classes. In many economic and algorithmic settings—such as firms learning demand, platforms optimizing engagement, or policymakers evaluating treatments—the decision maker's actions shape the data available for learning. When the agent's parametric model of the environment is misspecified (i.e., the true data-generating process is not contained within the candidate model set), the interaction between endogenous data generation and incorrect beliefs can produce complex, non-convergent dynamics.

While existing literature characterizes convergence to stable outcomes like Berk-Nash equilibria or pseudo-true parameters under deterministic best-response rules, the behavior of posterior-sampling algorithms (which induce endogenous exploration) under misspecification remains poorly understood. The central question is: What are the long-run belief dynamics induced by Thompson Sampling under misspecification, and how do these dynamics affect action frequencies and regret?

Methodology
The authors employ a stochastic stability framework to analyze the posterior belief process as a Markov chain on the belief simplex. The analysis proceeds in two stages:

Two-Arm, Two-Model Gaussian Bandit: The paper first provides a complete classification of posterior dynamics in the simplest nontrivial environment. The agent chooses between two arms, observing rewards drawn from a true distribution $g$ , while believing the rewards follow one of two misspecified Gaussian models ( $\nu$ or $\gamma$ ). The dynamics are governed by the log-odds process $S_t = \log(\pi_t / (1-\pi_t))$ . The authors characterize the asymptotic behavior based on two primitives:
- Model Agreement: Whether the two models recommend the same optimal action.
- Log-Likelihood Drift ( $\Delta_i$ ): The expected log-likelihood ratio increment under each action $i$ , defined as $\Delta_i = E[\log(f_\nu/f_\gamma) | A_t=i]$ . This measures which model is closer to the true data-generating process in Kullback-Leibler (KL) divergence when action $i$ is pulled.
General Finite Model Classes: The analysis is extended to $M$ models and arbitrary action sets. The posterior evolution is treated as a Markov process on the $(M-1)$ -dimensional simplex. The authors utilize the theory of general-state Markov chains (Meyn and Tweedie, 1993; Khasminskii, 2012) to establish conditions for ergodicity versus transience. Key tools include:
- Log-Odds Representation: Transforming the belief process into an additive random walk with state-dependent drift: $S_{t+1} = S_t + \xi(S_t) + \epsilon_t$ .
- Drift Geometry: Analyzing the vector field of the conditional drift $\xi(S)$ , which is a convex combination of vertex drift vectors $d(a_j)$ (the expected log-likelihood increments when model $j$ 's preferred action is taken).
- Lyapunov Functions: Deriving sufficient conditions for stability using an "angle condition" (drift points toward an interior fixed point) and a "spectral condition" (analyzing the symmetric part of the drift matrix $G$ relative to a softmax potential).
- Recursive Dimensionality Reduction: When beliefs drift toward the boundary of the simplex (model elimination), the dynamics on the resulting lower-dimensional face inherit the same structural form, allowing for inductive analysis.

Key Results and Classification
The paper provides a complete classification of long-run regimes based on the interplay between model-action mappings and statistical evidence.

1. The Two-Arm Case:
Three qualitative regimes emerge when models disagree on the optimal action:

Self-Confirming Regime ( $\Delta_1 > 0 > \Delta_2$ ): Each model is supported by data from the arm it recommends. The posterior converges to one of the two boundary vertices (concentrating on either model $\nu$ or $\gamma$ ). The selection is path-dependent and random, determined by early likelihood shocks. The limiting outcomes are strict Berk-Nash equilibria.
Uniform-Dominance Regime ( $\Delta_1, \Delta_2 > 0$ ): One model is closer to the truth under both actions. The posterior converges almost surely to this superior model, eliminating the other.
Self-Defeating Regime ( $\Delta_1 < 0 < \Delta_2$ ): Each model is undermined by data from its own recommended arm. The posterior does not converge to a point. Instead, it converges in distribution to a unique, non-degenerate stationary measure $\mu$ on the interior $(0,1)$ . This results in persistent stochastic oscillation between models and actions, even with infinite data.

2. Regret Implications:

In the Self-Defeating regime, the agent incurs linear regret because the posterior fluctuates in the interior, leading to a positive long-run frequency of suboptimal actions.
In the Self-Confirming regime, regret is linear if the posterior concentrates on the incorrect model (which recommends the wrong arm), but zero if it concentrates on the correct one.
In Uniform-Dominance, regret is zero if the dominant model is correct, or linear if it is incorrect.

3. General Multi-Model Setting:
The authors establish sufficient conditions for the posterior to be ergodic (stochastically stable in the interior) or transient (drifting to the boundary):

Interior Ergodicity: Requires an interior fixed point of the mean-field drift (the origin lies in the convex hull of drift vectors) and either the Angle Condition (drift points inward) or the Spectral Condition (the symmetric part of the drift matrix is negative definite, and noise is sufficiently small).
Transience: Occurs if the origin lies outside the convex hull of drift vectors, or if the spectral condition indicates instability. In transient cases, the posterior mass shifts to lower-dimensional faces, recursively eliminating models until it settles on a vertex or a lower-dimensional ergodic face.
High-Dimensional Behavior: In high-dimensional settings with random drift vectors, the probability of an interior fixed point existing declines exponentially, suggesting that recursive dimensionality reduction and convergence to low-dimensional faces or vertices are the generic outcomes.

Significance and Contributions
The paper claims to make three primary contributions:

Complete Dynamic Classification: It provides the first complete characterization of posterior dynamics under Thompson Sampling with model misspecification. Unlike previous work that focuses on convergence to point estimates (Berk-Nash equilibria), this paper identifies a regime where beliefs fail to converge to any point, instead converging to a non-degenerate invariant distribution.
Endogenous Exploration vs. Deterministic Rules: The paper highlights that the built-in randomization of Thompson Sampling interacts with misspecification to produce dynamics impossible under deterministic best-response rules. Specifically, the "self-defeating" regime generates persistent policy variation and stochastic cycling that would not occur if the agent were purely myopic or deterministic.
Diagnostic Framework: The authors provide a tractable, testable framework (based on the sign pattern of log-likelihood drifts and drift geometry) for practitioners to diagnose whether a learning system will converge, cycle, or eliminate models. This allows for the prediction of long-run welfare and regret outcomes based on the structural properties of the misspecified model class.

The paper concludes that under misspecification, the standard intuition that Bayesian learning eventually "settles down" or "gets it right" can fail fundamentally. When decision rules generate the data they learn from, the feedback loop can sustain persistent uncertainty and policy variation indefinitely.

Technical Summary: Dynamic Decision-Making under Model Misspecification

More like this