🧬 biology

Entrenchment in Human Exploration: A Future-Path Model of Early Convergence

This paper introduces a cognitive model of "entrenchment" in human exploration, demonstrating that people prematurely settle on suboptimal options by simulating their own future choices, a mechanism that uniquely explains increased non-greedy behavior with longer horizons and distinguishes itself from standard exploration strategies like UCB or Thompson sampling.

Original authors: Shinji Nakazato

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Shinji Nakazato

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Human beings are constantly faced with a fundamental choice: should we stick with what we know works, or should we try something new to see if it is better? This dilemma, known to scientists as the exploration-exploitation tradeoff, is the engine of learning. When we explore, we gather information about the world; when we exploit, we use that information to get the best reward. Ideally, we balance these two drives, sampling enough to find the best option and then settling in to enjoy it. However, in many situations, people seem to stop searching too soon. They lock onto a choice that is not actually the best one and keep making it, even when better options are available. This stubborn persistence, which researchers call "entrenchment," suggests that our brains might be simulating the future in a way that traps us in suboptimal habits.

A new study by Shinji Nakazato at the Tokyo University of Science offers a fresh explanation for why this happens. The research proposes that when we evaluate our current choices, we do not just look at the immediate reward. Instead, we run a mental simulation of our own future decisions, imagining what we will do next. Crucially, the study suggests that we perform this simulation while holding our current beliefs fixed, as if we will never learn or update our opinions in the future. This mental shortcut creates a specific kind of uncertainty about our future path. The model shows that when the time horizon is long—meaning we have many decisions left to make—this simulated uncertainty grows large enough to make us abandon the best-known option and drift toward worse ones. The findings challenge the idea that humans are always trying to maximize their rewards, suggesting instead that our own internal simulations of the future can lead us astray.

To test this idea, the researcher analyzed data from two different types of experiments where people played games involving repeated choices. In one set of experiments, known as the Wilson task, participants faced a simple two-option game. The researchers carefully controlled the game so that the participants' knowledge about the options was identical at the start of a decision, but the number of future turns they had to play varied. In some cases, participants knew they had only one turn left; in others, they knew they had six turns left. If people were simply trying to find the best option, their behavior should have been the same in both cases because their knowledge was the same. However, the data showed a clear difference: when participants knew they had more turns remaining, they were significantly more likely to choose the option that was not the best one. This result was so strong that it could not be explained by standard theories of how people learn or by the idea that they were just being more careful or curious. The only model that could reproduce this specific pattern was the one based on the "future-path" simulation, where the decision-maker imagines their future choices without anticipating that they will learn anything new.

The study went further to distinguish this behavior from other forms of exploration. Sometimes, people choose a worse option on purpose to gather more information about the best possible choice; this is called directed exploration. To see if the stubborn choices in the study were a form of smart exploration or just getting stuck, the researcher looked at a second, more complex game played on a grid with many possible spots. In this game, the true best spot was known to the researchers. The analysis revealed that when people made non-optimal choices in the long-horizon situations, they were overwhelmingly moving toward spots that were genuinely worse, not toward the best spot. They were not exploring to find the optimum; they were drifting away from it. This confirmed that the behavior was indeed "entrenchment," a self-reinforcing loop where a person gets stuck on a suboptimal choice because their mental simulation of the future makes that choice seem more attractive than it really is.

The research also examined how the size of the choice set affects this behavior. In the grid game, participants had to choose between 30 or 121 different spots. The model predicted that having more options would increase the uncertainty in the mental simulation of the future, making it even more likely that people would get stuck on a bad choice. The data supported this: as the number of available options increased, the rate of sticking with a non-optimal choice also went up. Furthermore, the study found that when the difference in value between the best option and the second-best option was very clear, people were less likely to get stuck. But when the gap was small, or when the future felt far away, the tendency to persist with a mediocre choice grew stronger.

These findings paint a picture of human decision-making that is less about cold calculation and more about the limitations of our imagination. The study suggests that our brains are not always looking ahead to optimize our future; sometimes, they are looking ahead and seeing a fog of possibilities that makes the current, familiar path seem safer, even when it leads nowhere. By simulating our future selves as if we will never change our minds, we inadvertently create a barrier to learning. The research does not claim that this is a flaw in human intelligence, but rather a structural feature of how we process time and uncertainty. It shows that the very mechanism we use to plan our lives—imagining what we will do next—can sometimes be the thing that keeps us from finding the best path forward. The results were robust across thousands of trials and held true even when the researchers ruled out other explanations, such as simple mistakes or a general preference for variety.

Ultimately, the paper provides a mathematical framework that captures this specific type of getting stuck. It shows that the probability of making a non-optimal choice increases as the remaining time grows and as the number of options expands. This is the opposite of what traditional theories of rational learning predict, which suggest that having more time should help people find the best solution. The study confirms that for human learners, having more time can actually make us more likely to settle for less. The research leaves open the question of exactly how the brain calculates this future uncertainty, but it firmly establishes that the way we imagine our future choices is a powerful driver of our present behavior, capable of locking us into patterns that are not in our best interest.

Technical Summary: Entrenchment in Human Exploration

Problem Statement
Human learners in exploration-exploitation tasks frequently exhibit "entrenchment": a pattern where they terminate exploration earlier than the value of information warrants and persist with a non-optimal option. While existing literature identifies this early convergence (e.g., Daw et al., 2006; Wilson et al., 2014), there is a gap in computational-cognitive accounts that can formally model this mechanism. Standard reward-seeking algorithms (e.g., GP-UCB, Thompson sampling) predict that longer decision horizons increase the probability of choosing the optimal option. Conversely, economic cost-based frameworks struggle with empirical identification and tractability in continuous environments. This paper seeks to formalize entrenchment as a specific cognitive mechanism that generates non-greedy choices, particularly as the decision horizon lengthens, distinguishing it from directed uncertainty exploration.

Methodology and Theoretical Framework
The paper proposes a cognitive model where a decision-maker (DM) evaluates a current option by simulating their own future choices under beliefs held fixed at the moment of evaluation. This "future-path distribution" ( $q$ ) represents the DM's projection of subsequent choices.

Model Setup: The environment is modeled as a Gaussian Process (GP) bandit problem, where the latent reward function $f$ is learned via GP regression. The DM observes noisy rewards $r_t = f(x_t) + \epsilon_t$ .
Future-Path Simulation: At time $t$ , the DM assumes a future choice path where the current action is fixed, and subsequent actions $X_{t+1}, \dots, X_T$ are drawn i.i.d. from a distribution $q$ over the option set $D$ . Crucially, the DM does not anticipate that their future self will update beliefs based on new information (a fixed-belief anticipated utility structure).
Choice Statistic: By combining the GP posterior with the simulated future path, the author derives a closed-form choice statistic, $\hat{\kappa}_q$ $\overset{κ}{^}_{q}$ , which governs the probability of choosing the posterior-best option versus a non-greedy option:
$\hat{\kappa}_q = \frac{\Delta\mu_{t-1}}{\sqrt{\Delta V_{t-1} + 2(S-1)V_q + 2S\rho^2}}$
Where:
- $\Delta\mu_{t-1}$ is the posterior mean advantage of the best option.
- $\Delta V_{t-1}$ is the posterior variance of the mean gap.
- $S = \sum_{i=0}^{A-1} \gamma^{2i}$ is a horizon-dependent scaling factor ( $A$ is the remaining horizon, $\gamma$ is the discount factor).
- $V_q$ is the variance of the simulated future path under the GP posterior.
- $\rho^2$ is observation noise.
  The probability of choosing the non-greedy option is $1 - \Phi(\hat{\kappa}_q)$ .

Key Contributions

Formalization of Entrenchment: The paper defines entrenchment as a two-stage event: (a) choosing a non-posterior-greedy option, and (b) maintaining that ranking after observing rewards due to GP shrinkage (lock-in). The model provides a closed-form expression for the probability of stage (a) without estimating a search-cost function.
Novel Comparative Statics: The derived statistic yields sign predictions that contradict reward-maximizing algorithms. Specifically, the model predicts that a longer remaining horizon ( $A$ ) or a more dispersed future path ( $V_q$ ) increases the probability of a non-greedy choice. This is because the variance term in the denominator of $\hat{\kappa}_q$ scales with the horizon, reducing the signal-to-noise ratio of the greedy advantage.
Distinction from Directed Exploration: Unlike uncertainty-directed exploration (which seeks to reduce uncertainty about the global optimum), this model predicts that non-greedy choices move toward genuinely inferior options, driven by the variance of the simulated future path rather than a strategic search for the optimum.

Empirical Results
The author tests the model against two publicly available datasets: the Wilson Horizon Task (two-armed bandit with within-subject horizon manipulation) and the Wu GP Bandit task (spatially correlated bandits on 1D and 2D grids).

Horizon Effect (Prediction P1): In the Wilson task, where the posterior is equated across conditions by design, the rate of non-greedy choices significantly increases when the horizon extends from $H=1$ to $H=6$ (from 15.4% to 26.3%). Maximum-likelihood comparisons show that horizon-free rules (Softmax, Thompson Sampling, GP-UCB) cannot reproduce this effect at any parameter value, whereas the future-path rule does.
Posterior Gap Effect (Prediction P3): Across both datasets, as the posterior mean gap ( $\Delta\mu$ ) between the best and second-best options increases, the probability of non-greedy choices decreases, consistent with the model.
Choice Set Size (Prediction P2): The model predicts that larger choice sets increase entrenchment via $V_q$ . While this holds descriptively across datasets (comparing $|D|=2$ to $|D|=30/121$ ), it is not identified as a controlled within-task effect due to collinearity with task identity.
Nature of Non-Greedy Choices: Analysis of the Wu task reveals that non-greedy choices predominantly move toward options with strictly lower true values (regret-increasing), rather than toward the global optimum. This distinguishes entrenchment from directed exploration.

Significance and Claims
The paper claims to provide a mechanism that generates entrenchment as a fittable, closed-form model, separating it from reward-seeking exploration and cost-based stopping rules. The primary empirical contribution is the demonstration that human choice behavior exhibits a horizon-dependent increase in non-greedy selection that is structurally impossible for standard horizon-free algorithms to replicate.

The author modestly notes that the empirical validation establishes "sign consistency" rather than full structural identification of parameters like the discount factor $\gamma$ or the specific form of $q$ . The model successfully rules out horizon-free accounts and simple choice noise, but a fuller separation from horizon-dependent directed exploration requires further within-subject designs. The paper concludes that the "future-path distribution" serves as a reduced-form primitive that captures the cognitive mechanism of early convergence and lock-in in human exploration.

Technical Summary: Entrenchment in Human Exploration

More like this