Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Experiments

1. Problem Statement

The paper addresses a critical gap in understanding whether Large Language Models (LLMs) can reliably simulate human decision-making behavior in complex, sequential environments under uncertainty. Specifically, it examines the Exploration-Exploitation (E&E) Tradeoff, a fundamental cognitive challenge where agents must balance leveraging known rewarding options (Exploitation) against seeking new possibilities that could yield better outcomes (Exploration).

Although LLMs are increasingly used as "digital twins" to simulate human behavior, it remains unclear whether they exhibit the same adaptive, goal-directed, and cognitively grounded strategies as humans, particularly in non-stationary environments. The authors ask: Can LLMs reliably simulate human E&E behavior, and how do "thinking" abilities (e.g., Chain-of-Thought, internal reasoning traces) influence these strategies?

2. Methodology

Experimental Design

The authors employed two canonical Multi-Armed Bandit (MAB) experiments widely used in cognitive science and psychiatry to ensure comparability with human data:

1. Stationary 2-Armed Bandit: A simple setting with 20 runs of 10 rounds each. Rewards are stochastic but stationary (fixed mean).
2. Non-Stationary 4-Armed Bandit: A complex setting with 300 rounds, where reward means follow a decaying Gaussian Random Walk (diffusion process). This tests adaptability to changing environments.

Agents and Conditions

The study compared three categories of agents:

• Human Participants: Data from controlled laboratory experiments (Gershman 2018; Chakroun et al. 2020a) with 45 and 31 participants respectively.
• LLM Agents: Three commercial model families (ChatGPT, Gemini, DeepSeek), tested in two configurations:
  • Base Models: Standard inference (e.g., GPT-4o-mini, Gemini-2.0-flash, DeepSeek-V3).
  • Thinking Models: Models with explicit reasoning capabilities (e.g., GPT-o3-mini, Gemini-2.5-flash, DeepSeek-R1).
  • Prompting Variations: For base models, the authors tested Chain-of-Thought (CoT) prompting ("thinking aloud") compared to direct answering. For thinking models, they activated internal reasoning modes or adjusted reasoning budgets.
• Algorithmic Baselines: Standard MAB algorithms, including Upper Confidence Bound (UCB), Thompson Sampling (TS), and $\epsilon$-greedy.

Behavioral Modeling and Estimation

To characterize decision strategies, the authors used interpretable choice models from cognitive science:

• Learning Rule: A Bayesian learner (Kalman Filter) to estimate expected rewards ($Q$) and uncertainty ($S$) for each arm.
• Choice Rules:
  • Softmax-based Models (SM-1, SM-2, SM-3): Contain parameters for random exploration ($\beta$), directed exploration/uncertainty bonus ($\phi$), and choice perseveration ($\rho$).
  • Probit Regression: Used for the 2-Armed task to model value differences and uncertainty.
• Estimation: A hierarchical Bayesian approach was used to estimate parameters at the population level while accounting for individual heterogeneity. Model selection was performed using Leave-One-Out Cross-Validation (LOO-CV).

3. Main Contributions

1. Direct Human-LLM Comparison in Complex Settings: Unlike previous studies limited to simple 2-Armed settings or algorithmic benchmarks, this work provides the first direct comparison of LLMs and humans in both stationary and non-stationary 4-Armed Bandit tasks under matched experimental conditions.
2. Disentangling "Thinking" Abilities: The study systematically isolates the influence of reasoning by comparing base models with CoT prompting against native "thinking" models with switchable internal reasoning functions. This goes beyond simple performance metrics to analyze the mechanism of decision-making.
3. Behavioral Parametrization: Instead of merely measuring regret, the authors map LLM behavior to established cognitive parameters ($\beta, \phi, \rho$), allowing a differentiated analysis of how LLMs explore (randomly vs. directed) rather than just how well they perform.
4. Analysis of Model Specification: The authors demonstrate that these cognitive models, even when applied to non-human agents (algorithms/LLMs), serve as a valid "projection framework" to extract interpretable behavioral signatures.

4. Main Results

Exploration Strategies

• Base LLMs: Show significantly higher random exploration and lower directed exploration compared to humans. They lack the consistent "choice perseveration" (repetition of a choice) observed in humans.
• Influence of Thinking: Activating reasoning processes (via CoT or internal reasoning) shifts LLM behavior toward human-like patterns:
  • Stationary Setting: Thinking-capable LLMs significantly increase directed exploration ($\phi$) and reduce random exploration ($\beta$), often reaching human levels or even exceeding them.
  • Non-Stationary Setting: Although thinking improves performance, LLMs still struggle to achieve human adaptability. They fail to maintain effective directed exploration in complex, changing environments, showing negative or near-zero $\phi$ values, while humans maintain positive directed exploration.

Performance Metrics (Regret and Exploitation)

• Stationary (2-Armed): Base LLMs achieve regret levels comparable to humans and algorithms. Thinking models perform similarly or slightly better.
• Non-Stationary (4-Armed):
  • Humans: Achieve the lowest cumulative regret and show superior adaptability to changing reward structures.
  • Base LLMs: Accumulate regret much faster than humans and show volatile exploitation rates (excessive sticking to early arms).
  • Thinking Models: Significantly reduce regret compared to base models and bring performance closer to human levels (e.g., Gemini-2.5-flash and DeepSeek-R1), but still lag behind human adaptability in the most complex scenarios.

Long-Term Dynamics

• In extended stationary tasks (300 rounds), thinking models stabilize their behavioral parameters over time, whereas base models exhibit high variance.
• An analysis using the QCARE framework shows that base models often suffer from "over-exploration" (no decay of structural noise), while thinking models achieve a better trade-off closer to the theoretical optimum ($\alpha \approx 0.5$).

5. Significance and Implications

• Limits of Human Simulation: Although LLMs can mimic human E&E strategies in simple, static environments when prompted to "think," they currently lack the robust adaptability required for complex, non-stationary real-world scenarios. They struggle to maintain goal-directed exploration when the environment changes.
• Role of Reasoning: The study provides empirical evidence that reasoning capabilities (not just generation randomness) are the primary driver for improved E&E strategies in LLMs. Activating reasoning traces is a cost-effective, efficient method to align LLM behavior with human-like decision-making without architectural changes.
• Practical Applications: For automated decision systems (e.g., scientific discovery, dynamic resource allocation), reliance on base LLMs in non-stationary environments can lead to suboptimal outcomes. However, integrating explicit reasoning modules or prompt engineering can significantly mitigate these risks.
• Future Directions: The paper suggests that future work should focus on better aligning LLMs with non-stationarity through fine-tuning and integrating demographic/individual variability into simulation prompts to improve the accuracy of "digital twins."

In summary, the paper concludes that while LLMs are powerful simulators of human behavior, their current "thinking" abilities represent a double-edged sword: they bridge the gap in simple tasks but reveal a fundamental limitation in adapting to complex, dynamic uncertainty compared to human intuition.