Should You Use Your Large Language Model to Explore or Exploit?

Technical Summary: Should You Use Your Large Language Model to Explore or Exploit?

Problem Statement
The paper addresses the capability of Large Language Models (LLMs) to function as decision-making agents facing the exploration-exploitation tradeoff, a core challenge in reinforcement learning (RL) and bandit problems. While prior research has evaluated LLMs on combined exploration-exploitation tasks with mixed results, this work systematically decouples these capabilities. It investigates whether LLMs can serve as effective "exploitation oracles" (selecting the best action given historical data) and "exploration oracles" (suggesting high-quality candidate actions from large, semantically rich action spaces) in isolation. The authors aim to determine if pre-trained, generalist LLMs can leverage their inductive biases to solve these tasks in-context, without specialized RL fine-tuning.

Methodology
The authors conduct a systematic evaluation across two distinct roles using various LLMs, including the GPT-5 family (GPT-5.2, GPT-5-MINI, GPT-5-NANO), GPT-4, GPT-3.5, Qwen-2.5, Gemma-3, Mistral-7B, and the reasoning model DeepSeek-R1-Distill-Qwen.

1. Exploitation Experiments:

   • Tasks: The authors define "exploit puzzles" based on Multi-Armed Bandits (MAB) and Contextual Bandits (CB).
     • MAB Puzzles: LLMs are given a history of rewards for $K$ arms and asked to select the arm with the highest empirical average reward.
     • CB Puzzles: LLMs are given a history of (context, action, reward) tuples and a new context, tasked with selecting the action that maximizes expected reward. Tasks include numerical linear CB, numerical non-linear CB, and text-based CB.
   • Metrics: Performance is measured by FracCorrect, the fraction of tasks where the LLM selects the empirically or theoretically optimal arm. Difficulty is parameterized by the empirical gap (MAB) or effective gap (CB).
   • Mitigations: To address context window limits and reasoning deficits, the authors test two mitigations:
     • Tool Use: Providing access to a Python code interpreter.
     • In-Context Summarization: Techniques including $k$-nearest neighbors, $k$-means clustering, and hybrid approaches to condense history before prompting.

2. Exploration Experiments:

   • Tasks: The authors design exploration tasks where the action space is large and text-based, making standard discretization methods impractical.
     • MovieLens: Recommending movies to users based on the MovieLens 100K dataset.
     • arXiv: Suggesting titles for research paper abstracts.
     • Q/A: Answering open-ended philosophical questions with contrarian ground truths.
   • Protocol: An LLM is called once to generate a small set of $K$ candidate actions. A standard bandit algorithm (UCB1) is then run over these candidates for $T=1000$ rounds.
   • Metrics: Performance is measured by the average cumulative reward (rew) of the UCB1 algorithm over the LLM-selected candidates.
   • Baselines: Comparisons include random selection, genre-based discretization (MovieLens), and Maximal Marginal Relevance (MMR) retrieval (arXiv).

Key Contributions and Results

1. LLMs as Exploitation Oracles

• MAB Performance: Frontier models (GPT-5 family) perform significantly better than older models (GPT-4, GPT-3.5). GPT-5-MINI achieves perfect performance on simple MAB puzzles. GPT-4 and GPT-3.5 struggle, particularly as history size ($T$) increases or the empirical gap decreases, explaining previous failures in end-to-end MAB tasks.
• CB Performance: In numerical linear CB tasks, performance degrades as dimensionality ($d$), number of arms ($K$), and history size ($T$) grow. While tool use (code interpreter) and in-context summarization mitigate context window issues and improve performance on linear tasks, LLMs consistently underperform a simple linear regression baseline.
• Non-Linear/Text-Based CB: In non-linear settings, even with mitigations, LLMs perform worse than linear regression. The authors attribute this to the difficulty LLMs face in executing numerical reasoning strategies correctly, exacerbated by non-linearity, whereas linear regression remains robust.
• Reasoning Models: Reasoning models (e.g., DeepSeek-R1) show promise but are often too slow or expensive for practical deployment.

2. LLMs as Exploration Oracles

• Superiority over Baselines: LLMs significantly outperform non-LLM baselines (random selection, genre-based, MMR) in all three exploration tasks.
• Semantic Leveraging: LLMs effectively leverage semantic meaning in action spaces to suggest diverse and relevant candidates. For instance, in the arXiv task, LLM-generated titles achieved higher cosine similarity to ground-truth titles than retrieval-based baselines.
• Robustness: Performance is consistent across different LLMs (GPT-4O, Qwen-2.5, Mistral-7B, Gemma-3) and prompting strategies (all-at-once vs. one-by-one, with/without diversity encouragement).
• Q/A Task: LLMs generate reasonable, semantically diverse answers that pass an "eye test," whereas random baselines fail catastrophically.

Significance and Conclusions
The paper concludes that current LLMs are not effective as standalone exploitation oracles for complex or large-scale decision-making tasks, particularly when compared to simple algorithmic baselines like linear regression. Despite advancements in model size and reasoning capabilities, LLMs struggle with the precise numerical reasoning required for exploitation, especially in non-linear or high-dimensional settings.

Conversely, LLMs are highly effective as exploration oracles for large, semantically meaningful action spaces. They can successfully propose high-quality candidate actions where traditional discretization methods fail due to high dimensionality or lack of structural priors. The authors suggest that the most promising path for integrating LLMs into decision-making agents is to use them for exploration (action proposal/discretization) while relying on traditional algorithms for exploitation. The work highlights that while LLMs have made rapid progress, their limitations in precise, in-context numerical decision-making persist longer than anticipated, challenging the notion that generalist LLMs can easily master all simple RL tasks.