🤖 AI

Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

This paper introduces the "semantic bandit" framework to demonstrate that large language models' exploration-exploitation trade-offs are significantly biased by semantic priors derived from pre-training, where informative labels can either enhance or degrade performance depending on alignment with rewards, and negative rewards disproportionately trigger exploration due to expected-scale biases.

Original authors: David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program that has read almost every book, article, and website on the internet. It can write poetry, solve math problems, and hold conversations that feel remarkably human. This is a large language model. Now, imagine giving this program a job where it must make a series of choices to get the best possible reward, like a farmer deciding which field to plant crops in, or a recommendation engine picking which shirt to show a customer. This is a decision-making task. In the world of computer science, there is a classic way to test how well an agent learns from experience: the multi-armed bandit problem. It is named after the row of slot machines in a casino, each with a different, unknown payout. To win the most money, you must balance two competing needs: sticking with the machine that seems to be paying out the most right now, and trying the other machines just in case one of them is actually better. This balance is called exploration versus exploitation. For decades, researchers have studied how to solve this mathematically. But when they started using these powerful language models as the decision-makers, they noticed something strange. The models were not just looking at the numbers; they were reading the words.

A team of researchers at McGill University and Mila decided to investigate this phenomenon by creating a new kind of test called the "semantic bandit." In a standard test, the choices are labeled with neutral codes like "Arm A" or "Arm B." In this new test, the researchers gave the choices names that carried meaning, such as "divine," "mild," or "nasty." They wanted to see if the computer would let the meaning of these words override the actual data it was collecting. They set up a scenario where the computer acted as a farmer choosing between three fields. Each field produced a random amount of crop yield, and the goal was to maximize the total harvest over time. The researchers gave the fields names that suggested their quality. In one version, the best field was named "divine" and the worst was named "nasty." In another version, they swapped the names, so the best field was called "nasty" and the worst was called "divine."

The results were striking. When the names matched the reality of the fields—when the "divine" field actually produced the most crops—the computer learned incredibly fast. It stopped trying the other fields almost immediately and stuck with the "divine" one, achieving a perfect score with very little trial and error. However, when the names were misleading, the computer made a catastrophic error. It became convinced that the field named "divine" was the best, even after the data showed it was producing the worst harvest. It ignored the evidence and kept choosing the "divine" field, while the field named "nasty" (which was actually the best) was left untouched. The computer was not acting on the numbers; it was acting on the associations it had learned from reading billions of words during its training. It had learned that "divine" usually means good and "nasty" usually means bad, and it applied that rule even when the specific numbers in front of it said otherwise.

The researchers also discovered that the computer reacted differently to positive and negative numbers. When the computer received a negative reward—a loss or a penalty—it suddenly became much more curious. It started trying all the different options again, exploring the unknown fields with high energy. But when it received a positive reward, even a small one, it tended to stop exploring and stick with what it knew. This suggests that the computer does not see numbers as abstract signals in a vacuum. Instead, it seems to have an internal expectation of what a "normal" reward looks like, likely shaped by the way rewards are described in the text it was trained on. A negative number breaks that expectation so severely that it forces the computer to re-evaluate everything, while a positive number feels safe and encourages it to stay the course.

This behavior was not limited to one specific computer model. The researchers tested three different large language models, ranging from open-source systems to advanced commercial versions, and they all showed the same bias. The effect was so strong that it could turn a simple decision problem into a failure. In a task where a classical computer program would solve the problem in a few steps, the language model could fail completely if the words were slightly misleading. The researchers found that they could partially fix this by explicitly telling the computer in its instructions to ignore the names and to keep exploring, but the bias was deep-seated. Even with warnings, the models often struggled to separate the meaning of the words from the reality of the numbers.

The study concludes that using language to describe a decision-making environment introduces unavoidable biases. Because these models are trained on the co-occurrence of words in human text, they carry a set of assumptions about how the world works. When those assumptions align with the task, the model is a genius, learning faster than any traditional algorithm. When they clash, the model becomes stubborn, ignoring clear evidence because it is too busy following the story the words are telling it. This is not a bug in the code, but a feature of how these systems learn from language. As these models are deployed in real-world situations, from financial advice to autonomous driving, understanding when their linguistic intuition helps and when it leads them astray is critical. The researchers found that in the simplest possible decision-making scenario, a computer can be tricked into making the worst possible choice simply by changing the label on a button.

Technical Summary: Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

Problem Definition

Large Language Models (LLMs) are increasingly deployed as decision-making agents in In-Context Reinforcement Learning (ICRL) settings, where they must balance exploration (gathering information) and exploitation (maximizing known rewards) based on feedback provided in natural language. While classical reinforcement learning (RL) agents operate on symbolic representations that strip away all non-essential information, LLMs process tasks through natural language, exposing them to semantic information (e.g., action labels, domain context, numeric sign) that has no formal counterpart in the task structure.

The core problem addressed is that these "semantic priors"—inductive biases arising from word co-occurrence statistics learned during pre-training—may systematically distort LLM exploration behavior. Existing evaluation frameworks based on classical RL often fail to account for these biases, potentially leading to misleading assessments of an agent's decision-making capabilities. The paper posits that the use of language to define environments introduces unavoidable biases that can either aid or hinder performance depending on the alignment between the semantic prior and the true reward distribution.

Methodology

To investigate this, the authors introduce the Semantic Bandit, a formal extension of the Multi-Armed Bandit (MAB) problem. Unlike a classical MAB defined solely by reward distributions, a semantic bandit instance explicitly includes:

Action Nomenclature ( $L$ ): Textual labels assigned to each arm (e.g., "divine," "mild," "nasty").
Scenario ( $c$ ): The thematic domain grounding the problem (e.g., farming, clothing recommendation, abstract bandit).

Experimental Design

The study evaluates three instruction-tuned LLMs (OLMo-3.1-32B, Qwen3-32B, and Gemini 3.1 Flash Lite) across three scenarios with varying levels of semantic complexity. The experiments manipulate two primary variables:

Action Nomenclature: Four types were tested:
- Alphanumeric: Random strings (control condition).
- Sentiment: Adjectives with positive, neutral, or negative valence.
- Ordinal: Explicit rank labels (Highest, Intermediate, Lowest).
- World Knowledge: Domain-specific labels where value is inferred from pre-training (e.g., "Productive Grassland" vs. "Arid Desert").
- Alignment: Each nomenclature was tested in "Helpful" (semantic label matches high reward) and "Misleading" (semantic label matches low reward) configurations.
Reward Scale: The polarity and magnitude of observed rewards were varied, including positive scales (High+, Low+) and negative scales (High-, Low-), to test for expected-scale biases.

Metrics

Performance was measured using:

Normalized Cumulative Regret: The difference between the optimal arm's reward and the agent's accumulated reward, normalized to $[0, 1]$ .
Exploration Count: The number of distinct arms selected over the episode.
Exploration Probability: In scale-sweep experiments, the probability of switching arms immediately after observing a specific reward value.

Key Contributions

The Semantic Bandit Framework: The paper formalizes the extension of the MAB problem to include action nomenclature and scenario context, providing a controlled environment to isolate the effects of semantic priors on exploration.
Characterization of Semantic Bias: The authors demonstrate that semantic priors are not merely noise but active drivers of decision-making that can override reward signals.
Reward Polarity Effects: The study identifies a distinct bias where negative rewards trigger significantly more exploration than equivalent positive rewards, suggesting LLMs interpret rewards relative to an expected scale derived from pre-training conventions.
Interaction of Priors and Rewards: The paper reveals that negative rewards can nullify the influence of action nomenclature, forcing the model to explore even when semantic labels suggest exploitation.

Results

The experiments yielded three primary findings:

Action Nomenclature Drives Exploration/Exploitation:
- When nomenclature was aligned with the reward structure (e.g., positive labels on high-reward arms), models exploited semantic priors to achieve low regret with minimal exploration, often outperforming classical baselines like UCB1.
- When nomenclature was misaligned (e.g., positive labels on low-reward arms), models were severely misled, sticking to suboptimal actions and suffering high cumulative regret.
- Ordinal labels induced the strongest bias, followed by Sentiment and World Knowledge. Alphanumeric labels resulted in behavior closest to the normative UCB1 baseline.
- Different models exhibited distinct behaviors: OLMo struggled to explore without explicit task framing; Qwen3 showed consistent exploitation of ordinal labels; Gemini explored quickly but was susceptible to world-knowledge biases in complex contexts.
Reward Polarity as a Contextual Cue:
- Negative reward values triggered substantially more exploration than positive values. This suggests LLMs do not treat rewards as abstract numerical signals but interpret them relative to an expected scale (e.g., negative values are "unexpected" or "bad," prompting a search for better options).
- This effect was most pronounced in OLMo and Qwen3, while Gemini showed a more muted response.
Mitigation via Prompting:
- Explicitly warning models about potential label bias or instructing them to "explore" increased exploration counts.
- However, the impact on cumulative regret was inconsistent: in "Helpful" scenarios, forcing exploration often increased regret (by disrupting efficient exploitation), while in "Misleading" scenarios, it decreased regret.

Significance and Claims

The paper argues that the reliance on natural language for ICRL tasks introduces a fundamental divergence from classical decision-making theory. The most significant claim is that semantic priors are strong enough to completely determine behavior in one of the simplest decision-making problems (a stationary 3-armed bandit with no variance).

Evaluation Implications: Practitioners evaluating LLM agents on tasks with naturalistic, semantically coherent labels may be measuring a mixture of genuine reasoning and semantic prior exploitation. A model appearing to perform well may simply have priors that align with the evaluation environment, leading to silent failures in deployment settings where that alignment does not hold.
Reliability Concerns: The findings raise concerns about the stability of LLM agents in real-world settings (e.g., recommendation systems, autonomous agents). The ability of surface-level text to override reward signals suggests that robustness cannot be assumed merely because a model performs well on standard benchmarks.
Future Directions: The authors conclude that understanding when semantic context aids adaptation versus when it induces systematic errors is a prerequisite for safe deployment. They propose the Semantic Bandit as a tool for future work on the robustness and calibration of LLM decision-making.

The paper remains modest in its scope, acknowledging limitations such as the small number of replicates due to computational constraints and the simplicity of the synthetic environment. It calls for further research to ground these effects in realistic agentic deployment settings.