🤖 machine learning

Benchmarking In-context Experiential Learning Through Repeated Product Recommendations

This paper introduces BELA, a benchmark for experiential learning in repeated product recommendation scenarios using real-world products and synthetic customer personas, which reveals that while current LLM-based agents can adapt within individual interactions, they struggle to improve their strategies across multiple episodes by inferring shared latent structures.

Original authors: Gilbert Yang, Yaqin Chen, Thomson Yen, Hongseok Namkoong

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gilbert Yang, Yaqin Chen, Thomson Yen, Hongseok Namkoong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues are hidden inside a conversation. In the world of artificial intelligence, there is a big difference between solving a single puzzle and learning how to solve puzzles better over time. Most AI tests today are like giving a student a single math problem with all the numbers already written down; the goal is just to get the right answer right now. But real life is messier. It's more like a detective who talks to a witness, asks a few questions, gets an answer, and then has to use that experience to interview the next witness more effectively. This paper dives into a specific corner of AI research called "experiential learning." It asks a simple but tricky question: Can an AI get smarter at figuring out what people want just by talking to them over and over again? The researchers are testing if these digital agents can learn from their past conversations to become better detectives in the future, rather than just being good at solving one isolated case.

The authors of this paper, a team from Columbia Business School and Sun Yat-sen University, decided to test this idea using a scenario we all know: shopping for products. They built a giant, virtual playground called BELA (Benchmark for Experiential Learning and Active exploration). Think of BELA as a massive, infinite mall where the AI plays the role of a salesperson. In this mall, the AI has to talk to thousands of different "customers" (who are actually computer simulations of people with specific tastes) to figure out which product they should buy. The twist is that the AI doesn't just get one shot. It talks to a customer, makes a recommendation, gets feedback, and then moves on to a new customer. The big test is whether the AI remembers what it learned from the first customer and uses that knowledge to ask better questions of the second, third, and tenth customer.

To make this test fair and huge, the researchers didn't just make up a few fake people. They created a system with 71,000 real products from Amazon and 2,000 different groups of products (like a group of hair gels or a group of rice dishes). They then paired these with 1 million different customer personalities (simulated by other AIs) who have very specific, consistent likes and dislikes. This creates a mind-boggling 2 billion possible combinations of "customer + product group." It's like having a library of every possible shopping trip you could ever imagine, allowing the researchers to see if the AI can spot patterns across the crowd.

The researchers ran their experiments with the smartest AI models available today, including giants like GPT-5.4, Gemini-3.1, and Claude-Opus. They watched these models try to sell products over a series of 10 shopping trips (called episodes). They wanted to see two things: First, could the AI learn during a single conversation (asking the right follow-up questions)? Second, could the AI learn across conversations (getting better at the job as it gained more experience)?

Here is the surprising result: The AI models were actually pretty good at the first part. When talking to a single customer, they could learn from the conversation and ask better questions as the chat went on. However, they completely failed at the second part. After talking to nine different customers, the AI was no better at guessing what the tenth customer wanted than it was with the first one. It was like a salesperson who is great at figuring out what one person likes, but every time a new person walks in, they start from scratch, forgetting everything they learned from the previous nine people.

The paper suggests that these current "frontier" models are missing a crucial skill: they can't look back at their past experiences and say, "Oh, I see a pattern here; I should ask this type of question next time." Instead, they seem to treat every new customer as a completely fresh mystery, even if they just met someone with very similar tastes five minutes ago. The researchers also found that the AI didn't get better at guessing how confident it should be; it kept asking the same number of questions, even when it should have learned to ask fewer.

To prove that the task wasn't just too hard, the researchers created an "ideal path." They manually guided the AI through the best possible questions and showed that if the AI did use its past experiences correctly, it could have drastically improved its recommendations and asked fewer questions. This proves that the information was there to be learned, but the current models just weren't picking it up.

In short, this paper suggests that while our current AI is getting very good at having a single, smart conversation, it hasn't yet figured out how to be a "learning machine" that gets smarter over its entire career. It's a bit like a student who can ace a test if they study the night before, but forgets everything the moment the bell rings, unable to use what they learned on Monday to help them on Tuesday. The authors conclude that for AI to truly handle the messy, shifting real world, it needs to develop this ability to learn from experience across multiple episodes, a skill that today's most advanced models still struggle to master.

Technical Summary: Benchmarking In-context Experiential Learning Through Repeated Product Recommendations

Problem Statement

Current evaluations of Large Language Model (LLM) agents largely overlook the capability of experiential learning across multiple episodes. While recent benchmarks have addressed uncertainty resolution within a single task (episode), they fail to measure an agent's ability to refine its strategy by inferring shared latent structures across similar, sequential tasks.

In real-world scenarios, such as a sales agent recommending products to a stream of unknown customers, an intelligent agent should not only resolve uncertainty within a specific interaction but also adapt its information-gathering strategy based on the distribution of preferences observed in previous interactions. Without this capability, agents cannot tailor their questioning to the specific task distribution revealed at test time. The paper argues that current models struggle to extract and utilize high-level information from past experiences to improve future performance, a gap this work aims to expose and measure.

Methodology: The BELA Benchmark

To isolate and measure in-context experiential learning, the authors introduce BELA (Benchmark for Experiential Learning and Active exploration), a scalable product recommendation framework. The benchmark is designed to prioritize learnability and scalability over strict realism, creating a controlled environment where latent preference structures are consistent and learnable.

Core Components

Choice Sets Module:
- Derived from 100K products sampled from the Amazon Reviews dataset.
- Filtered and mapped to a predefined category hierarchy (ASINSpotlight), resulting in 71K products organized into 2K distinct choice sets.
- Categories are selected to ensure they function as reasonable choice sets (e.g., excluding overly broad categories like "Home & Kitchen").
Customer Module (Persona Simulation):
- Utilizes 1 million synthetic customer personas from Li et al. [11] to ensure diversity and scalability.
- Preference Generation: Product scores ( $y_a$ ) for each persona-product pair are generated by averaging outputs from GPT-4o and Gemini-1.5-Pro. The scoring process is verified for consistency, showing low variation across seeds compared to the standard deviation within categories.
- Interaction Simulation: An LLM (GPT-4o) acts as the customer, responding to agent questions based on the persona profile.
- Feedback Mechanisms: The system supports three feedback types:
  - Regret: The difference between the optimal score and the recommended score.
  - Stars: A 1–5 star rating derived from the regret ratio.
  - Free-form Text: Natural language feedback generated by the customer LLM, reflecting the persona's voice and specific preferences.

Experimental Setup

The benchmark evaluates agents across a stream of episodes, where each episode $e$ is defined by a customer $c_e$ and a choice set $S_e$ . The agent's goal is to minimize regret ( $y^* - y_r$ ) while balancing interaction efficiency (number of questions asked).

Three sequential settings are tested:

Same customer, different choice sets per episode.
Same category, different customers per episode.
Different customers and choice sets per episode.

Baselines include RANDOM (uniform selection), POPULARITY (highest original rating), and ORACLE (an agent with full access to the persona description, serving as a lower bound for regret).

Key Results

The authors evaluated a suite of frontier models (including GPT-5.4, Gemini-3.1-Pro-Preview, and Claude-Opus-4.6) against the BELA benchmark.

Failure in Cross-Episode Learning:
- While models perform better than random baselines, they fail to exhibit meaningful improvement across episodes. Regret scores remain stagnant or fluctuate without a downward trend as the number of episodes increases.
- In contrast, "ideal trajectories" constructed by manually curating high-quality questions and reusing information across episodes demonstrate that significant regret reduction is possible if the agent successfully leverages cross-episode information.
Lack of Proactive Information Gathering:
- Models do not adapt their questioning strategies based on experience. The number of questions asked remains constant or even decreases over episodes, indicating a failure to recognize the need for more information gathering when uncertainty is high.
- Models fail to calibrate their uncertainty estimates. Even when confidence in their predictions increases over episodes, this confidence does not correlate with actual performance improvements.
Within-Episode vs. Across-Episode:
- Models exhibit within-episode learning (improving their recommendations as they gather more turns of information within a single session).
- However, they lack across-episode learning (refining their initial questioning strategy or latent model of the user base based on previous sessions).

Significance and Claims

The paper posits that in-context experiential learning—the ability to refine strategies by inferring shared uncertainty structures from past episodes—is a hallmark of intelligence that current frontier models lack.

Benchmark Contribution: BELA provides a uniquely scalable testbed (supporting up to 2 billion unique persona-choice set pairs) specifically designed to evaluate whether agents can exploit consistent latent preferences across episodes.
Diagnostic Value: The results highlight a critical deficiency in current state-of-the-art models: they cannot extract high-level information from past experiences to tailor their information-gathering to the observed task distribution.
Future Direction: The authors conclude that advancing experiential learning capabilities is essential for building agents capable of reliable decision-making in shifting real-world environments. They explicitly note that their work focuses on evaluation rather than training methods, leaving the development of models capable of such learning as a direction for future research.

The paper emphasizes that while the simulated customers are not intended to perfectly replicate real human behavior, the controlled consistency of the latent structures in BELA is necessary to rigorously test the specific capability of learning from experience, which is often confounded by noise in real-world data.