💬 NLP

Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games

This paper demonstrates that Large Language Models can effectively substitute for human subjects in generating training data for economic choice prediction within language-based persuasion games, often outperforming models trained on actual human data while revealing that capturing history-dependent decision patterns is crucial for accurate prediction.

Original authors: Eilam Shapira, Omer Madmon, Roi Reichart, Moshe Tennenholtz

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Eilam Shapira, Omer Madmon, Roi Reichart, Moshe Tennenholtz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital economy, every click and every scroll is a choice. When a traveler reads a hotel review on a booking site, they are weighing information against their own desires, trying to decide if a room is worth the price. This is a classic economic puzzle: how do people decide when faced with incomplete information and the possibility of being misled? For decades, economists have studied these moments by asking real humans to play games where one person tries to persuade another. These experiments are difficult to run; they require recruiting participants, building custom software, and waiting for people to make decisions, a process that is slow, expensive, and often limited in size.

Recently, a new tool has emerged that promises to change how we study these decisions: the large language model. These are artificial intelligence systems trained on vast amounts of text, capable of understanding and generating human language with remarkable fluency. Researchers have begun to wonder if these machines can do more than just chat. Could they act as stand-ins for real people in economic experiments? If a computer can mimic human behavior well enough, it might generate the massive amounts of data needed to train better prediction models, solving the problem of scarce human data. The question is not just whether machines can talk like humans, but whether they can think and decide like them in high-stakes situations.

A team of researchers at the Technion in Israel set out to answer this question by placing artificial intelligence agents into a complex game of persuasion. In their setup, one player, acting as an expert, tries to convince a second player, the decision-maker, to visit a specific hotel. The expert knows the true quality of the hotel but can only share a single written review from a list of options. The decision-maker must then choose whether to visit the hotel based on that review and their past experiences with the expert. If the hotel is actually good, the decision-maker wins; if it is bad, they lose. The expert, however, always wins if the decision-maker chooses to visit, regardless of the hotel's true quality. This creates a strategic tension where the expert might lie to get a better outcome, and the decision-maker must learn to spot the deception over time.

The researchers first gathered data from hundreds of real human players who engaged in this game over many rounds. They then replaced the human decision-makers with various large language models, instructing the machines to play the game using the same rules and facing the same hotel reviews. To make the simulations more realistic, the researchers gave each machine a specific personality, such as an optimistic traveler who always sees the bright side, or a price-sensitive shopper who cares deeply about cost. By running thousands of these simulated games, the team created a massive dataset of decisions made by machines, which they used to train a computer program to predict how a human would behave in the same situation.

The results were surprising. When the researchers trained a prediction model using only the data generated by the artificial intelligence players, the model became better at predicting human behavior than a model trained on the actual human data itself, provided the machine dataset was large enough. In fact, the machine-generated data outperformed the human data in almost every scenario tested. This suggests that the artificial intelligence agents were not just mimicking the words of humans, but were capturing the deeper strategic patterns of how people learn and adapt during repeated interactions. The machines learned to weigh the history of past reviews and the trust built over time, rather than just reacting to the immediate words on the screen.

However, the study also revealed a significant flaw in relying solely on machines. While the models trained on artificial data were highly accurate, they were poorly calibrated. In simple terms, this means that when the model said it was confident in a prediction, it was often wrong more often than a model trained on real humans would be. The machine data was so abundant that it taught the model the right answer most of the time, but it failed to teach the model how to gauge its own certainty. To fix this, the researchers found that mixing a small amount of real human data with the vast amount of machine data created the perfect balance. This hybrid approach produced a model that was both highly accurate and trustworthy in its confidence levels.

The researchers dug deeper to understand why the machines were so successful. They discovered that the key to predicting human behavior was not the sentiment of the review itself, but the history of the interaction. When the artificial intelligence agents were allowed to remember many past rounds of the game, they generated data that closely matched human decision-making patterns. But when their memory was restricted to just the last one or two turns, their ability to simulate human behavior collapsed. This finding highlights that human decision-making in these games is fundamentally about strategy and history, not just about reading the emotional tone of a single message. The machines succeeded because they learned to play the long game, just like people do.

To ensure their findings were not limited to just hotel reviews, the team tested their approach in a different, more abstract persuasion game where players could write any message they wanted, rather than choosing from a fixed list. Even in this freer environment, where the rules were less rigid, the data generated by artificial intelligence players proved effective at training models to predict human choices. This suggests that the ability of machines to generate useful training data is a robust phenomenon that extends beyond a single type of experiment.

The study concludes that while artificial intelligence cannot yet perfectly replace the nuance of human behavior, it can serve as a powerful engine for generating the data needed to understand it. By using machines to simulate the strategic complexities of human interaction, researchers can overcome the bottlenecks of traditional economic experiments. The path forward involves a careful partnership: using the scale of artificial intelligence to build broad models of behavior, while grounding those models with just enough real human data to ensure they remain reliable and true to life. This approach offers a new way to study the intricate dance of trust, deception, and decision-making that defines our economic world.

Technical Summary: Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games

1. Problem Statement

Human choice prediction in economic contexts is vital for applications in marketing, finance, and public policy. However, acquiring high-quality human choice data is often constrained by high costs, privacy concerns, legal hurdles, and the inefficiency of experimental setups (e.g., participant inattention). While Large Language Models (LLMs) have shown promise in simulating human behavior in strategic environments, a critical open question remains: Can LLMs generate synthetic training data that effectively predicts human decision-making in complex, language-based economic settings?

This paper investigates this question within the framework of language-based persuasion games. Unlike traditional economic models that abstract messages into simple signals, these games involve natural language interactions where a sender (expert) attempts to persuade a decision-maker (DM) to accept an offer (e.g., a hotel) based on private information (hotel quality). The complexity arises from the interplay between linguistic signals (textual reviews) and strategic interaction history (repeated rounds of trust, cooperation, or punishment).

2. Methodology

Experimental Setup

The study utilizes the language-based persuasion game introduced by Apel et al. (2022) and Shapira et al. (2025).

Game Mechanics: An expert (sender) observes a set of textual reviews and numerical scores for a hotel. The expert's private information is the hotel's true quality (High/Low). The expert selects one review to send to a Decision-Maker (DM). The DM decides whether to "Go" (accept) or "Stay Home" (reject).
Payoffs: The expert gains utility if the DM accepts, regardless of quality. The DM gains utility only if the hotel is high quality and they accept.
Strategies: The study employs a representative set of expert strategies, including Greedy (always sends the best review), Honest (sends best/worst based on true quality), and Adaptive strategies that depend on interaction history.
Data Sources:
- Human Data: A dataset of 71,579 decisions from 210 human players interacting with bot experts (Shapira et al., 2025).
- LLM Data: Synthetic datasets generated by replacing human DMs with various LLMs (Qwen-2-72B, Llama-3-70B/8B, Gemini-1.5, Chat-Bison). To diversify behavior, LLMs were assigned personas (e.g., Optimistic, Pessimistic, Price-sensitive) via prompt prefixes.
- Baseline: A "Sentiment Baseline" where agents make decisions based solely on the sentiment of the current review, ignoring interaction history.

Prediction Framework

The core task is to train a machine learning model to predict the DM's action ( $a_t \in \{0, 1\}$ ) given the interaction history and the current review.

Models: The study evaluates four architectures: LSTM, Mamba, Transformer, and XGBoost.
Input Features: Engineered Features (EFs) capturing topical, structural, and stylistic aspects of reviews, along with strategic context features.
Evaluation: Models are tested on a held-out set of 100 human players. Performance is measured by Accuracy and Expected Calibration Error (ECE).

Advanced Integration Strategies

The paper explores several ways to integrate LLMs into the prediction pipeline:

LLM as Data Generator: Training simple predictors on LLM-generated data.
LLM as Predictor: Using the LLM directly (zero-shot or fine-tuned) to predict human actions.
Fine-Tuning: Fine-tuning an LLM on human data to improve its data generation capabilities.
DUAL (Double Use of human data for Augmented Learning): Fine-tuning an LLM on human data, generating synthetic data, and then training a simple predictor on a mixture of the original human data and the new synthetic data.

3. Key Contributions and Results

A. LLMs as Effective Data Generators

Superiority at Scale: Prediction models trained on LLM-generated data (specifically using Qwen-2-72B) outperform models trained on actual human data when the synthetic sample size is sufficiently large (e.g., >1,000 LLM players).
Strategic Understanding: LLM-generated data significantly outperforms the "Sentiment Baseline." This indicates that successful prediction requires modeling strategic behavior (history-dependent decisions) rather than just linguistic sentiment analysis.
Persona Diversification: Assigning diverse personas to LLM agents reduces the sample complexity required to achieve high accuracy, suggesting that behavioral diversity in synthetic data is crucial.

B. The Calibration Trade-off

Accuracy vs. Calibration: While LLM-only models achieve higher accuracy, they suffer from poor calibration (higher ECE) compared to models trained on human data. The confidence levels of LLM-trained models do not align as well with empirical outcomes.
The DUAL Solution: The DUAL approach (mixing fine-tuned LLM-generated data with the original human data) achieves the best of both worlds: it maintains the high accuracy of the synthetic data while significantly improving calibration (nearly halving the ECE compared to using synthetic data alone).

C. The Critical Role of Interaction History

History vs. Sentiment: An analysis of behavioral similarity reveals a counter-intuitive finding:
- History-based similarity (how closely an LLM's decision patterns match humans regarding past interactions) is positively correlated with predictive accuracy.
- Sentiment-based similarity is negatively correlated with predictive accuracy.
Implication: Human decision-making in these games is fundamentally a strategic, history-dependent task. Models that capture long-term interaction patterns (trust, punishment) are effective, whereas those focusing solely on the immediate linguistic signal (sentiment) are less effective, even if they mimic human sentiment.
Memory Length: Restricting the LLM's visible interaction history (e.g., to 1 or 2 turns) degrades the quality of the generated data, confirming that long-term temporal context is essential for simulating human-like strategic behavior.

D. Generalizability

The findings were validated on a different persuasion framework (GLEE), which involves free-form textual messages and varied game parameters. Models trained on LLM-vs-LLM data in GLEE also outperformed those trained on limited human data, suggesting the approach generalizes beyond the specific "hotel" scenario.

4. Significance and Claims

The paper claims that LLMs can serve as a cost-effective, scalable, and high-quality alternative to traditional human data collection for training human choice prediction models in complex economic settings.

Paradigm Shift: The study challenges the notion that synthetic data is merely a stopgap. It demonstrates that with sufficient volume and appropriate behavioral modeling (personas, history), LLM-generated data can surpass human data in predictive power.
Methodological Insight: It highlights that the "human-likeness" required for prediction is not about mimicking surface-level sentiment but about capturing the strategic dynamics of repeated interactions.
Practical Utility: The DUAL strategy offers a robust solution for data-scarce environments, allowing researchers to leverage the volume of synthetic data while grounding the model's confidence with a small amount of real human data.
Ethical Note: The authors acknowledge that while this approach alleviates privacy and cost issues in experimental economics, the ability to accurately predict human behavior using AI also carries risks of misuse, necessitating ethical guidelines.

In conclusion, the paper posits that LLMs can effectively replace human choice prediction labs for training purposes, provided that the synthetic data generation process explicitly models the strategic, history-dependent nature of human decision-making.