🤖 machine learning

LLM-SAA: LLM-persona Generated Distributions for Decision-making

This paper introduces and evaluates the "LLM-SAA" framework, demonstrating that distributions generated by LLM personas are practically effective for optimizing decisions in assortment, pricing, and newsvendor problems—particularly in low-data regimes—while cautioning that decision-agnostic metrics like Wasserstein distance can be misleading for assessing their utility.

Original authors: Jackie Baek, Yunhan Chen, Ziyu Chi, Will Ma

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jackie Baek, Yunhan Chen, Ziyu Chi, Will Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a business owner trying to set the perfect price for a new product, decide which items to put on a shelf, or figure out how much inventory to stock. To make the best decision, you usually need to know the "mood" of your customers: How much are they willing to pay? What do they like? How much will they buy?

Normally, you'd wait for real sales data to figure this out. But what if you're launching a brand-new product and have zero data? You're flying blind.

This paper introduces a new way to use Large Language Models (LLMs)—the smart AI chatbots we know today—to act as a "crystal ball" for these empty-data situations. They call this method LLM-SAA.

Here is the simple breakdown of what they did and what they found:

1. The Core Idea: The AI "Simulated Crowd"

Instead of waiting for real customers to show up, the researchers asked the AI to pretend to be a crowd of thousands of different people.

The Setup: They told the AI, "Here is a description of a sushi roll. Now, imagine you are a rich person in Tokyo, then imagine you are a student in Osaka. Tell me how much you'd pay for it."
The Result: The AI generates a huge list of fake answers. From this list, they build a "map" (a distribution) of what people might want.
The Decision: They then feed this AI-generated map into a standard math calculator to find the best price or inventory level.

2. The Big Question: Is the AI Map Accurate?

The researchers asked: Does it matter if the AI's fake map looks exactly like the real world map?

In the past, people judged AI by how closely its fake data matched real data (like checking if the AI's "average price" was the same as the real "average price"). The authors say this is the wrong way to judge it.

The Analogy:
Imagine you are trying to guess the winner of a horse race.

Old Way (Decision-Agnostic): You check if the AI's description of the horses (their speed, weight, color) matches the real horses perfectly.
New Way (Decision-Aware): You don't care if the description is perfect. You only care: Did the AI pick the winning horse?

The paper argues that an AI can make a "lucky guess" that leads to the right business decision even if its description of the world is slightly wrong. Conversely, an AI can describe the world perfectly but still give you a bad business recommendation.

3. The Three Tests

To prove their point, they tested the AI on three classic business problems:

The Sushi Shelf (Assortment): "Which 5 sushi rolls should we sell?" They used real data from Japanese sushi lovers as the "truth."
The Chocolate Price (Pricing): "How much extra should we charge for fancy chocolate?" They used real auction data from the Philippines.
The Fashion Stock (Newsvendor): "How many pairs of trousers should we order?" They used real sales data from H&M.

4. The Surprising Results

The AI is a Great "Low-Data" Hero: When the researchers had no real data (or very little), the AI-generated maps were incredibly useful. They helped businesses make much better decisions than just guessing randomly.
The "Persona" Trick: When they told the AI to pretend to be specific types of people (e.g., "a 20-year-old student"), the business decisions got even better.
- The Twist: The AI was actually terrible at imitating specific individuals. If you asked, "What would this specific person buy?", the AI often got it wrong.
- The Lesson: You don't need the AI to be a perfect actor for one person. You just need it to capture the vibe of the whole crowd. Even if the AI's "characters" are fake, the group they form is good enough to make smart business moves.
The "Distance" Trap: The paper found that standard math tools used to measure how "close" two data sets are (like Wasserstein distance) were misleading.
- Example: In the sushi test, a random guess (like rolling dice) actually looked "closer" to the real data than the AI did, according to standard math. But when it came to making the actual business decision, the AI crushed the dice roll. This proves that "statistical closeness" does not equal "good decision making."

5. The Bottom Line

If you are a business owner with a new product and no sales history yet:

Don't panic about the lack of data. You can use an AI to simulate a crowd of customers.
Don't worry if the AI isn't perfect. It doesn't need to mimic every single human perfectly to help you set the right price or stock the right shelves.
Ignore the fancy "closeness" scores. Just test if the AI's advice actually leads to better profits or outcomes.

In short: LLMs are not just chatbots; they are powerful tools for generating "what-if" scenarios that help businesses make smart moves even when they are flying blind.

Technical Summary: LLM-SAA: LLM-persona Generated Distributions for Decision-making

1. Problem Statement

In many operational decision-making problems (e.g., assortment optimization, pricing, inventory management), the optimal action depends on an unknown probability distribution $F$ of outcomes (e.g., customer preferences, willingness-to-pay, demand). While historical data is the standard source for estimating $F$ , it is often sparse, costly, or non-existent for new products and evolving markets.

Large Language Models (LLMs) offer a potential alternative by generating synthetic data or describing distributions based on world knowledge. The paper investigates the LLM-SAA (LLM Sample Average Approximation) pipeline: using an LLM to construct an estimated distribution $\hat{F}$ (via synthetic samples or descriptions) and then optimizing the decision based on $\hat{F}$ as if it were the ground truth.

The central challenge addressed is evaluation: How can we determine if an LLM-generated distribution $\hat{F}$ is "good" for decision-making when the true distribution $F$ is unknown? The authors argue that traditional metrics measuring statistical distance (e.g., Wasserstein distance) between $\hat{F}$ and $F$ are insufficient because a distribution can be statistically distant yet yield optimal decisions, or statistically close yet yield poor decisions.

2. Methodology

2.1 Decision-Aware Evaluation Metrics

The paper proposes evaluating $\hat{F}$ based on the quality of the resulting decisions rather than statistical alignment.

Competitive Ratio (CR): The performance of the action $\hat{a}$ (optimized under $\hat{F}$ ) is compared against the optimal action $a^*$ (optimized under the ground truth $F$ ).
$C_\theta(F, \hat{F}) := \frac{\min\{|R_\theta(a^*)|, |R_\theta(\hat{a})|\}}{\max\{|R_\theta(a^*)|, |R_\theta(\hat{a})|\}}$
where $R_\theta$ is the expected reward. A ratio closer to 1 indicates better decision quality.
Worst-Case Competitive Ratio (WorstCR): To avoid "lucky guesses" where $\hat{F}$ yields the right decision for the wrong reason, the authors identify the problem parameters $\theta$ (e.g., product costs, budget constraints) that minimize the competitive ratio. This involves solving an optimization problem where the adversary selects $\theta$ to make the LLM-based decision perform as poorly as possible relative to the optimal decision.
Average-Case Competitive Ratio (AvgCR): The expected competitive ratio over a distribution of problem parameters.

2.2 Generation Methods

The study evaluates various prompting strategies to generate $\hat{F}$ :

Sampling: The LLM generates individual outcomes (e.g., rankings, willingness-to-pay) repeatedly.
Persona-sampling: The LLM generates outcomes while imitating specific customer personas (e.g., demographics, preferences).
Batch-generation: The LLM generates a batch of outcomes in a single query.
Description: The LLM describes the distribution (e.g., mean, standard deviation, or probability mass points) rather than generating raw samples.

These methods are tested against baselines including Random (uniform distributions) and $d$ real data (empirical distributions from small sample sizes).

2.3 Problem Domains

The methodology is applied to three canonical operations research problems:

Assortment Optimization: Selecting a subset of items to offer to maximize expected utility based on customer rankings.
Pricing: Setting a price to maximize profit given a distribution of willingness-to-pay.
Newsvendor: Determining inventory levels to minimize expected overage and underage costs.

For each problem, the authors derive efficient algorithms to compute the WorstCR by exploiting the mathematical structure of the optimal decision rules (e.g., piecewise-linear profit functions in pricing, monotonicity in assortment).

3. Key Results

3.1 Performance in Low-Data Regimes

LLM-SAA vs. Random: LLM-generated distributions consistently outperform uninformed random baselines across all three problem domains.
LLM-SAA vs. Small Real Data: In low-data regimes (e.g., $d=5$ or $d=15$ real samples), LLM-SAA is often competitive with, and sometimes superior to, empirical distributions derived from limited real data.
Persona-Steering: While "Persona-sampling" did not accurately mimic individual-level responses (high PersonaMAE), it significantly improved downstream decision quality (high CR). This suggests that personas help the LLM capture the diversity of the population distribution rather than just individual fidelity.

3.2 Failure of Decision-Agnostic Metrics

The paper demonstrates that standard distributional distances are misleading proxies for decision quality:

Wasserstein/Kolmogorov Distances: In several experiments, the Random baseline achieved the lowest (best) Wasserstein distance compared to LLM methods, yet the Random baseline produced the worst decisions. Conversely, LLM methods with higher statistical distances often yielded near-optimal decisions.
Conclusion: Statistical closeness to the true distribution does not guarantee good decision-making; capturing specific structural features relevant to the optimization (e.g., sharp drops in survival functions for pricing) is more critical.

3.3 Specific Findings by Domain

Assortment: LLMs captured complex item affinities (e.g., substitutability) and "polarizing" items better than simple popularity scores, especially in "Hard" regimes where item sizes correlated with popularity.
Pricing: LLMs successfully replicated the human tendency to report round numbers (multiples of 5 or 0) in willingness-to-pay. This structural alignment allowed the LLM to identify optimal prices located at "drops" in the survival function, even if the overall distribution was not perfectly aligned.
Newsvendor: LLMs improved upon the worst-case performance significantly compared to random baselines, though average-case performance was comparable to random baselines in some settings.

4. Significance and Contributions

The paper makes three primary contributions:

LLM-SAA Pipeline: It formalizes the use of LLMs to generate distributions for downstream optimization in data-scarce environments, moving beyond using LLMs merely as decision-makers or survey simulators.
Decision-Aware Evaluation: It introduces a rigorous framework for evaluating LLM-generated distributions based on decision quality (competitive ratios) rather than statistical similarity. It provides theoretical algorithms to compute worst-case competitive ratios for specific operational problems.
Empirical Validation: It demonstrates that LLM-generated distributions are practically useful in low-data regimes, often outperforming uninformed baselines and competing with small real datasets. Crucially, it shows that high-fidelity individual simulation is not a prerequisite for good decision-making; capturing population-level diversity and structural features is sufficient.

The authors conclude that while LLM-SAA is promising, deployment requires domain-specific validation and fairness auditing, as biased distributions could lead to poor or unfair operational decisions. The work establishes a new benchmark for evaluating LLMs in operational decision-making contexts.