📊 statistics

Large Language Models for Market Research: A Data-augmentation Approach

This paper proposes a novel statistical data-augmentation framework that effectively integrates LLM-generated data with real human responses in conjoint analysis to reduce estimation error and costs, demonstrating that while LLMs cannot directly substitute human data due to inherent biases, they serve as a valuable complement when used within a robust statistical approach.

Original authors: Mengxin Wang (Naveen Jindal School of Management, The University of Texas at Dallas), Dennis J. Zhang (Olin School of Business, Washington University in St. Louis), Heng Zhang (W. P. Carey School of B

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengxin Wang (Naveen Jindal School of Management, The University of Texas at Dallas), Dennis J. Zhang (Olin School of Business, Washington University in St. Louis), Heng Zhang (W. P. Carey School of Business, Arizona State University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a product manager trying to figure out what features people want in a new sports car or a vaccine. Traditionally, you'd have to hire thousands of real people, pay them to take surveys, and wait weeks for the results. It's expensive, slow, and hard to scale.

Enter Large Language Models (LLMs) like the AI behind ChatGPT. These AIs are incredibly smart; they've read almost everything on the internet. You might think, "Why not just ask the AI what people would choose? It's free and instant!"

The Problem: The AI is a "Too-Smart" Student
The paper explains that while the AI is brilliant, it's not a perfect substitute for a real human.

The Analogy: Imagine you are trying to predict how a real person will react to a spicy pepper. You ask a brilliant student who has read every book on peppers but has never actually tasted one. The student might say, "I would definitely eat it because the book says it's healthy!" But a real human might say, "No way, it hurts my stomach."
The Reality: The AI often makes choices that are too logical, too rational, or just slightly "off" compared to real human quirks and emotions. If you simply replace your real survey data with AI data (or mix them together naively), your results will be biased. You'll end up designing a product that the AI loves, but real humans hate.

The Solution: The "AI Tutor" Method (Data Augmentation)
The authors propose a clever statistical trick called AI-Augmented Estimation (AAE). Instead of treating the AI as a replacement for humans, they treat it as a tutor that helps you learn faster.

Here is how it works, step-by-step:

The Small Class (Real Data): You gather a small group of real humans (say, 50 people) and ask them what they prefer. This is your "Ground Truth." It's accurate but expensive.
The Massive Library (AI Data): You ask the AI the same questions for thousands of different scenarios. The AI gives you a massive amount of data, but it's "fuzzy" or slightly wrong.
The Translation Layer (The Magic Step):
- The researchers use the small group of real humans to teach a statistical model: "Here is what the AI thinks (Input), and here is what the real human actually did (Output). Learn the difference."
- Think of this like a translator. The AI speaks "Robot Logic," and humans speak "Human Logic." The model learns the specific dialect differences between the two.
The Correction: Once the model understands the "Robot-to-Human" translation, it goes back to the massive library of AI data. It takes the AI's answers and translates them into what a human would have said.
The Result: You now have a dataset that is as big as the AI's library but as accurate as the real humans.

Why is this a Game-Changer?
The paper tested this on real-world problems, like choosing between different COVID-19 vaccines and picking sports cars.

The "Naive" Approach: If you just mixed real and AI data together, the results were often worse than just using the small group of real humans. The AI's bias dragged the accuracy down.
The "AAE" Approach: By using their translation method, they were able to reduce the amount of real human data needed by 25% to 80% while getting the same (or better) accuracy.

The Bottom Line
You don't need to fire your human survey team and replace them with robots. Instead, you can use the robots to do the heavy lifting of generating data, and then use a small, smart statistical filter to "correct" the robots' answers so they sound like humans.

It's like having a super-fast, infinite assistant who is great at brainstorming but bad at understanding human nuance. You don't let the assistant make the final decision; instead, you use a small team of experts to teach the assistant how to think like a human, and then let the assistant do the rest of the work.

Key Takeaway: AI data is not a substitute for human data; it's a powerful complement—but only if you know how to translate it correctly.

1. Problem Statement

The paper addresses a critical challenge in market research, specifically conjoint analysis (a method for understanding consumer preferences by evaluating trade-offs between product attributes).

The Bottleneck: Traditional data collection via human surveys is resource-intensive, costly, and slow to scale.
The Proposed Alternative: Large Language Models (LLMs) can generate synthetic consumer responses rapidly and cheaply.
The Core Issue: Recent studies indicate a significant distributional gap between LLM-generated data and real human data. LLMs often exhibit biases (e.g., impatience, lack of real-world experience) that cause them to deviate from actual human behavior.
The Failure of Naïve Approaches: Simply pooling LLM-generated data with real human data (naïve augmentation) or using LLM data alone often exacerbates estimation errors rather than reducing them. As more biased LLM data is added, the estimator drifts further from the ground truth.

2. Methodology: The AI-Augmented Estimator (AAE)

The authors propose a novel statistical data augmentation framework inspired by transfer learning and knowledge distillation. Instead of treating LLM outputs as direct labels, they treat them as "surrogate" labels that require correction.

The Two-Stage Estimation Process

The method utilizes two datasets:

Primary Data ( $m$ samples): Contains features ( $x$ ), real human labels ( $y$ ), and LLM-generated labels ( $z$ ). This is a small, expensive dataset.
Auxiliary Data ( $n$ samples): Contains features ( $x$ ) and LLM-generated labels ( $z$ ), but no real human labels ( $y$ ). This is a large, cheap dataset.

Step 1: Learning the Discrepancy (Mapping Function)
Using the small primary dataset, the model learns the conditional probability distribution of the real label given the features and the LLM label:
$P(y = j | x, z) = g_j(x, z; \theta^*)$
This is typically modeled using a neural network or a Multinomial Logit (MNL) model. The goal is to estimate the parameter $\hat{\theta}$ that captures how the LLM's "fuzzy" reasoning ( $z$ ) relates to human reasoning ( $y$ ) given the context ( $x$ ).

Step 2: Corrected Estimation
Using the large auxiliary dataset and the learned mapping $\hat{\theta}$ , the authors construct the AI-Augmented Estimator (AAE). Instead of maximizing the likelihood of the raw LLM labels, they maximize a corrected objective function:
$\hat{\beta}_{AAE} \in \arg \max_{\beta} \left\{ \frac{1}{n} \sum_{i=1}^n \sum_{j \in K} g_j(x_i, z_i; \hat{\theta}) \log \sigma_j(x_i; \beta) \right\}$
Here, $g_j(x_i, z_i; \hat{\theta})$ acts as a soft label (a probability distribution) that corrects the bias in the LLM output $z_i$ before it is used to estimate the true consumer preference parameters $\beta$ .

3. Key Contributions

Theoretical Contributions

Consistency and Asymptotic Normality: The authors prove that $\hat{\beta}_{AAE}$ is a consistent and asymptotically normal estimator of the true preference parameters $\beta^*$ , even when the LLM data is biased. This contrasts with naïve estimators which remain biased even as sample size approaches infinity.
Variance Reduction: Theoretically, the variance of the AAE is shown to be strictly lower than that of an estimator using only human data ( $\hat{\beta}_P$ ), provided the auxiliary dataset is sufficiently large and the LLM labels contain some information.
Finite-Sample Bounds: The paper derives high-probability finite-sample error bounds, demonstrating that the estimation error depends critically on the accuracy of the first-stage mapping (learning the discrepancy between AI and humans).
Comparison with PPI: The authors distinguish their method from Prediction-Powered Inference (PPI). While PPI corrects at the loss-function level, AAE corrects at the mapping-function level. Theoretically, AAE retains more information and outperforms PPI in high-dimensional feature spaces.

Empirical Contributions

Robustness Across Models: The framework was tested against various LLMs (GPT-3.5, GPT-4, GPT-4o) and prompting strategies (Basic, Chain-of-Thought, Few-Shot, Fine-tuning).
Superiority over Baselines: The AAE consistently outperformed:
- Human-only baselines (reducing error).
- AI-only baselines (which were highly biased).
- Naïve pooling baselines (which increased error).
- PPI and PPI++ baselines (especially in high-dimensional settings).

4. Key Results

The authors validated their approach using two empirical studies:

Study 1: COVID-19 Vaccine Preferences

Setup: A conjoint analysis with 11 features (efficacy, side effects, endorsements, etc.) based on a real survey of 2,000 US adults.
Performance:
- Error Reduction: AAE reduced estimation error (MAPE) significantly compared to using human data alone.
- Data Savings: The method allowed for a reduction in the required size of human surveys by 24.9% to 79.8% while maintaining the same level of accuracy.
- Prompt Sensitivity: Interestingly, while naïve approaches performed worse with more advanced models (like GPT-4 with Chain-of-Thought) due to "over-rationality," the AAE performed better with these advanced models because the underlying mapping between AI and humans became more learnable.

Study 2: Sports Car Choices

Setup: A robustness check using a dataset of 200 participants choosing sports cars with 5 attributes.
Performance: AAE again outperformed all baselines, achieving data savings ranging from 5.1% to 60.3%.
Dimensionality Effect: The study confirmed that AAE's advantage over PPI-based methods increases as the number of features (dimensionality) increases. In low-dimensional cases, they perform similarly; in high-dimensional cases, AAE is superior.

5. Significance and Implications

Paradigm Shift: The paper moves beyond the question of "Can LLMs replace humans?" to "How can we statistically integrate imperfect AI data with limited human data?"
Cost Efficiency: It provides a rigorous mathematical framework for market researchers to drastically cut the costs of data collection (recruiting participants) without sacrificing the statistical validity of their conjoint analysis.
Scalability: The approach is scalable. As LLMs improve, the "discrepancy" becomes easier to learn, making the AAE framework even more powerful in the future.
Generalizability: While focused on conjoint analysis, the authors note the framework is applicable to other statistical problems involving surrogate labels, including regression with continuous targets.

Conclusion: The paper demonstrates that LLM-generated data is not a direct substitute for human data but is a valuable complement when processed through a robust, two-stage statistical augmentation framework. This approach effectively mitigates AI bias, reduces estimation variance, and offers substantial economic savings for market research.