Transforming the Voice of the Customer: Large Language Models for Identifying Customer Needs

1. Problem Statement

The Challenge of Scaling Voice-of-the-Customer (VOC) Analysis:
Identifying Customer Needs (CNs) is fundamental to product innovation but remains a bottleneck in marketing research. Traditionally, VOC analysis relies on professional human analysts to manually interpret qualitative data (interview transcripts, online reviews) and abstract them into "jobs to be done."

• Limitations: This process is cognitively demanding, time-consuming, expensive, and difficult to scale.
• Current AI Gaps: While foundational Large Language Models (LLMs) excel at text generation, they struggle with the specific nuance of CN abstraction. Without specialized training, foundational LLMs tend to hallucinate, paraphrase customer opinions as needs, or suggest specific product solutions rather than abstract needs.
• The Core Question: Can Supervised Fine-Tuned (SFT) LLMs automate the abstraction of CNs to a level of quality comparable to, or exceeding, professional human analysts?

2. Methodology

The authors conducted five interrelated studies using a rigorous experimental design involving professional VOC data and blinded evaluations.

Data Sources:

• Training Data: 1,549 matched pairs of source verbatims (customer reviews/interviews) and professionally formulated CNs from 10 different product categories (e.g., activewear, glucose monitors).
• Negative Examples: 11,975 uninformative sentences to teach the model when not to generate a CN (outputting []).
• Evaluation Categories: The models were tested on out-of-sample categories not used in training: Wood Stain (Studies 1-3), Oral Care (Study 4), and Professional Services (Study 5).

Model Architecture & Training:

• Foundational Model: Primarily Vicuna 13B (open weights, academic license) was used as the baseline. Other models (Qwen3, DeepSeek, LLaMA variants) were tested in Study 2.
• Fine-Tuning Approach: The authors applied Supervised Fine-Tuning (SFT) using the instruction-tuning format (Question-Answer pairs).
  • Input: Source text + Product Category + Special Tag (<GPT-VOC>).
  • Output: Abstracted CN or [].
• Evaluation Protocol:
  • Human Evaluation (Study 1, 4, 5): Blinded professional analysts rated CNs on three dimensions: (1) Is it a valid CN? (2) Does it have sufficient detail? (3) Does it follow from the source (no hallucination)?
  • LLM-as-a-Judge (Study 2, 3): To scale evaluation, the authors trained a RoBERTa-based classifier on the human ratings from Study 1. This classifier was used to evaluate thousands of variations in model size, training data volume, and prompting strategies.

Study Breakdown:

• Study 1: Direct comparison of SFT LLM vs. Foundational LLM vs. Professional Analysts on Wood Stain data.
• Study 2: Boundary conditions: Impact of training data size (75–1,549 examples), model size (0.6B–14B parameters), and foundational architecture.
• Study 3: "Under the Hood" analysis: Investigating whether the model memorizes CNs or learns a transformation process.
• Study 4: Convergent validity test using Oral Care data (different methodology: CN-to-sentence mapping).
• Study 5: Application to a service category (PDMA membership) using interview data.

3. Key Contributions

1. Validation of SFT for CN Abstraction: The paper provides empirical evidence that SFT LLMs perform at least as well as professional analysts and significantly better than foundational LLMs in abstracting CNs.
2. Democratization of VOC: The research demonstrates that high-quality CN abstraction does not require massive models. Small models (e.g., 0.6B–1.7B parameters) fine-tuned on modest datasets (~600 examples) achieve performance comparable to 13B+ models.
3. Mechanism Discovery (Transformation vs. Memorization): The authors prove that SFT LLMs do not "memorize" specific customer needs. Instead, they learn a syntactic and semantic transformation process. They function as specialized "CN machines" that convert raw text into the specific professional format of a "job to be done," effectively forgetting general-purpose reasoning in favor of task-specific constraints.
4. Prompting vs. Fine-Tuning: The study establishes that while advanced few-shot prompting improves foundational LLMs, it cannot match the stability and performance of SFT. Fine-tuning is necessary for reliable, professional-grade output.

4. Key Results

Performance Metrics:

• Vs. Foundational LLMs: SFT LLMs significantly outperformed foundational models (e.g., Vicuna 13B) on all metrics. Foundational models failed to distinguish between opinions, solutions, and actual needs, often hallucinating or providing vague statements.
• Vs. Professional Analysts:
  • Validity: SFT LLMs were judged as "typical VOC CNs" at rates equal to or slightly higher than human analysts (e.g., 80%+ agreement).
  • Detail: SFT LLMs captured sufficient detail comparable to humans.
  • Grounding: SFT LLMs were less prone to hallucination than humans, with 91%+ of their outputs judged as strictly following the source material.
• Data Efficiency: Performance plateaued after 600 training examples. Adding more data yielded diminishing returns.
• Model Size: Performance was robust across model sizes. A 0.6B parameter model performed comparably to a 14B model after fine-tuning.

Strategic Impact (Affinity Diagrams):

• In the Wood Stain study, the SFT LLM identified CNs covering 100% of primary and secondary categories, whereas human analysts missed one primary category (maintenance ease).
• The SFT LLM scaled to process the full corpus (14,341 sentences) efficiently, whereas human analysis was limited to a sample due to cost.

Failure Modes:

• Foundational LLMs: Frequently produced "solutions" (e.g., "add a timer"), "opinions" (e.g., "battery is bad"), or "vague goals."
• SFT LLMs: Rarely made these errors. The fine-tuning successfully instilled constraints to avoid solutions, processes, and opinions, adhering strictly to the "need" syntax.

5. Significance and Managerial Implications

• Transformation of the Analyst Role: The role of the professional analyst shifts from extracting needs (tedious, low-value) to winnowing and validating needs (high-value). Analysts can now focus on strategic interpretation rather than manual abstraction.
• Scalability: Firms can now analyze massive datasets (e.g., millions of reviews) to discover "high-leverage" needs that were previously hidden due to sampling limitations.
• Cost Reduction: VOC studies become accessible to smaller firms and entrepreneurs, as the cost barrier of hiring armies of analysts is removed.
• Service Sector Applicability: The success in the PDMA (service) study confirms that this approach is not limited to physical products but applies to service design and membership organizations.
• Future Outlook: While CN abstraction is now automatable, the paper notes that prioritization of needs remains a human-centric challenge. The next frontier is using these standardized, high-quality CNs to enable automated affinity diagramming and continuous monitoring of emerging needs.

Conclusion:
The paper concludes that Supervised Fine-Tuning transforms LLMs from general-purpose chatbots into specialized, professional-grade tools for marketing research. By learning the specific "language" of customer needs, these models automate the most tedious part of the VOC process, enabling faster, more comprehensive, and more accurate product innovation strategies.