💬 NLP

Dimensionality in Satisfaction Ratings

This paper demonstrates that using a large language model to decompose customer satisfaction into specific axes (such as agent performance and outcome) provides a more nuanced and comprehensive understanding of customer experience than traditional survey-based metrics, revealing significantly lower satisfaction levels when analyzing all support conversations rather than just self-selected survey respondents.

Original authors: Andrew Hong, Jason Potteiger

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrew Hong, Jason Potteiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a giant lemonade stand. Every time someone buys a cup, you ask them, "How was it?" Most people just keep walking, but a few stop to give you a star rating. You've been building your entire business strategy on the opinions of those few stoppers, assuming they represent everyone.

This paper is like a detective story where the researchers bring in a super-smart robot (an AI called GPT-4.1) to read the actual conversations between the customers and the lemonade stand staff. The robot doesn't just guess a star rating; it breaks the experience down into five specific ingredients: Overall happiness, Agent kindness, Outcome (did they get their lemonade?), Product quality (was the lemon fresh?), and Effort (how hard did the customer have to work?).

Here is what the robot found, and why it changes everything we thought we knew about customer happiness.

The Big Reveal: The "Silent Majority" is Unhappy

The most shocking discovery is that the people who stop to give you a rating are not a fair sample of everyone.

The Rating Crowd: The few people who actually stopped to rate the stand gave an average score of 3.62 out of 5.
The Silent Crowd: When the robot read the transcripts of everyone (including the 8,343 people who just walked away without rating), the average score was only 2.91.

The paper argues that relying only on the people who fill out surveys is like judging a whole movie based only on the reviews from the people who loved it. The "silent majority" is actually having a much worse time than the surveys suggest. The robot's ability to read every single conversation, even the ones without a rating, reveals a hidden gap of 0.71 points that surveys completely miss.

The Robot's Scorecard: Good at Some Things, Weak at Others

The researchers tested if the robot's breakdown of the experience matched what customers actually said about themselves.

The Winners: The robot was pretty good at guessing the Overall feeling, the Agent's performance, and the Outcome (did the problem get fixed?). These matched the customers' own ratings with a correlation of about 0.65. If you look only at the cases where the robot and the customer agreed closely, that number jumps to 0.811, and if you ignore the few weird outliers, it hits 0.914.
The Loser: The robot struggled to guess Product satisfaction. It couldn't tell if the customer liked the lemonade itself just by reading the chat. The paper suggests this isn't necessarily because the robot is bad, but because the survey questions used to check it were too "noisy" (like asking if they'd buy it again, which depends on price and habit, not just the chat). So, the product score is still a "maybe" and needs more testing.
The Effort Twist: The robot measured how much "effort" the customer had to put in. This score went down when satisfaction went up (a correlation of -0.54), which makes sense: the harder you have to work, the less happy you are.

The "Magic Trick" That Didn't Work (But That's Okay)

The researchers asked a tricky question: "If we add up all five ingredients (Agent + Outcome + Product + Effort), can we predict the customer's final star rating better than just asking the robot for an 'Overall' score?"

The answer is no.
The paper explicitly rules out the idea that breaking things down helps you predict the final score better. The five ingredients are so similar to each other (they move in lockstep) that adding them together doesn't give you any new information. It's like trying to predict the weather by measuring temperature, humidity, and wind speed separately, then adding them up—it doesn't tell you more than just looking at the sky.

So, what is the point of the breakdown?
The value isn't in predicting the number; it's in understanding the story.

The "Blended" Trap: A single star rating hides the truth. You might get a 4-star rating because the Agent was amazing (5 stars), even though the Product was terrible (1 star).
The "Census" Power: Because the robot reads every conversation, it can spot these hidden patterns. It found that in about 4.8% of cases, the agent was great but the product was broken. A single survey rating would have just said "4 stars," hiding the fact that the product is the real problem. The breakdown acts like an X-ray, showing exactly where the experience is broken, not just that it is broken.

Why the Robot and the Customer Sometimes Disagree

The robot and the customers didn't always agree. In about 83 specific cases, their scores were off by two or more points. The paper didn't just shrug this off; it read every single transcript to figure out why.

The "Polite but Empty" Trap (Overprediction): Sometimes the robot gave a high score because the conversation was polite and flowed smoothly, even though the customer's actual problem wasn't solved. The robot saw the "manner" but missed the "mission."
The "Helpful Handoff" Trap (Underprediction): Sometimes the robot gave a low score because the problem wasn't fixed in the chat. But the customer gave a high rating because the agent was so helpful they routed them to a human or a case number. The customer was happy with the process, but the robot was scoring the unresolved outcome.

The paper suggests that if the robot were taught to look for "task completion" (did the specific job get done?), it could fix most of these errors. The disagreements aren't random noise; they are a map of what the robot needs to learn next.

The Bottom Line

This paper doesn't claim to have solved everything. It admits the robot was only tested on the "happy" customers who left ratings, so we aren't 100% sure how it performs on the very unhappy ones. It also admits the "Product" score is shaky.

However, the main finding is solid: Surveys lie by omission. They only hear from the satisfied few. By using AI to read every single conversation, we can finally see the full picture, which is much less happy than we thought. The real power isn't in getting a perfect prediction of a star rating; it's in having a full census of why people are unhappy, allowing companies to fix the specific broken parts of their service instead of just guessing.

Technical Summary: Dimensionality in Satisfaction Ratings

Problem Statement
Customer-care organizations face a fundamental asymmetry: they generate vast amounts of conversational data (transcripts) but rely on a tiny, self-selected minority of customers who voluntarily provide satisfaction ratings (typically <6%). This reliance on survey data introduces unquantifiable nonresponse bias, as the customers who rate are systematically more satisfied than those who do not. Furthermore, a single overall satisfaction rating compresses a complex interaction—encompassing agent behavior, resolution outcomes, product quality, and customer effort—into a holistic verdict, obscuring the specific drivers of the experience. The paper asks whether Large Language Models (LLMs) can act as annotation instruments to decompose satisfaction into component axes across the entire population of contacts, and whether such a decomposition can be validated against self-reported data.

Methodology
The study utilized a global consumer-goods firm's dataset of 8,880 distinct support sessions (automated chat and voice calls) over a 30-day period. Only 537 sessions (6%) contained a customer-provided satisfaction rating.

Annotation Instrument: A single LLM (GPT-4.1) was applied via a prompt function to read each transcript independently. The model performed recognition, not generation, mapping satisfaction-bearing language to a structured schema.
Decomposition Structure: The model emitted five axes on a 1–5 scale:
1. Overall Satisfaction: The holistic session score.
2. Agent Satisfaction: Handling quality, isolated from the problem.
3. Outcome Satisfaction: Resolution status, isolated from agent demeanor.
4. Product Satisfaction: Satisfaction with the product/service itself.
5. Predicted Effort: Customer effort required (inverted polarity, where 5 = highest effort/worst experience).
Validation Design: The study employed convergent and discriminant validity frameworks (Campbell & Fiske, 1959) across three distinct cohorts:
1. In-Chat Base (n=432): Sessions with native in-chat ratings, used to test overall axis alignment.
2. Survey Battery (n=92 cases): Sessions with detailed post-contact surveys, used to test dimension-specific alignment (e.g., outcome axis vs. resolution item).
3. The Census (n=8,880): The full population, used to measure the annotation's ability to reveal nonresponse bias and provide full-coverage attribution.

Key Results

Convergent Validity: Four of the five axes track self-reported satisfaction closely. The overall, agent, and outcome axes correlate with self-reported ratings at approximately 0.65 (Spearman $\rho$ ). The effort axis correlates at −0.54, confirming it behaves as a distinct, loyalty-relevant construct rather than a re-labeled sentiment score. The product axis showed weak correlation (0.06) against survey proxies (recommendation intent), attributed to the proxies measuring downstream attitudes rather than immediate contact satisfaction.
Discriminant Validity & Collinearity: The component axes are highly collinear (inter-axis correlations ranging from 0.84 to 0.94). Consequently, adding component axes to the overall score does not improve the prediction of the customer's rating; the decomposition offers no incremental predictive power over the overall axis alone.
The Divergent Tail: The 0.65 correlation understates alignment because disagreements are concentrated in a small, readable tail (83 sessions differing by $\ge$ $\geq$ 2 points). Excluding these divergent cases raises the correlation to 0.811 (severe divergences excluded) and 0.914 (full tail excluded).
- Overprediction: Occurs when the model scores the polite, fluent surface of a conversation while the customer's specific task remains incomplete.
- Underprediction: Occurs when the model scores an unresolved outcome negatively, while the customer rates the contact highly for a successful handoff, escalation, or process capture.
Nonresponse Bias: When applied to the full census, the annotation reveals a stark gap between surveyed and unsurveyed populations. The surveyed minority averages a predicted satisfaction of 3.62, while the unsurveyed majority averages 2.91. This 0.71-point gap quantifies the nonresponse bias inherent in traditional survey programs.
Attribution Value: While the axes do not improve prediction, they provide attribution. The decomposition identifies dominant drivers (e.g., resolution, agent handling, product quality) for every contact, not just the surveyed ones. It also isolates specific patterns, such as "capable agent, faulty product" scenarios, which a blended score would obscure.

Significance and Claims
The paper claims three primary contributions:

Methodological: It demonstrates that a five-part decomposition can be validated using convergent and discriminant tests, moving beyond simple accuracy against a single label to assess whether the axes measure separable constructs.
Diagnostic: It provides a granular reading of the "divergent tail," showing that model-customer disagreements are not random noise but stem from identifiable frame mismatches (e.g., task completion vs. conversational polish).
Substantive: It quantifies survey nonresponse bias by applying the validated annotation to the unsurveyed majority, revealing that the unsurveyed population is meaningfully less satisfied than the self-selected respondents.

The authors emphasize that the value of this decomposition is not in predicting the customer's rating more accurately than the overall axis (which it does not), but in attribution and coverage. It transforms satisfaction measurement from a "scorecard" of the self-selected few into a "diagnosis" of the entire population, identifying where dissatisfaction concentrates and what drives it across all contacts.

Limitations
The study acknowledges several constraints:

Validation Range: Validation occurred only on the satisfied minority (those who rated), meaning the annotation's performance on the lower end of the satisfaction scale (the unsurveyed majority) is an out-of-sample extension.
Single Model/Prompt: All results are conditional on GPT-4.1 and a specific prompt design; no run-to-run reliability ceiling was established.
Product Axis: The product axis remains provisional due to the lack of a clean survey proxy for immediate product quality.
Task Completion: The analysis identifies that the current decomposition lacks a specific "task completion" axis, which explains a significant portion of overprediction errors.

The Big Reveal: The "Silent Majority" is Unhappy

The Robot's Scorecard: Good at Some Things, Weak at Others

The "Magic Trick" That Didn't Work (But That's Okay)

Why the Robot and the Customer Sometimes Disagree

The Bottom Line

More like this