What sentiment analysis can't see: Measuring whether customers were helped, and what went wrong, across 70,000 support conversations

Technical Summary: What Sentiment Analysis Can't See

Problem Statement
The paper addresses a fundamental construct validity gap in customer support analytics: the conflation of tone (sentiment) with outcome (satisfaction). Organizations widely rely on sentiment analysis to measure customer experience, treating the affective polarity of a customer's language as a proxy for whether the interaction was successful. The authors argue that this approach measures "how customers sound" rather than "whether they were helped." This distinction is critical because customers often report civil, neutral, or even positive language while experiencing significant friction, or conversely, express frustration while achieving a satisfactory resolution. The paper posits that sentiment analysis compresses complex operational realities into a single dimension, failing to distinguish between a "quietly satisfied" customer and one who has "quietly given up," or to surface "tolerated friction"—fixable problems absorbed by satisfied customers that never trigger a dissatisfaction alarm.

Methodology
The study utilizes a dataset of 70,450 customer support conversations from a leading online fundraising platform, exchanged via Intercom/Fin between November 2025 and April 2026. The corpus was filtered to include only genuine, multi-turn exchanges (minimum three messages).

The core methodology involves a comparative validation of three distinct readings derived from the same text using independent GPT-5.4 prompt functions:

1. Sentiment: A conventional three-category label (Positive, Neutral, Negative) capturing the customer's tone.
2. Predicted Satisfaction: A holistic 1-to-5 scalar estimate of the customer's satisfaction with the interaction, derived solely from the conversation text.
3. Issue Presence: A Boolean flag indicating whether the customer reported a concrete, fixable problem.

These model-generated readings were validated against the "Verdict": the self-reported 1-to-5 rating customers left on the 5.55% of conversations (N = 3,911) that included a rating. The study employed skew-robust metrics (precision, recall, balanced accuracy, Matthews correlation coefficient) to account for the heavy positive skew in the rating data (where 91% of ratings are favorable). Crucially, the prompt functions were run independently to ensure that disagreements between readings reflected distinct measurements of the construct rather than shared anchoring within a single prompt.

Key Results
The study yields several quantitative and qualitative findings:

• Superior Validity of Predicted Satisfaction: Predicted satisfaction correlates with the customer's self-reported rating at 0.47, significantly outperforming sentiment's correlation of 0.36. In detecting low ratings (1–3), predicted satisfaction achieves a precision of 0.51 compared to sentiment's 0.29, meaning sentiment flags are correct less than a third of the time when they indicate a problem.
• High Divergence Rate: Sentiment and predicted satisfaction disagree in 44% of all conversations. This disagreement is not symmetric noise; it is heavily concentrated in cases where the tone is Neutral or Negative, but the predicted satisfaction is high (e.g., a customer politely explaining a refund process).
• The "Tolerated Friction" Segment: By crossing the axes of State (Satisfied/Unsatisfied) and Cause (Issue Present/Absent), the authors identify four distinct operational states. The largest segment (43,704 conversations) consists of customers who are Satisfied but reported an Issue. These customers sound neutral or positive but are reporting fixable product or workflow problems. Sentiment analysis collapses this group into a "positive" bucket, rendering the friction invisible.
• Neutral is Not a Single State: The "Neutral" sentiment label (covering 28,276 conversations) masks a wide range of operational realities, from customers who are quietly satisfied to those who are quietly dissatisfied. The structured read separates these groups, revealing that "Neutral" customers with issues have significantly lower average ratings (3.3) than "Neutral" customers without issues (4.3).
• Scale of the Blind Spot: Because the structured measure applies to all 70,000+ conversations, it exposes risk in the 94.45% of interactions that are never rated. The authors estimate that billions of dollars in processed volume sit behind unrated conversations containing fixable issues that sentiment dashboards would miss.

Significance and Claims
The paper claims that LLM-based annotation can move beyond the limitations of document-level polarity to capture the customer's state and the operational cause of their contact. The primary significance is not merely that a new model performs better, but that the current default metric (sentiment) is measuring the wrong construct for operational decision-making.

The authors argue that a structured read recovers what sentiment compresses:

1. Actionability: It transforms a monitoring signal into a worklist by identifying specific, fixable problems (via the issue flag) rather than just mood.
2. Risk Visibility: It exposes "tolerated friction"—a standing inventory of fixable problems that satisfied customers absorb—which is the largest source of operational risk but is invisible to tone-based dashboards.
3. Construct Validity: It demonstrates that a text-derived estimate of satisfaction tracks the customer's own verdict more faithfully than tone does, validating the shift from "polarity" to "state and cause" as the standard for customer experience measurement.

The paper concludes that while sentiment is not worthless, relying on it as a proxy for satisfaction leads to false alarms and missed risks. The proposed structured approach offers a more valid, granular, and actionable foundation for monitoring support operations at scale.