💬 NLP

LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text

This study demonstrates that GPT-4.1 can accurately infer baseball fans' overall experience ratings from unstructured text with high consistency, revealing that while the AI captures salient, emotionally intense moments, its systematic underestimation compared to self-reported scores reflects a fundamental difference between the model's focus on memorable events and the fans' holistic evaluative judgment.

Original authors: Jason Potteiger, Andrew Hong, Ito Zapata

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Potteiger, Andrew Hong, Ito Zapata

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a baseball team manager. You want to know if your fans had a good day. You ask them two questions after the game:

The Verdict: "On a scale of 0 to 10, how was your day?"
The Story: "Tell us why you gave that number."

Usually, you look at the number (the Verdict) to decide if you're doing well. But what if you could read the Story and guess the number before you even saw it? That's exactly what this paper did. They used a super-smart AI (a Large Language Model) to read thousands of fan stories and guess the score they would give.

Here is the simple breakdown of what they found, using some everyday analogies.

1. The "Crystal Ball" is Pretty Good

The AI read the text and guessed the score. It was surprisingly accurate.

The Result: About two-thirds of the time, the AI's guess was within just one point of the actual score the fan gave.
The Analogy: Imagine a weather forecaster trying to guess the temperature based on how people are dressed. If the fan says, "It was a bit chilly," the AI guesses 6/10. If the fan says, "It was perfect," the AI guesses 9/10. It gets the general vibe right most of the time.

2. The AI is a "Rock-Solid" Reader

The researchers ran the AI three times on the exact same stories to see if it would change its mind.

The Result: It almost never changed its mind. If it guessed a "7" the first time, it guessed a "7" the next two times.
The Analogy: Think of the AI like a very consistent judge. If you show a judge a painting three times in a row, they will give it the same score every time. They aren't guessing; they are reading the text with laser focus.

3. The Big Twist: The "Gap" is Real (and Useful)

Here is the most interesting part. The AI's guesses were systematically lower than the fans' actual scores.

The Scenario: A fan gives a 10/10 (Perfect!). But in the text, they write: "The game was great, but the parking cost $40 and the line for hot dogs was 40 minutes long."
The AI's Reaction: The AI reads the complaints about parking and hot dogs and guesses a 6 or 7.
The Reality: The fan did give a 10. But the AI didn't make a mistake. It just heard a different part of the story.

4. Why the Gap Exists: The "Verdict" vs. The "Highlight Reel"

The paper explains that the fan and the AI are doing two different mental jobs.

The Fan's Verdict (The Score): This is like a final grade for the whole day. It includes the excitement of the win, the love for the team, the memories of childhood, and the fact that you had a beer. It's a "gut feeling" that sums everything up, even the boring good parts that don't need mentioning.
The Fan's Story (The Text): This is like a highlight reel of the drama. Humans are wired to talk about what went wrong or what was weird. We don't usually write, "The seats were comfortable and the sun was nice." We write, "The parking was a nightmare!"
The AI's Job: The AI reads the "Highlight Reel of Drama." It sees the parking complaint and thinks, "Oh, this was a bad experience." It doesn't feel the fan's underlying love for the team that pushed the score to a 10.

The Metaphor:
Imagine you go on a date.

The Verdict (Score): You give it a 9/10. You had a great time overall.
The Story (Text): You tell your friend, "The waiter spilled soup on my shirt and the traffic was terrible."
The AI: Reads your story and says, "Sounds like a 4/10."

The AI isn't wrong about the soup or the traffic. You aren't wrong about the date. You are just reporting on different things.

5. Why This Matters for Teams

The paper argues that teams shouldn't try to fix the AI to make it match the score perfectly. Instead, they should use the gap between the two as a warning system.

If the Score is high (10) but the AI guess is low (6): This is "Fragile Satisfaction." The fans love the team, but they are annoyed by specific things (parking, prices, lines). They are forgiving now, but if things get worse, their love might turn into anger.
If the Score and AI match: The experience is consistent.

The Takeaway:
The AI is a tool that tells you what fans are talking about (the friction points). The survey score tells you how much they still love you.

The Score answers: "Are we doing okay?"
The AI Guess answers: "What is hurting our fans right now?"

By looking at both, a team can see that while fans are happy, they are getting annoyed by parking costs. If they only looked at the score, they would think everything is fine and miss the problem until it's too late.

Summary

The AI is a reliable, consistent reader that can guess a fan's mood based on their complaints and compliments. It doesn't always match the fan's final score because the fan's score includes their love for the team, while the AI only sees the specific problems the fan wrote down. That difference isn't an error; it's a secret signal that helps teams fix problems before fans stop coming.

1. Problem Statement

The paper addresses the challenge of extracting actionable insights from unstructured text feedback at scale. While organizations collect both numeric ratings (e.g., 0–10) and open-ended text responses, the text is often too voluminous for human analysis.

The Core Question: Can a Large Language Model (LLM) reliably predict a fan's self-reported overall experience rating (0–10) based solely on the text of their open-ended response?
The Hypothesis: There is a fundamental divergence between what fans write about (salient, memorable, or frustrating moments) and what they rate (a holistic, integrative verdict). The paper investigates whether an LLM can recover the numeric rating and, crucially, what the systematic gap between the predicted rating and the actual rating reveals about the nature of experience measurement.

2. Methodology

Data Source:

Sample: 9,977 post-game survey responses from fans of five Major League Baseball (MLB) teams during the 2025 season (March–June).
Input: Only the fan's open-ended text response explaining their rating.
Exclusions: The model received no demographic data, no aspect ratings (parking, concessions, etc.), no team records, and no game outcomes.

Model & Prompting:

Model: GPT-4.1.
Prompt Strategy: A deliberately simple, unoptimized, zero-shot prompt. The model was asked to "Estimate the fan's self-reported overall experience rating... Use ONLY evidence in the fan's text."
Design Rationale: The lack of calibration, few-shot examples, or domain-specific engineering was intentional to establish a baseline performance for the simplest viable approach.
Output Fields:
1. predicted_rating: Integer 0–10.
2. predicted_rating_confidence_score: High/Medium (based on textual clarity).
3. predicted_rating_evidence: A 15-word summary of the strongest textual evidence.

Validation Design:

Reproducibility: The prompt was run three times independently on the same dataset to assess stochasticity and measurement stability.
Metrics:
- Exact Match: % of predictions equal to the actual score.
- Within ±1: % of predictions within one point of the actual score.
- Mean Absolute Error (MAE): Average magnitude of error.
- Bias: Mean signed difference (Predicted − Actual).
- Correlation: Spearman rank-order correlation.

3. Key Results

A. Accuracy and Alignment

High Agreement: 67% of predictions fell within ±1 point of the self-reported rating; 36% were exact matches.
Rank Correlation: Strong rank-order agreement ( $r = 0.82$ ), indicating the model preserves the relative ordering of fan experiences.
Systematic Bias: The model consistently underpredicted ratings by approximately 1.0 point on average.
- Low Scores (0–2): Model overpredicted (generous).
- High Scores (9–10): Model underpredicted slightly.
- Mid-High Scores (7–8): Largest underprediction gap (~1.64 points). This corresponds to "fragile satisfaction" (fans rating high but writing about specific frictions).

B. Reliability (Stability)

Near-Deterministic: Across three independent runs, 87% of predictions were identical, and 99.9% were within ±1 point.
Implication: The measurement is stable; the gap between predicted and actual scores is not due to model noise or randomness but represents a systematic construct difference.

C. Construct Validity

Holistic vs. Salient: The predicted rating correlates most strongly with the overall experience score ( $r=0.82$ ), not with specific aspect ratings (e.g., parking, concessions).
No Single Driver: The underprediction gap is not driven by any single aspect (e.g., parking complaints). It is a broad phenomenon where text emphasizes "salient moments" (often negative or unusual) while the survey score represents an "integrated verdict."

D. Confidence Diagnostics

High Confidence: When the model flagged a prediction as "High Confidence" (text was unambiguous), accuracy jumped to 82% within ±1.
Medium Confidence: Accuracy dropped to 47% within ±1. These cases typically involved mixed sentiment (e.g., "Great game, but parking was terrible"), where the text does not resolve to a single clear score.

4. Key Contributions

1. Reframing the "Gap" as a Construct Difference
The paper argues that the discrepancy between the LLM's prediction and the fan's rating is not an error to be eliminated, but a signal to be preserved.

The Verdict (Survey Score): A holistic, integrative judgment that includes routine positives, team affinity, and the "experiencing self."
The Salience (Text/Prediction): A retrieval of specific, vivid, emotionally intense, or actionable moments (often negative) that the "remembering self" chooses to report.
Conclusion: The two numbers measure different cognitive operations. The gap reveals "fragile satisfaction"—fans who are happy overall but experiencing operational friction.

2. Validation of LLMs as Measurement Instruments
The study demonstrates that a simple, unoptimized LLM prompt can achieve reliability levels (87% exact agreement across runs) that exceed typical human intercoder reliability benchmarks ( $\kappa \approx 0.60–0.75$ ). This establishes LLMs as stable, deterministic tools for converting unstructured text into structured metrics.

3. Practical Framework for "Fragile Satisfaction"
The paper introduces a diagnostic framework where the divergence between predicted and actual scores serves as an early warning system. A widening gap indicates that while aggregate satisfaction remains high, specific operational frictions are becoming more salient in fan narratives, potentially threatening future ratings.

5. Significance and Implications

Operational Efficiency: Organizations can use a single scoring run (due to high stability) to convert massive volumes of text into comparable numeric scores without human coding.
Dual-Metric Strategy: Practitioners should not aim for perfect alignment between text-derived scores and survey scores. Instead, they should use both:
- Survey Score: Answers "How are we doing?" (The Verdict).
- Predicted Score: Answers "What are fans talking about?" (The Salience).
Early Warning System: Monitoring the gap between these two metrics allows organizations to detect emerging operational issues (e.g., parking, concessions) before they cause a drop in the overall satisfaction rating.
Generalizability: While tested on MLB data, the findings suggest that this construct divergence (salience vs. verdict) is likely universal across customer experience domains. The method is applicable to any context where text feedback exists without a paired numeric score (e.g., social media, support tickets).

Limitations Noted:

The study relies on a single model (GPT-4.1).
The "gold standard" is the self-reported rating, not human-coded text, though the authors argue human coders would likely exhibit similar divergence due to cognitive biases.
The sample is specific to sports; generalizability to other industries requires further testing.

In summary, the paper establishes that LLMs can reliably infer experience ratings from text, but the true value lies not in matching the survey score, but in leveraging the systematic divergence to uncover the specific, actionable moments that drive fan sentiment.