The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text

1. Problem Statement

The paper addresses the challenge of using Large Language Models (LLMs) to predict self-reported experience ratings (0–10 scale) based solely on open-ended survey text. While previous research established that unoptimized LLMs can predict ratings within one point of the ground truth 67% of the time, a significant "residual gap" remains.

The core problem is determining the source of this gap: Is it a measurement error (fixable via better engineering/prompts) or a construct difference (an inherent divergence between what fans write and how they rate)? The authors aim to decompose this "ceiling" of accuracy to understand how much prompt design and model selection can actually improve alignment, and where the limits lie.

2. Methodology

The study utilized a two-factor experimental design on a dataset of approximately 10,000 post-game surveys from five Major League Baseball (MLB) teams (March–June 2025).

• Data: Open-ended text responses paired with self-reported "overall experience" ratings (0–10).
• Task: The LLM received only the open-ended text and was tasked with predicting the integer rating the fan would have given.
• Experimental Configurations: Four combinations were tested:
  1. Baseline Prompt + GPT 4.1 (Unoptimized generic prompt).
  2. Customized Prompt + GPT 4.1 (Added a 3-step decision process and a rationale field).
  3. Customized Prompt + GPT 4.1-mini (Smaller, cheaper model).
  4. Customized Prompt + GPT 5.2 (Newer, more advanced model).
• Metrics:
  • Exact Match: Percentage of predictions matching the score exactly.
  • Within ±1: Percentage of predictions within one point of the ground truth.
  • Mean Absolute Error (MAE): Average magnitude of error.
  • Mean Signed Error: Directional bias (over- vs. under-prediction).
• Validation: The study analyzed alignment across different text types (positive-only, negative-only, mixed, no-lexicon hits) and score bands.

3. Key Contributions

The paper makes three primary theoretical and practical contributions:

1. Decomposition of the "Ceiling": The authors argue that the accuracy ceiling is composed of two distinct parts:
   • The Measurement Component (Correctable): An instrument bias where LLMs, trained on review data, over-weight negative surface lexicon (negativity bias). This can be mitigated via prompt engineering.
   • The Cognitive Component (Invariant): A fundamental construct difference where open-ended text captures salient incidents (what stood out), while the rating scale captures integrative judgment (peak-end rule and affect). The information required to bridge this gap (how the fan weighted specific incidents against the whole) is not present in the text, making it impossible for engineering to close.
2. Prompt Architecture vs. Model Selection: The study demonstrates that prompt design (specifically bias-correction instructions) is a more effective lever than model selection. A customized prompt on an older model (GPT 4.1) outperformed a newer model (GPT 5.2) using the same prompt, and the mini-model failed to transfer the prompt's benefits.
3. Textual Clarity as the Binding Constraint: The study proves that the linguistic character of the input text is the primary determinant of accuracy. The variance in accuracy between different text types (e.g., positive vs. negative) is 20x larger than the variance caused by changing prompts or models.

4. Key Results

• Prompt Customization: The customized prompt improved "Within ±1" agreement from 67% to 69% on GPT 4.1. This improvement was consistent across all text types, confirming it acts as a bias-correction mechanism rather than a general reasoning upgrade.
• Model Selection Failure:
  • GPT 5.2: Performed worse than the customized GPT 4.1, reverting to baseline levels (67% within ±1).
  • GPT 4.1-mini: Performed significantly worse (61% within ±1), showing a severe under-prediction bias.
• Systematic Under-Prediction: All configurations exhibited a directional bias, systematically under-predicting ratings (Mean Signed Error: -0.83 to -1.18).
  • 86% of large errors (outside ±1) were under-predictions.
  • This bias is concentrated in the mid-range scores (5–8) where fans describe operational friction (lines, parking) but still rate the experience positively.
• The "Textual Clarity" Gap:
  • Positive-only text: 86% within ±1.
  • Negative-only text: 44% within ±1.
  • The gap between these text types is massive compared to the marginal gains from engineering.
• Operational Utility: While exact match is low (39%), the model is a high-precision classifier.
  • Top-box precision: 94–96% (When the model predicts a high score, it is almost certainly correct).
  • Low-experience recall: 94–96% (The model catches nearly all genuinely dissatisfied fans).

5. Significance and Implications

• Redefining LLM Annotation: The paper shifts the paradigm from viewing LLMs as tools to "fix" measurement error to viewing them as instruments measuring a different construct (salience) than the survey score (integrative judgment).
• Engineering Strategy:
  • Prompt Design: Should be treated as a bias-correction exercise. Instructions must explicitly counter known model priors (e.g., "do not over-penalize isolated complaints").
  • Model Selection: Upgrading models does not guarantee better performance if the prompt architecture does not address the specific bias.
  • Data Quality: "Text-rating coherence" (the model's confidence label) is a powerful proxy for accuracy. High-coherence predictions are robust; low-coherence predictions should be routed for human review.
• Operational Application: The predicted rating should not replace the survey score but complement it.
  • Fragile Satisfaction: High rating + Negative text (identified by the model's under-prediction).
  • Unexpressed Loyalty: Low rating + Positive text.
  • These divergent cases represent the highest value for operational intervention, as they reveal the gap between what fans say and how they feel.

Conclusion: The paper concludes that while prompt engineering can correct instrument bias (gaining ~2%), it cannot bridge the cognitive gap between text and verdict. The "ceiling" is determined by the information available in the text itself. Future work should focus on leveraging complementary signals (e.g., game outcome, affect) rather than attempting to force the LLM to infer missing cognitive data from text alone.