GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

1. Problem Statement

Current Large Language Model (LLM) benchmarks focus heavily on reasoning, coding, and knowledge retrieval, but lack rigorous evaluation for User Understanding in recommendation systems (RecSys).

• The Gap: Traditional RecSys metrics (e.g., Click-Through Rate, Next-Item Ranking) measure prediction accuracy but conflate user interest with confounding factors like positional bias, temporal context, and content quality. They do not verify if an LLM can accurately extract and represent user interests from raw behavioral logs.
• The Challenge: Existing "LLM-as-a-Judge" frameworks often evaluate the plausibility of a generated profile rather than its factual groundedness against the user's actual interaction history. There is a need for metrics that verify whether a predicted interest is supported by sufficient, specific engagement signals (e.g., distinguishing between a "like" and a "skip") without relying on subjective ground-truth labels.

2. Methodology

A. Dataset Construction (GISTBench)

The authors introduce a novel synthetic dataset grounded in real user interactions from a global short-form video platform, designed to preserve privacy while maintaining statistical fidelity.

• Synthetic User Construction: Real user data is anonymized by aggregating interactions from cohorts of users sharing similar interest clusters (10 clusters, 10+ users each). Interaction histories are shuffled to break temporal patterns.
• Rich Signal Taxonomy: The dataset includes four signal types:
  • Explicit Positive: Likes, comments, shares, saves.
  • Implicit Positive: Extended watch time.
  • Implicit Negative: Skips (rapid swipes).
  • Explicit Negative: Dislikes (rare in video, common in reviews).
• Content Representation: Video content is described via natural language summaries generated by a proprietary Vision-Language Model (VLM), providing rich textual context for the LLM to parse.
• External Validation: The benchmark also evaluates models on four public datasets (KuaiRec, MIND, Amazon Music, Goodreads) to test generalization across domains and signal sparsity.

B. Evaluation Framework

The framework evaluates LLMs on two primary axes: Interest Groundedness (IG) and Interest Specificity (IS).

1. Interest Prediction Pipeline:

   • LLMs are prompted to generate a User Interest Profile ($I_u$) consisting of natural language descriptors and cited evidence objects ($O_j$) from the interaction history.
   • Prompts include strict, programmatically encoded thresholds (e.g., "require $\ge$ 3 implicit positive signals") to ensure a closed-loop between instruction and evaluation.

2. Interest Groundedness (IG):

   • Concept: Verifies if a predicted interest is supported by observable behavioral evidence, penalizing hallucinations and rewarding coverage.
   • Mechanism: An LLM judge filters cited evidence for semantic relevance. A verified interest must satisfy dataset-specific thresholds (e.g., $\ge$ 2 explicit positives OR $\ge$ 3 implicit positives) while respecting negative signal constraints (e.g., max 3 implicit negatives).
   • Decomposition: IG is split into:
     • Precision (IGP): Fraction of predicted categories that are verified (penalizes hallucination).
     • Recall (IGR): Fraction of "oracle" categories (union of verified categories across all models) that the model covers (penalizes incomplete coverage).
   • Score: $IGF1$ is the harmonic mean of IGP and IGR.

3. Interest Specificity (IS):

   • Concept: Assesses whether verified interests are granular enough to distinguish the user's true interests from distractors.
   • Mechanism: A retrieval task where an LLM judge must identify the specific evidence objects that support a predicted interest from a pool of distractors.
   • Constraint: IS is calculated only over verified categories to prevent inflating scores with hallucinated interests.

4. Taxonomy Normalization:

   • Fine-grained interests are mapped to a standardized 35-domain, 325-category taxonomy. This prevents "depth-breadth" conflation (e.g., predicting "NBA," "Lakers," and "LeBron" counts as one category, not three).

3. Key Contributions

1. GISTBench Benchmark: A comprehensive framework for evaluating LLMs on user understanding, featuring a large-scale synthetic dataset with rich, multi-modal engagement signals and four public datasets.
2. Novel Metrics (IG & IS):
   • Interest Groundedness: A precision-recall framework that verifies interests against behavioral evidence rather than subjective labels.
   • Interest Specificity: A retrieval-based metric ensuring predicted interests are discriminative.
3. Empirical Insights: The paper provides the first large-scale analysis of LLM capabilities in counting and attributing heterogeneous engagement signals, revealing specific bottlenecks in current models.

4. Key Results & Findings

The authors evaluated eight open-weight LLMs (ranging from 7B to 120B parameters).

• Performance Bottlenecks:

  • Coverage vs. Hallucination: The primary bottleneck is Recall (Coverage), not Precision. Models are generally precise (low hallucination) but fail to discover the full range of user interests ($IGP \gg IGR$).
  • Evidence Counting: The most common failure mode is insufficient positive evidence. Models struggle to accurately count and weight implicit signals (e.g., watch time) versus explicit signals.
  • Signal Type Sensitivity: Datasets with only implicit signals (e.g., MIND click/non-click) are significantly harder, causing many models to achieve 0% IGF1.

• Model Architecture Insights:

  • Instruction Following: High instruction-following capability (Chatbot Arena Elo) is necessary to produce structured output but does not predict ranking once a threshold (~1200 Elo) is met.
  • Reasoning vs. Specificity: Larger models with reasoning training (e.g., DeepSeek-R1) significantly outperform distilled versions in Groundedness (IG) but perform similarly in Specificity (IS). This suggests that counting and attributing evidence requires deep reasoning capacity, while generating specific labels does not.
  • Context Length: Increasing the interaction history length improves Groundedness (Recall) significantly by providing more evidence, but has a marginal effect on Specificity.

• Validation:

  • The proposed metrics correlate moderately with user self-reported survey data ($\rho = 0.67$), confirming that behavioral grounding and conscious self-identity capture complementary aspects of user understanding.

5. Significance

• Paradigm Shift: Moves RecSys evaluation from "does this prediction improve CTR?" to "does the model correctly understand the user's history?"
• Verifiable Grounding: Introduces a method to audit LLM outputs against raw behavioral data, addressing the "black box" nature of user profiling in generative AI.
• Guidance for Model Development: Highlights that current LLMs lack the specific capability to perform rigorous evidence counting and signal attribution. Future models for recommendation systems require training specifically on multi-step signal reasoning rather than just general instruction following.
• Privacy-Preserving Benchmark: Demonstrates a viable path for creating high-fidelity evaluation datasets using synthetic aggregation of real user data, enabling public research without compromising privacy.

In conclusion, GISTBench establishes that while LLMs can generate plausible user profiles, they currently struggle with the quantitative rigor required to accurately map complex, heterogeneous engagement histories to specific user interests. The benchmark provides the necessary tools to measure and improve this critical capability.