Technical Summary: SPECTRA

Problem Statement

Current approaches to modeling user preferences with Large Language Models (LLMs) predominantly rely on Direct Generation (DG), where the model autoregressively decodes a ranked list of items. The authors argue this paradigm suffers from two critical limitations:

1. Opacity: The model's internal preference state remains hidden, as the output is a single list rather than a structured distribution.
2. Bias and Information Loss: Autoregressive decoding systematically suppresses minority and long-tail preferences. Theoretical analysis suggests that once a category is omitted from a decoded list, any downstream distribution derived solely from that list cannot assign probability mass to it. This leads to a compounding loss of long-tail preferences at scale, favoring frequent (head) categories due to frequency bias in the decoding process.

Existing methods often compress user states into embeddings or produce item-level rationales, but they fail to provide an explicit, interpretable probability distribution over semantically meaningful categories (e.g., movie genres, restaurant types).

Methodology: SPECTRA

The authors propose SPECTRA (Softmax Probing for Extracted Category-level Token Readouts and Analysis), a framework that treats a finetuned LLM as an implicit probabilistic model to extract a probability distribution over preference categories.

Core Pipeline

1. Alignment (CPT + LoRA SFT):

   • Continuous Pretraining (CPT): The LLM is pretrained on textualized user metadata, item metadata, and interaction traces to learn domain context.
   • Supervised Fine-Tuning (SFT): The model is fine-tuned using LoRA to predict the user's long-term preference distribution over a fixed set of $K$ categories, rather than predicting a single next item. The target is the empirical distribution of future interactions.

2. Inference (Probing):
   Instead of generating a list, SPECTRA probes the model's logits to compute a score for each category $c_j$. Two primary methods are used:

   • Likelihood-based Probing: The model is prompted with a yes/no question regarding a specific category. The score $S_j$ is the difference between the average logit of affirmative tokens (e.g., "Yes") and negative tokens (e.g., "No").
   • Generative Classification: A multi-choice prompt asks the model to select a category. The score $S_j$ is the logit of the corresponding answer token.
   • Output: A softmax over these scores produces the category-level preference distribution $\theta_S$.

3. Hierarchical Scaling:
   For large category spaces (e.g., $K > 1,000$), a hierarchical pipeline is employed. The model first probes coarse-grained categories (Level 1) and then refines the distribution by probing fine-grained subcategories (Level 2) only within high-probability branches, reducing computational cost from $O(K)$ to $O(|L_1| + \sum |L_2|)$.

Theoretical Foundation

The paper provides theoretical guarantees comparing SPECTRA to DG:

• Theorem 1 (Ranking Advantage): If SPECTRA's readout has bounded error relative to the true distribution and the true top-$k$ categories have a sufficient ranking gap, SPECTRA recovers the true top-$k$ exactly. In contrast, DG is structurally unable to recover categories outside its decoded list.
• Theorem 2 (DG Failure Mode): Under a frequency-biased decoding regime (where decoding probability is amplified by token frequency), the probability of DG missing a rare (tail) category in the top-$k$ list is bounded from below. This formally characterizes the regime where SPECTRA strictly outperforms DG.

Key Contributions

1. Methodological Shift: Proposes probing an LLM's logits to model user preferences as a distribution over categories, rather than decoding a ranked item list.
2. Empirical and Theoretical Validation: Demonstrates that probing yields more faithful preference distributions and better long-tail coverage than Direct Generation baselines. Theoretical results show that SPECTRA recovers true top categories while sparse list decoding provably misses tail categories under frequency bias.
3. Impact on Interpretability: Delivers named-category distributions that preserve minority preferences, offering a path toward structured, auditable, and reusable preference modeling.

Experimental Results

The authors evaluated SPECTRA on MovieLens (19 genres), Yelp (26 domains, 1,311 subcategories), and a large-scale Short-Video platform (78 categories, 50M interactions).

1. Distributional Alignment

• SPECTRA significantly reduces the Jensen-Shannon (JS) divergence to the empirical preference distribution compared to DG.
• Reductions: 38–44% relative reduction in JS divergence across public datasets (e.g., from 0.51 to 0.32 on MovieLens).
• Mechanism: The improvement stems from assigning probability mass to long-tail categories that DG zeroes out, while keeping head categories close to their true frequencies.

2. Long-Tail Recovery and Fairness

• Entropy: SPECTRA increases top-3 category exposure entropy by 23% on MovieLens, indicating a more diverse set of top picks.
• Cross-User Fairness: The gains are not concentrated on head-preference users. SPECTRA reduces JS divergence more significantly for users with niche (tail) preferences (1.28× greater reduction on MovieLens for tail-leaning users vs. head-leaning users).
• Bias Structure: SPECTRA systematically under-predicts head categories and over-predicts tail categories, effectively redistributing mass to recover the true distribution.

3. Downstream Application Value

• Ranking Quality: SPECTRA improves category-ranking NDCG by 41–46% on MovieLens and Yelp compared to DG.
• Industrial Deployment: On a short-video platform, SPECTRA achieved a 7× improvement in NDCG@20 for long-tail category ranking compared to a production head-optimized ranker.
• Integration: When used as a category prior for classical recommenders (SASRec, GRU4Rec), SPECTRA provided significant lifts (up to 62%) on mid-popularity item slices.

4. Robustness

• Results hold across different context window sizes, user activity levels, and base models (Qwen3-8B and DeepSeek-R1-Distill-Llama-8B).
• The method is robust to variations in prompt templates and verbalizer choices.

Significance and Claims

The paper claims that SPECTRA offers a theory-grounded approach to more interpretable and balanced user preference modeling. By moving away from generative decoding toward softmax probing, the method avoids the structural information loss inherent in list generation.

The authors emphasize that their approach is regime-conditional: it isolates a specific failure mode where sparse decoded lists cannot represent categories outside their support. In this regime, SPECTRA provides a provable advantage. The resulting distributions are not just ranking signals but are inspectable, comparable across cohorts, and composable with classical recommendation systems, addressing the "black box" nature of LLM-based preference modeling.

The study acknowledges limitations, noting it focuses on fixed taxonomies and offline evaluation. It suggests future work should explore broader decoding strategies and evaluate the long-term impact of cached category distributions on user-facing outcomes.