Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization

1. Problem Statement

The paper addresses the challenge of personalizing Large Language Models (LLMs) without the high computational cost of fine-tuning (e.g., RLHF or LoRA). While Retrieval-Augmented Generation (RAG) offers a lightweight alternative by injecting user history into the prompt, existing methods suffer from two critical flaws:

• Relevance $\neq$ Utility: Current systems select user records based on semantic or lexical similarity to the query. However, a record can be semantically similar yet detrimental to generation quality due to redundancy, conflicting information, or misalignment with the user's underlying intent.
• Greedy Selection Failure: Existing approaches typically select the top-$k$ most relevant records independently. This ignores inter-record dependencies; a combination of individually "good" records may degrade performance if they conflict or dilute the conditioning signal. The utility of a user profile is non-monotonic—adding more records does not guarantee better results.

The core problem is: How to construct an ordered user profile from a history of records that maximizes downstream generation quality (utility), rather than just semantic relevance?

2. Methodology: PURPLE Framework

The authors propose PURPLE (oPtimizes UseR Profiles for LLM pErsonalization), a framework that casts profile construction as a Contextual Bandit problem.

A. Problem Formulation

• Context ($C$): The current query ($x$) and the user's full history of records ($H$).
• Action ($P$): Selecting an ordered tuple of $K$ records from $H$ to form a profile. The order matters because LLMs are sensitive to prompt sequence.
• Reward ($R$): The quality of the LLM's generation given the profile and query, measured against a reference response ($y$).
• Objective: Learn a policy $\pi_\theta$ that maximizes the expected reward $E[R]$.

B. Model Architecture

1. Plackett-Luce (PL) Ranking Model:
   • Instead of independent scoring, PURPLE uses a PL model to assign probabilities to permutations of records.
   • It outputs a propensity score for each record, parameterizing a distribution over all possible ordered profiles.
   • This allows the model to capture inter-record dependencies (e.g., recognizing that Record A and Record B are redundant when combined).
2. User Record Encoder ($f_\theta$):
   • Utilizes a late interaction strategy.
   • A pre-trained encoder (Contriever) generates token embeddings for the query and records.
   • Records undergo token-level cross-attention with the query to fuse intent.
   • A Transformer encoder then models dependencies across the resulting record embeddings (without positional encodings to avoid bias).
3. Training via Policy Gradient:
   • Since the reward is non-differentiable, PURPLE uses the likelihood ratio gradient estimator (REINFORCE).
   • During training, it samples $M$ profiles from the PL distribution, generates responses, and computes gradients based on the reward.
   • Reward Function: Instead of task-specific metrics (e.g., ROUGE), PURPLE uses the log-likelihood of the reference response generated by the frozen LLM. This provides a semantically rich, fine-grained signal that correlates better with generation quality than coarse metrics.

C. Inference

• At inference time, the model computes propensity scores for all candidate records.
• The top-$K$ records are selected and ordered by these scores to construct the final user profile, which is then fed to the frozen LLM.

3. Key Contributions

1. Novel Formulation: Introduces PURPLE, the first framework to treat retrieval-augmented personalization as a contextual bandit problem, optimizing for utility rather than relevance.
2. Dependency Modeling: Employs a Plackett-Luce model to handle the non-additive nature of record utility, capturing complex interactions and ordering effects that greedy selection misses.
3. Semantically Rich Reward: Proposes using the log-likelihood of the reference response as the training signal, which outperforms traditional task-specific metrics (Accuracy, ROUGE) by providing finer-grained feedback.
4. Comprehensive Evaluation: Validated across nine personalization tasks (classification, regression, short/long-text generation) on the LaMP and LongLaMP benchmarks.

4. Experimental Results

The authors evaluated PURPLE using frozen LLMs (Phi-4-Mini, Llama-3-8B, and Llama-3-70B) against strong baselines:

• Baselines: Heuristic retrievers (BM25, Contriever), In-Context RALMs (IC-RALM, REPLUG), and Zero-Shot Rerankers (RankGPT, ICR).
• Performance: PURPLE consistently outperformed all baselines across all tasks and model sizes.
  • It achieved higher accuracy/F1 in classification, lower MAE in regression, and better ROUGE/METEOR scores in generation.
  • Notably, PURPLE outperformed much larger models (like GPT-5-nano based RankGPT) while being significantly more efficient.
• Efficiency: PURPLE maintains high computational throughput, outperforming ICR and RankGPT in the performance-throughput trade-off.
• Human Evaluation: In a blind study on tweet paraphrasing, human evaluators preferred PURPLE's outputs over the best baseline (ICR) by a 14.4% margin, citing better capture of user persona, tone, and style.
• Ablation Studies: Confirmed that removing token-level cross-attention, inter-record dependency modeling (Transformer), or switching to metric-based rewards significantly degraded performance.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the standard "relevance-based" retrieval paradigm in RAG, demonstrating that utility-based optimization is crucial for personalization.
• Scalability: By avoiding parameter updates to the LLM and using a lightweight reranking policy, PURPLE offers a scalable, transparent, and deployable solution for real-time personalization.
• Theoretical Insight: It establishes that user profiles are order-sensitive and that record utility is non-monotonic, necessitating list-aware reasoning rather than independent selection.
• Future Work: The authors note limitations regarding the reliance on reference responses for training (supervision) and the need for better generalization across unseen tasks, suggesting directions for leveraging implicit user feedback.

In summary, PURPLE provides a principled, scalable, and highly effective approach to personalizing LLMs by treating user profile construction as a sequential decision-making problem optimized for downstream generation quality.