Technical Summary: CURP (Codebook-based Continuous User Representation for Personalized Generation with LLMs)

1. Problem Statement

Current approaches to user modeling for Large Language Models (LLMs) face a fundamental tension between personalization fidelity and computational efficiency. Existing methods generally fall into two paradigms, both of which have significant limitations:

• Non-parametric Prompting: Injects user history or persona descriptions directly into the prompt. While lightweight and general-purpose, this approach often suffers from noise, redundancy, and context length inflation, leading to imprecise or stereotyped outputs.
• Parametric Fine-tuning: Adapts model weights (e.g., via LoRA or full fine-tuning) to specific user data. While this achieves high fidelity, it is computationally prohibitive and impractical at scale, as it typically requires training a separate model or adapter for every user.

The central challenge addressed by this work is how to achieve faithful user simulation that balances high personalization quality with computational efficiency and scalability.

2. Methodology: The CURP Framework

The authors propose CURP, a framework that models users through a shared, structured latent prototype space rather than raw text or user-specific weights. The core idea is inspired by Self-Categorization Theory (SCT), which posits that identity is formed through a blend of shared social prototypes (e.g., "Teenager," "Faithful," "Like Sports") rather than a fixed, isolated self-concept.

Core Components

1. User Encoder ($E$): A bidirectional, pretrained encoder-only architecture (e.g., Contriever) that distills a user's historical behavior data into dense tensor representations. This replaces naive text concatenation, reducing noise and context length.
2. Prototype Codebook ($C$): A learnable, discrete codebook where each entry represents a coherent behavioral prototype.
   • Quantization: User embeddings are decomposed into $L$ subspaces and quantized using Product Quantization (PQ). Each user event is mapped to a sparse combination of $L$ discrete indices from the shared codebook.
   • Interpretability: The resulting representation is a combination of prototypes (e.g., #Teenager + #Faithful + #Funny), offering interpretability and privacy benefits (transmitting only indices).
3. Decoder ($D$): A frozen LLM (e.g., Qwen-2.5) that generates personalized responses. A lightweight Multi-Layer Perceptron (MLP) projects the quantized prototype embeddings into the decoder's embedding space, which are then prepended to the task query to condition generation.

Two-Stage Training Pipeline

1. Stage 1: Prototype Codebook Construction (PCC):
   • Trained on a large-scale behavioral pool (e.g., 24M histories from 150k users).
   • Uses a balanced k-means algorithm to initialize the codebook and optimizes for three objectives:
     • Quantization Loss ($L_{quant}$): Minimizes reconstruction error between original and quantized embeddings.
     • Diversity Loss ($L_{div}$): Ensures codebook entries are distinct.
     • Usage Loss ($L_{usage}$): Prevents codebook collapse by encouraging uniform utilization of all entries.
2. Stage 2: Prototype Behavior Aligning (PBA):
   • A lightweight adapter (MLP) is trained to map the user's quantized prototype representation into the frozen LLM's space.
   • The LLM itself remains frozen; only the adapter and the codebook (if fine-tuned) are updated. This makes the approach model-agnostic and highly efficient.

3. Key Contributions

• Prototype-Based Codebook Mechanism: Replaces fragile textual descriptions with compact, controllable, continuous embeddings. The quantized codebook enables privacy-preserving cloud-edge collaboration by transmitting only discrete code indices rather than full embeddings or raw history.
• Two-Stage Training Framework: Enables the model to understand and utilize user prototypes without requiring additional training of the large LLM backbone, ensuring scalability.
• Balanced Performance: Achieves a practical balance between personalization fidelity and efficiency, using approximately 20M trainable parameters (about 0.2% of the total model size) compared to methods requiring per-user fine-tuning.

4. Experimental Results

The authors evaluated CURP across four diverse personalization tasks: Reddit QA (RQ), News Headline generation (NH), Tweet Paraphrasing (TP), and Review Writing (RW).

• Performance: CURP consistently outperformed strong baselines, achieving state-of-the-art results on 70% of the metrics across all tasks. It surpassed both prompting-based methods (e.g., ICL, RAG, CoT) and training-based methods (e.g., LoRA, OPPU, Soft Prompt).
• Efficiency: Unlike OPPU, which requires 8M parameters per user, CURP uses a shared codebook and a small adapter, drastically reducing computational costs.
• Robustness:
  • History Length: Unlike ICL and LoRA, which degrade as context length increases, CURP remains stable even with varying numbers of historical interactions.
  • Generalization: The codebook trained on one dataset (Reddit QA) maintained high utility when applied to other tasks with frozen weights, demonstrating transferable semantic patterns.
  • Architecture Independence: The framework performed competitively across different encoder-decoder combinations (e.g., RoBERTa+Qwen, Contriever+LLaMA).
• Interpretability: Analysis of the codebook indices revealed that different subspaces correspond to coherent semantic concepts (e.g., topic-level consistency, sentiment, and stance), validating the structured nature of the learned prototypes.

5. Significance and Claims

The paper claims that CURP offers a novel solution to the efficiency-fidelity trade-off in LLM-based personalization. By decomposing users into combinations of learnable, interpretable prototypes, the framework:

1. Mitigates Data Sparsity: The shared latent space allows learning from aggregated signals across users while preserving individual differences through flexible blending.
2. Enhances Privacy: The discrete nature of the codebook indices prevents the transmission of sensitive raw user data or full embeddings.
3. Scalability: The approach is practical for real-world deployment, avoiding the prohibitive costs of per-user fine-tuning while maintaining high generation quality.

The authors note limitations, including the lack of explicit persona descriptions in downstream datasets and the potential loss of fine-grained behavioral patterns (e.g., habitual word usage) due to quantization. However, they assert that the framework provides a robust, scalable, and interpretable foundation for personalized generation.