POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

1. Problem Statement

Large Language Models (LLMs) are typically aligned with population-level preferences, resulting in a "one-size-fits-all" behavior that fails to account for substantial variation among individual users. While existing personalization methods exist, they face significant challenges:

• Scalability: Fine-tuning a separate model for every user (e.g., via RLHF or DPO) is computationally infeasible.
• Context Overhead: Methods that condition a shared generator on raw user signals (e.g., interaction history, few-shot examples) often require massive context windows, leading to high latency and cost.
• Noise and Heterogeneity: Raw user signals are often sparse, noisy, and heterogeneous, making it difficult to extract stable preference information directly.
• Lack of Transferability: Many approaches couple the personalization mechanism tightly to a specific generator architecture, preventing the reuse of learned preferences with different models (especially black-box commercial APIs).

The core challenge is to create a scalable, efficient, and modular framework that can infer stable user preferences from sparse signals and apply them to generate personalized responses without retraining the generator for each user.

2. Methodology: POPI Framework

The authors propose POPI (Personalizing LLMs via Optimized Natural Language Preference Inference), a modular framework that decomposes personalization into two distinct components connected by a natural-language interface.

A. Modular Architecture

1. Preference Inference Model ($\pi_\phi$): A shared model that takes heterogeneous user signals ($s_i$) (e.g., personas, interaction history, few-shot preference triplets) and distills them into a concise natural-language preference summary ($z_i$).
2. Conditioned Generator ($\pi_\theta$): A shared model that generates responses ($y$) conditioned on the task prompt ($x$) and the natural-language summary ($z_i$).

Key Design Principle: The interface ($z_i$) is natural language. This allows the summary to be inferred once per user and reused across different generators, including frozen, off-the-shelf, or black-box models, without retraining the inference model.

B. Unified Optimization Objective

Both components are trained under a unified objective derived from preference alignment algorithms (e.g., DPO).

• Summary-Augmented Objective: The standard Direct Preference Optimization (DPO) loss is adapted to condition the generator on the summary $z_i$.
• Information-Theoretic Decomposition: The authors prove that minimizing this unified loss simultaneously optimizes two terms:
  1. Generator Approximation Error: Ensuring the generator accurately predicts the preferred response given the summary.
  2. Summary Informativeness: Ensuring the summary contains preference-relevant information that the prompt alone does not convey.

C. Training Procedure (Two-Stage)

To preserve modularity and transferability, POPI uses a two-stage training process:

1. Stage 1 (Reinforcement Learning): The inference model $\pi_\phi$ is optimized using Group Relative Policy Optimization (GRPO). Since the summary $z_i$ is discrete (text), the DPO loss is non-differentiable with respect to $\pi_\phi$. GRPO treats the generation of summaries as actions, assigning rewards based on how well the resulting summary helps the frozen generator distinguish between preferred and rejected responses. A KL-divergence regularization term is added to prevent the summaries from degenerating and to encourage compression.
2. Stage 2 (Optional Fine-Tuning): The generator $\pi_\theta$ is fine-tuned using standard backpropagation on the same objective, conditioning on the summaries learned in Stage 1.

3. Key Contributions

1. Modular Framework: Introduces POPI, which separates preference inference from generation via a natural-language interface, enabling generator-transferability (summaries work with any generator, including black-box APIs).
2. Unified Objective & Theory: Proposes a unified training objective that jointly optimizes inference and generation. The paper provides an information-theoretic proof showing this objective decomposes into minimizing generator error and maximizing summary informativeness.
3. Efficiency: Demonstrates that optimizing the inference step leads to emergent compression, where summaries become significantly shorter (e.g., ~50 tokens vs. ~500 tokens) without explicit length penalties, drastically reducing context overhead.
4. Empirical Validation: Extensive evaluation across four benchmarks (Review, ELIX, Roleplay, AlignX) showing POPI outperforms baselines in personalization quality while reducing context usage by up to an order of magnitude.

4. Experimental Results

The authors evaluated POPI on four benchmarks: Review (sentiment/conciseness), ELIX (educational explanations), Roleplay (dialogue), and AlignX (forum discussions).

• Performance vs. Baselines:

  • Without Generator Fine-Tuning: POPI-Inference-Only (using a frozen generator) significantly outperformed "Raw-Prompting" (feeding raw signals) and "Summary-Prompting" (using unoptimized summaries).
    • Example (Review): Win rate improved from 57.65% (unoptimized summary) to 80.99% (POPI).
    • Example (ELIX): Win rate improved from 47.40% to 63.84%.
  • Context Efficiency: POPI reduced context overhead by roughly 10x (e.g., from ~3,175 tokens to ~52 tokens on ELIX) compared to raw prompting.
  • With Generator Fine-Tuning: POPI-Full achieved state-of-the-art results on ELIX and Roleplay, matching or exceeding methods that require heavy context or per-user embeddings.

• Generator Transferability:

  • Summaries learned by POPI (trained on a 3B model) were successfully applied to a diverse set of frozen, off-the-shelf generators, including Mistral, DeepSeek, Claude, GPT-4o, and LLaMA variants (up to 90B parameters).
  • POPI summaries consistently outperformed raw prompting and unoptimized summaries across all tested generators, proving the "generator-agnostic" nature of the natural-language interface.

• Qualitative Analysis:

  • Optimized summaries were found to be concise, linguistically well-formed, and highly specific to user personas (e.g., distinguishing between "child-centric language" and "technical jargon" for different user levels), whereas unoptimized summaries were verbose and generic.

5. Significance and Impact

• Scalable Personalization: POPI offers a practical path to user-level personalization that does not require per-user model training, making it feasible for large-scale deployment.
• Black-Box Compatibility: By using natural language as an interface, POPI allows users to personalize responses from commercial, closed-source LLMs (like GPT-4 or Claude) without needing access to their internal weights.
• Efficiency: The emergent compression of user signals into short, high-value summaries addresses the growing cost and latency issues associated with long-context LLM usage.
• Complementarity: The framework is compatible with existing handcrafted profile extraction methods, acting as a refinement layer to further optimize those profiles.

In conclusion, POPI demonstrates that treating preference inference as an optimizable, end-to-end task rather than a static preprocessing step yields superior personalization, efficiency, and flexibility in the era of large language models.