Technical Summary: Personalizing LLMs with Binary Feedback (C-BPO)

1. Problem Statement

Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences. While existing methods utilize retrieval-based prompting or parameter-efficient fine-tuning (PEFT) on isolated user histories, they often neglect the critical role of inter-user differences. Recent behavioral science insights suggest that individuality is defined by how a user deviates from the general population.

Current preference optimization frameworks, such as Direct Preference Optimization (DPO), require contrastive pairs (preferred vs. non-preferred completions for the same input), which are unavailable in raw, unpaired historical user data. While recent advances allow for alignment via binary feedback (e.g., "thumbs-up" or "thumbs-down" on individual data points), applying these methods directly to personalization introduces a fundamental flaw: preference overlap.

In a personalized context, treating data from other users as purely "negative" (thumbs-down) signals is problematic. These auxiliary datasets inevitably share generic task-specific knowledge and community-wide stylistic preferences with the target user. Naively penalizing this data forces the model to erroneously suppress shared, helpful features, thereby degrading general performance while attempting to capture individual nuances.

2. Methodology: C-BPO Framework

The authors propose C-BPO (Preference-Calibrated Binary Feedback Preference Optimization), a framework that personalizes LLMs by leveraging inter-user variability as a natural contrastive signal while correcting for preference overlap.

2.1 Core Formulation

C-BPO frames personalization as a binary-feedback optimization problem:

• Positive Signals ($D_+$): The target user's historical interaction data ($H_{tar}$).
• Implicit Negative Signals ($D_-$): Data from a pool of auxiliary users ($H_{aux}$).

Unlike standard Binary Feedback Preference Optimization (BPO) methods (e.g., KTO, BCO) which assume "clean" negative labels, C-BPO acknowledges that $H_{aux}$ is a mixture of true negative signals (user-specific dislikes) and positive signals (shared community preferences).

2.2 Preference-Calibrated Objective via PU Learning

To address the preference overlap issue, C-BPO derives an optimization objective grounded in Positive-Unlabeled (PU) learning theory. The core insight is that the auxiliary data acts as an unlabeled set containing a mixture of positive and negative distributions.

The authors reformulate the risk estimation to recover the true negative risk by subtracting the "positive bias" (the expected risk of target preferences found within the broader user data). The final objective function ($L_{C-BPO}$) is defined as:

$$L_{C-BPO} = L_{pos} + \frac{1}{1-\alpha} \max \{0, L_{pure\_neg}\}$$

Where:

• $L_{pos}$: The loss on the target user's data (Positive Alignment).
• $L_{pure\_neg}$: The Purified Negative Loss, calculated as $E_{H_{aux}}[l(g, -1)] - \alpha E_{H_{tar}}[l(g, -1)]$.
• $\alpha$: A correction coefficient ($0 < \alpha < 1$) representing the degree of preference feature overlap between the target user and auxiliary users. It quantifies the proportion of auxiliary data that functionally serves as a positive signal.

This formulation explicitly "purifies" the negative signals, ensuring the model focuses on unique individual idiosyncrasies rather than suppressing general helpfulness.

2.3 Handling Data Imbalance

In personalized scenarios, auxiliary data volume often vastly exceeds target user data. Standard BPO methods estimate a reference point ($\delta$) by averaging rewards across a batch, which can drift if the batch is imbalanced. C-BPO introduces an independent Exponential Moving Average (EMA) reference point estimation. It maintains separate EMA statistics for positive and auxiliary rewards, computing $\delta_{EMA}$ as their mean. This ensures a stable decision boundary invariant to batch-level data imbalance.

3. Key Contributions

1. Novel Framing: The paper is the first to frame LLM personalization as a binary-feedback preference optimization problem, utilizing inter-user variability as a natural contrastive signal for individual-level alignment without requiring paired data.
2. Theoretical Innovation: The authors identify the "preference overlap" issue in standard BPO for personalization and formulate a preference-calibrated objective based on PU learning theory to achieve unbiased preference alignment.
3. Empirical Validation: Extensive experiments across five personalized generation tasks (News Headline, Scholarly Title, Abstract, Review, and Topic Writing) and various backbone LLMs (LLaMA, Qwen, Mistral) demonstrate that C-BPO consistently outperforms strong baselines, including SFT-based adapters and existing binary-feedback methods.

4. Experimental Results

The authors evaluated C-BPO on the LaMP and LongLaMP benchmarks using Mistral-7B, LLaMA-3, and Qwen2.5 backbones.

• Performance: C-BPO consistently achieved the best ROUGE-1 and ROUGE-L scores across all tasks. For instance, on the News Headline task, C-BPO achieved a ROUGE-L of 0.198, outperforming the best baseline (CoPE, a DPO-based method) at 0.184 and standard BPO methods (BCO at 0.179).
• Comparison with Baselines:
  • Standard binary-feedback methods (KTO, BCO) often underperformed SFT-based baselines (OPPU) when applied naively to personalization, confirming the detrimental effect of uncorrected preference overlap.
  • C-BPO outperformed DPO-based methods (CoPE) which require complex rejection sampling to construct negative pairs.
• Ablation Studies:
  • Auxiliary Data Volume: Increasing auxiliary data volume does not guarantee monotonic gains; a sufficient amount of target user data is required to anchor the debiasing process.
  • User Uniqueness: C-BPO showed the most significant improvements in "Unique" user groups (high inter-user distance), while standard BPO methods failed to gain in "Non-unique" groups (high overlap).
  • Reference Point: The independent EMA strategy was crucial for stability, preventing performance drops when data ratios were imbalanced.

5. Significance and Claims

The paper claims that C-BPO offers a robust, theoretically sound, and scalable solution for aligning LLMs with nuanced individual preferences. Its significance lies in:

• Efficiency: It eliminates the need for constructing expensive contrastive pairs (as in DPO) or relying on high-resource LLMs for textual comparisons.
• Data Utilization: It effectively leverages raw, unpaired historical data from both the target and auxiliary users, turning the "noise" of shared preferences into a calibrated signal.
• Generalization: The framework is model-agnostic and effective across different backbone architectures and task types.

The authors acknowledge limitations, including the sensitivity to the calibration coefficient $\alpha$ (though an estimation method is provided) and the requirement for access to a centralized auxiliary dataset, which poses privacy challenges in federated or on-device settings. However, the work establishes a new paradigm for personalization that moves beyond isolated history modeling to explicit, preference-calibrated inter-user comparison.