Users as Annotators: LLM Preference Learning from Comparison Mode

Technical Summary: Users as Annotators: LLM Preference Learning from Comparison Mode

Problem Statement
The alignment of Large Language Models (LLMs) with human preferences typically relies on pairwise preference data, where annotators select the better response between two model outputs. Traditionally, this data is collected via professional human annotators, a process that is costly and limited in scale. While end-users of LLM platforms (e.g., via comparison modes in ChatGPT) offer a vast, scalable source of preference data, this approach suffers from a critical lack of quality control. Unlike professional annotators, users are not incentivized to provide consistent judgments and may label data randomly. Existing quality control techniques, such as "golden questions" or self-consistency checks, are impractical in this setting because platforms cannot control user queries or force repeated annotations. The core challenge is to distinguish high-quality, attentive user feedback from noisy, casual inputs without access to ground-truth labels or expert oversight.

Methodology
The paper proposes a framework to assess user annotation quality and filter data for downstream alignment tasks (specifically Direct Preference Optimization, or DPO). The methodology relies on three key components:

1. Asymmetric Response Generation: Unlike standard preference pipelines where two responses are generated from the same model, this framework generates two responses ($y_{i1}, y_{i2}$) from two different LLMs ($P_A$ and $P_B$) with a known performance gap. This asymmetry creates a statistical handle: an attentive user is expected to prefer the stronger model more often, whereas a casual user chooses randomly.
2. User Behavioral Model: The authors model the annotation behavior of user $j$ using a latent "attentiveness" parameter $\eta_j \in [0, 1]$.
   • If $\eta_j = 1$, the user acts as a perfect annotator, selecting the response preferred by the population with probability $p(x_i)$.
   • If $\eta_j = 0$, the user selects randomly ($50/50$).
   • The probability of user $j$ selecting response 1 is modeled as $P(z_{ij}=1|x_i) = 1/2 + \eta_j \cdot (p(x_i) - 1/2)$.
   • Here, $p(x_i)$ represents the true preference probability of the population (the probability that Model A is better than Model B).
3. Expectation-Maximization (EM) Algorithm: Since $\eta_j$ is unobservable, the authors develop an EM algorithm to estimate the parameters of the user behavior model (including the distribution of $\eta_j$ and the global preference probability $\mu$).
   • E-step: Computes the posterior distribution of each user's attentiveness $\eta_j$ given their annotation history and current parameter estimates.
   • M-step: Updates the model parameters to maximize the expected complete-data log-likelihood.
   • The algorithm assumes a known or estimable global preference probability $\mu$ (the average win rate of Model A over Model B) to ensure identifiability, though the paper discusses extensions for unknown $\mu$.
   • Once $\eta_j$ is inferred, users are filtered based on a threshold (e.g., retaining only those with high posterior probability of being attentive).

Key Contributions

• Novel Annotation Framework: The paper introduces the first probabilistic model specifically designed to capture user behavior in LLM comparison modes, leveraging the asymmetry of response generation from two different models to infer data quality.
• Statistical Inference via EM: It develops an EM algorithm to estimate latent user attentiveness and the underlying preference distribution. The authors provide theoretical guarantees, showing that under specific regularity conditions, the algorithm converges to the true parameters as the number of users and samples increases.
• Data Filtering for Alignment: The framework demonstrates a practical pipeline where inferred attentiveness is used to filter out low-quality annotations before training. This allows for the utilization of massive, unvetted user data while mitigating noise.

Experimental Results
The authors validate their approach through synthetic experiments and downstream DPO tasks:

• Synthetic Convergence: Simulations confirm that the EM algorithm converges to the true parameters regardless of initialization. The algorithm also demonstrates robustness to model misspecification (e.g., estimating a discrete distribution when the true distribution is continuous).
• DPO Performance: In experiments using datasets like UltraFeedback and HelpSteer3, the authors fine-tuned LLMs (e.g., Qwen2.5-7B) using filtered data.
  • Filtering for attentive users (retaining the top 80% by inferred attentiveness) consistently improved DPO performance compared to training on unfiltered data.
  • Improvements included higher average reward scores (over 10% increase in some cases) and higher win rates against baseline models.
  • The experiments also highlighted a trade-off: while filtering improves quality, excessive filtering reduces data volume. An optimal threshold exists depending on the underlying distribution of user attentiveness.
  • The performance gap between the two generation models (represented by $\mu$) was found to be critical; if $\mu$ is too close to 0.5, attentive and casual users become indistinguishable, hindering the filtering process.

Significance and Claims
The paper claims to offer a scalable solution for LLM alignment by transforming end-users into effective annotators. By formalizing user behavior and introducing a statistical mechanism to filter noise, the work enables the use of vast amounts of user-generated preference data that were previously considered too noisy for training. The authors position this as a complementary approach to existing robust training methods (like robust DPO), focusing on data quality before training rather than just noise tolerance during training. The work also opens the door for future user-incentive mechanisms, such as rewarding users based on their inferred attentiveness levels. The theoretical contribution lies in the global convergence guarantees of the proposed EM algorithm for this specific behavioral model, providing a solid mathematical foundation for the approach.