Technical Summary: LLM Alignment using Implicit Feedback from Users

Problem Statement

Current methods for aligning Large Language Models (LLMs) with human preferences rely heavily on explicit feedback (e.g., thumbs-up/down ratings). This approach faces two critical limitations:

1. Scarcity and Cost: Only 1–3% of users provide explicit feedback, making high-quality preference annotation expensive and difficult to scale.
2. Underutilization of Implicit Signals: While implicit feedback (e.g., click-through rates) has been vital for search engines and recommender systems, it remains underexplored for LLMs. Existing systems often lack the "click" signals found in other domains, and it is unclear how to interpret user interactions (like reading behavior) as reliable training signals.

The paper posits that users' natural interactions with LLM responses—specifically mouse movements and eye-gazing patterns—contain rich, scalable signals of preference that are currently ignored.

Methodology

1. Data Collection: The IFLLM Dataset

The authors constructed IFLLM (Implicit Feedback for Large Language Models), the first dataset capturing mouse and eye-gazing trajectories alongside explicit preferences in a realistic, multi-turn conversational setting.

• Participants: 59 unique Amazon Mechanical Turk (MTurk) workers.
• Task: Users engaged in multi-turn Q&A sessions on 1,336 topics (derived from Wikipedia). They interacted with four LLMs (DeepSeek V3, GPT-4o Mini, Claude Sonnet 4.5, Llama 3.3 70B).
• Settings:
  • Pairwise: Comparing two side-by-side responses.
  • Pointwise: Scoring a single response on a 5-point Likert scale.
• Instrumentation: A custom web-based crowdsourcing platform using WebGazer.js to track eye gaze via standard webcams (eliminating the need for specialized lab equipment) and recording mouse coordinates at 10Hz (every 0.1s).
• Calibration: Users underwent a calibration process to ensure gaze tracking accuracy (70% threshold) before data collection.

2. User Behavior Analysis

The authors analyzed the collected trajectories to understand reading patterns:

• Response Length Dependency: For short responses, mouse and gaze trajectories are weakly correlated. For medium and long responses, they become strongly correlated because users must scroll, forcing mouse movement to track reading progress.
• Attention Distribution: Users tend to focus more on the beginning of responses, but attention shifts based on layout (e.g., left vs. right columns in pairwise settings).
• Individual Variability: Reading behaviors are highly diverse, ranging from linear reading to extensive back-and-forth reviewing.

3. Reward Model Training

The authors trained a Random Forest (RF) reward model to predict user preferences using features extracted from the implicit feedback.

• Feature Extraction: Features included text properties (query/response length) and trajectory metrics (reading time, max character index viewed, reviewing time, and the ratio of gaze to mouse movement).
• Feature Selection: The model identified that implicit feedback features (especially mouse movement) were more predictive than text-based features or even the identity of the generating LLM.
• Comparison: The implicit-feedback model was compared against standard text-only models (ModernBERT) and zero-shot LLM judges.

4. LLM Alignment (DPO)

The trained reward models were used to align eight different base LLMs (ranging from 1B to 4B parameters) using Direct Preference Optimization (DPO) and rDPO (regularized DPO with Negative Log Likelihood).

• Process: The models were first Supervised Fine-Tuned (SFT) on the responses, then optimized using the preferences predicted by the reward models.
• Evaluation: The quality of the aligned LLMs was evaluated by GPT-4.1-mini and validated via a small-scale human experiment where workers verified factual claims.

Key Results

1. Preference Prediction Accuracy

• Baseline: A standard text-based reward model (ModernBERT) achieved an accuracy of approximately 55%.
• Implicit Feedback: Incorporating implicit feedback (mouse and gaze) via the Random Forest model boosted accuracy to 64%.
• Signal Importance: Removing mouse data caused a significant drop in performance, whereas removing gaze data had a smaller impact. This suggests mouse movement is a stronger signal, particularly for longer responses where scrolling is required.

2. LLM Alignment Improvements

• Response Quality: Applying DPO with the implicit-feedback reward model significantly improved response quality compared to SFT alone.
• Win Rates: The "DPO vs. SFT" win rate increased from 0.12 (using text-only rewards) to 0.35 (using implicit feedback rewards).
• Efficiency: The implicit-feedback approach produced higher-quality responses with significantly shorter average lengths (202.5 tokens vs. 228.6 tokens for SFT), suggesting better conciseness and relevance.
• Model Agnosticism: The improvements were consistent across various base models, including recent Qwen series models.

Significance and Contributions

The paper claims three primary contributions:

1. IFLLM Dataset: The creation and release of the first dataset pairing webcam-based eye-gaze and mouse trajectories with explicit preferences in a multi-turn conversational context. The authors also release the source code for the data collection website to facilitate future research.
2. Systematic Analysis: The first in-depth analysis of diverse reading behaviors (gaze and mouse) and a comparative evaluation of their effectiveness for training reward models. The study reveals that mouse trajectories are a critical, previously underutilized signal for long-form content.
3. Scalable Alignment: Demonstration that implicit feedback can substantially improve reward model accuracy and, consequently, the quality of aligned LLMs. The authors argue this lays the groundwork for scalable personalized alignment, potentially creating a "self-reinforcing data flywheel" where ordinary user interactions continuously improve system performance without requiring expensive explicit labeling.

The authors maintain a modest tone regarding limitations, noting that their reward model requires users to view responses to collect data (a prerequisite for RLHF anyway) and that they simplified multi-turn contexts into single-turn sessions for this study. They also acknowledge ethical considerations regarding privacy and the potential for reduced diversity in LLM choices if data flywheels become dominant.