Technical Summary: PreferThinker

Problem Statement

Personalized image preference assessment aims to evaluate an individual user's image preferences using only a small set of reference images as prior information. While existing methods effectively handle general preferences (e.g., text-image alignment, aesthetics) using large-scale datasets, they struggle with personalized preferences. This challenge arises from two primary factors:

1. Data Scarcity and Heterogeneity: Unlike general preferences where users share common criteria, personalized data for each user is limited, distinct, and difficult to scale for training.
2. Complexity of Taste: Individual preferences are often diverse and multi-dimensional (covering art style, color, medium, etc.), making them harder to capture than clear general human preferences.

Current approaches, including CLIP-based methods and Multimodal Large Language Models (MLLMs), typically fail in this domain because they either lack interpretability (outputting only numerical scores) or cannot leverage sufficient personalized data for large-scale training. Existing personalized methods like ViPer implicitly use reference images but fail to fully leverage the prior information contained within them or provide interpretable reasoning steps.

Methodology

The authors propose PreferThinker, a reasoning-based personalized image preference assessment system that follows a predict-then-assess paradigm. The core insight is to bridge various users by introducing a Common Preference Profile, which consists of shared visual elements (e.g., art style, color, art medium, saturation, detail). Although individual tastes are unique, these underlying visual elements are common and can be learned across users.

1. Visual Preference Profile

The system formalizes user preferences into a profile comprising five key visual elements identified through user studies and analysis of text prompts: Art Style, Color, Art Medium, Saturation, and Detail. This element-level profile allows for the characterization of complex individual tastes and serves as a bridge for training on large-scale user samples.

2. Dataset Construction (PreferImg-CoT)

To enable structured reasoning, the authors constructed PreferImg-CoT, the first large-scale Chain-of-Thought (CoT) style personalized assessment dataset.

• Generation: They first created PreferImg, a dataset with 80,000 simulated users, each assigned distinct preference profiles. Reference images (preferred and non-preferred) and candidate images were generated using text-to-image models based on these profiles.
• Annotation: Using Claude 3.7, they generated high-quality CoT-style responses for 60,000 user samples. These annotations include explicit prediction of the user's preference/non-preference profiles and multi-dimensional, interpretable scoring of candidate images.

3. Training Strategy

PreferThinker employs a two-stage training strategy to master structured reasoning:

• Stage 1: Cold-Start Supervised Fine-Tuning (SFT): The model (initialized from Qwen2.5-VL-7B) is fine-tuned on the PreferImg-CoT dataset. This stage teaches the model to first predict the user's visual preference profiles from reference images and then use these profiles to generate interpretable, multi-dimensional scores for candidate images.
• Stage 2: Reinforcement Fine-Tuning (RFT) with GRPO: To enhance generalization and explore more reasonable assessment paths, the authors apply Group Relative Policy Optimization (GRPO). This stage incentivizes the model to explore diverse reasoning paths without requiring a critic model.

4. Similarity-Aware Prediction Reward

A key innovation is the Similarity-Aware Prediction Reward. The authors observed that accurate profile prediction facilitates more reasonable assessments. To improve this, they designed a reward function that evaluates the accuracy of the predicted preference profile by measuring:

• Text Similarity: Semantic similarity between the predicted profile and the ground truth (using SBERT).
• Image Similarity: Visual similarity between images generated from the predicted profile versus images generated from the ground truth profile (using DreamSim).
  This reward is combined with format correctness and assessment accuracy rewards to guide the RL training.

Key Contributions

1. PreferThinker System: A reasoning-based framework that achieves accurate personalized assessment and interpretable, multi-dimensional scoring by predicting user preference profiles first.
2. PreferImg-CoT Dataset: The first large-scale CoT-style dataset for personalized assessment, annotated with diverse preference profiles and high-quality reasoning, enabling explicit supervision of structured reasoning.
3. Training Strategy: A novel two-stage approach combining cold-start SFT and GRPO-based RL, augmented by a similarity-aware prediction reward to improve profile prediction and assessment quality.
4. Empirical Superiority: Extensive experiments demonstrating that PreferThinker outperforms state-of-the-art (SOTA) methods on both synthetic and real-user benchmarks.

Experimental Results

The authors evaluated PreferThinker on the proposed PreferImg benchmark (1,500 users, split into seen/unseen and single/multi-preference groups) and the real-user PickaPic dataset.

• Accuracy: PreferThinker achieved the highest accuracy across all settings. On the seen single-preference (SP) subset of PreferImg, it reached 96.6%, outperforming the previous SOTA (ViPer) by 2.4% and the base model by 21.2%.
• Generalization: The method maintained strong performance on unseen users (96.4% on unseen SP), demonstrating robust generalization.
• Real-World Performance: On the PickaPic dataset (where labels reflect general preferences but are grouped by user), PreferThinker ranked second among unseen methods, confirming its generalizability despite the domain shift.
• Ablation Studies: Removing either the SFT cold-start or the RL post-training significantly degraded performance. The inclusion of the similarity-aware prediction reward further improved both profile prediction accuracy and final assessment accuracy.
• Efficiency: Even without explicit CoT generation (inference mode), PreferThinker outperformed SOTA methods while achieving the fastest inference speed (0.80s) compared to other MLLMs.

Significance and Claims

The paper claims that PreferThinker represents the first reasoning-based personalized image preference assessment system that successfully bridges the gap between limited personalized data and complex individual tastes. By introducing the Common Preference Profile, the method leverages large-scale user data to train profile prediction, effectively mitigating data scarcity. The system not only provides accurate assessments but also offers interpretable, multi-dimensional scores grounded in the user's reference images, addressing the "black box" limitation of previous reward models. Furthermore, the predicted profiles are shown to be beneficial for personalized image generation, suggesting the framework's utility extends beyond assessment to content creation. The work establishes a new paradigm for personalized alignment in generative AI, moving beyond simple score regression to structured, explainable reasoning.