HP-Edit: A Human-Preference Post-Training Framework for Image Editing

1. Problem Statement

While diffusion models have become the standard for image editing via Supervised Fine-Tuning (SFT), they face two critical limitations:

1. Misalignment with Human Preferences: SFT data often includes mixed sources (e.g., cartoons, synthetic images) that do not reflect real-world human aesthetic preferences, leading to outputs that are technically correct but visually unnatural or aesthetically poor.
2. Scalability of Preference Learning: Applying Reinforcement Learning from Human Feedback (RLHF) to image editing is hindered by the high cost of manual annotation required to build large-scale preference datasets. Existing methods struggle to balance task accuracy (e.g., removing an object) with preference alignment (e.g., ensuring the result looks realistic).

2. Methodology: HP-Edit Framework

The authors propose HP-Edit, a post-training framework designed to align image editing models with human preferences using a three-stage pipeline. The core innovation is the use of a Visual Large Language Model (VLM) as an automatic, task-aware scorer to replace expensive manual annotation.

Stage 1: Task-Aware HP-Scorer Development

• Data Collection: A small set of editing triples (Input Image, Instruction, Edited Output) is collected for each of eight editing sub-tasks.
• Human Annotation: Humans rate these triples on a 0–5 scale based on instruction adherence and visual quality.
• VLM Training: A pretrained VLM (e.g., Qwen2.5-VL) is fine-tuned with carefully designed, task-specific system prompts (e.g., specific questions for "Object Swapping" vs. "Bokeh") to approximate human judgments. This model becomes the HP-Scorer.
• Validation: The HP-Scorer is iteratively refined until its scores closely match human judgments, achieving a high Pearson correlation coefficient (0.89 in user studies).

Stage 2: Efficient Human Preference Data Construction (RealPref-50K)

• Dataset Creation: The authors introduce RealPref-50K, a dataset of ~55,000 real-world editing cases covering eight tasks (Addition, Removal, Swapping, Background Replacement, Color Change, Bokeh, Relighting, Style Transfer).
• Hard-Case Filtering: A key insight is that pretrained models often score "easy" cases highly (score 5), providing weak training signals for RL. The pipeline filters out high-score samples, retaining only "hard cases" (lower scores) where the model struggles. This ensures the RL process focuses on difficult, preference-critical scenarios.
• Balancing: The dataset is balanced across MS-COCO object categories (e.g., person, car, cake) to ensure generalizability.

Stage 3: Task-Aware RL Post-Training

• Reward Model: The HP-Scorer serves as the reward function. Scores are normalized to [0, 1] using a sigmoid function.
• Algorithm: The framework employs Flow-GRPO (Group Relative Policy Optimization). Unlike standard Flow Matching which uses deterministic ODEs, Flow-GRPO converts the process into a stochastic SDE to enable online exploration.
• Process: The model generates a group of images for a given prompt. The HP-Scorer evaluates them, and the policy is updated to maximize the relative advantage of high-scoring generations within the group, guided by the task-specific prompts.

3. Key Contributions

1. HP-Edit Framework: A novel post-training pipeline that unifies a VLM-based scorer, an efficient hard-case data construction pipeline, and task-aware RL to align models with human preferences without prohibitive annotation costs.
2. RealPref-50K Dataset: A large-scale, real-world dataset covering eight common editing tasks with balanced object distribution, specifically curated to include challenging "hard cases" for effective RL training.
3. RealPref-Bench: A new benchmark for evaluating real-world editing performance using manually verified instructions and real-world images, addressing the lack of object-balanced benchmarks in the field.
4. Task-Aware Scoring Prompts: The development of specific prompting strategies for the HP-Scorer that decompose complex editing tasks into verifiable criteria (e.g., checking for "color bleed" in color changes or "background preservation" in object swapping).

4. Experimental Results

The framework was evaluated on RealPref-Bench and GEdit-Bench, using Qwen-Image-Edit-2509 and FLUX.1-Kontext-dev as base models.

• Quantitative Performance:

  • HP-Edit achieved an Overall HP-Score of 4.667 on RealPref-Bench, outperforming the strong baseline Qwen-Image-Edit-2509 (4.472) and other SOTA methods like Step1X-Edit and BAGEL.
  • Significant improvements were observed in tasks requiring fine-grained realism, such as Bokeh (4.545 vs 4.165), Relighting (3.913 vs 3.54), and Color Change (4.75 vs 4.358).
  • On GEdit-Bench-EN, HP-Edit achieved state-of-the-art results (8.35 overall), surpassing the base model by a clear margin.

• Qualitative Improvements:

  • Visual comparisons show HP-Edit produces more realistic results with fewer artifacts, better scene structure preservation, and higher adherence to complex instructions compared to baselines.
  • Baselines often introduced distortions in large-area edits or produced stylized/painted looks, whereas HP-Edit maintained photorealism.

• Ablation Studies:

  • Dataset Filtering: Using the filtered RealPref-50K (hard cases) instead of raw data significantly improved training stability and reward gains.
  • Scorer Quality: Replacing a generic scorer with the task-aware HP-Scorer yielded the highest performance, confirming the necessity of specialized prompts.
  • RL Algorithm: Flow-GRPO outperformed standard DPO, demonstrating the benefit of online sampling and group-relative optimization in flow models.

5. Significance

HP-Edit addresses a critical bottleneck in generative AI: the transition from "technically capable" models to "human-preferred" models. By leveraging VLMs as scalable, task-aware reward models, the paper demonstrates that high-quality human alignment can be achieved without massive manual annotation budgets. The introduction of RealPref-50K and RealPref-Bench provides the community with essential resources for training and evaluating future image editing systems, setting a new standard for real-world, preference-aligned image generation. The framework is particularly significant for applications requiring high-fidelity realism, such as professional design, marketing, and content creation.