Technical Summary: Edit-Compass & EditReward-Compass

Problem Statement

Recent image editing models have advanced significantly in instruction following, multimodal understanding, and complex visual manipulation. However, the field faces two critical evaluation gaps:

1. Image Editing Benchmarks: Existing benchmarks often fail to faithfully reflect human judgment, particularly for frontier models. This is attributed to insufficient task difficulty and coarse-grained evaluation protocols that cannot distinguish subtle capability differences. Furthermore, many benchmarks rely on automated metrics or simple MLLM prompts that lack alignment with human reasoning in complex scenarios.
2. Reward Modeling Benchmarks: As reinforcement learning (RL) becomes central to optimizing image editing, reliable evaluation of reward models is crucial. Current reward model benchmarks suffer from distribution mismatches; they often evaluate models on static, cross-model comparisons rather than simulating the realistic, intra-model candidate distributions encountered during RL optimization (e.g., FlowGRPO scenarios).

Methodology

The authors introduce a unified evaluation suite comprising Edit-Compass (for image editing models) and EditReward-Compass (for reward models).

1. Edit-Compass (Image Editing Benchmark)

• Scale and Scope: Contains 2,388 carefully annotated instances spanning 36 fine-grained task categories across six progressively challenging dimensions:
  • General Tasks: Subject addition, removal, replacement, extraction, color/size/material changes, style transfer, and text editing.
  • Dynamic Manipulation: Object movement, swapping, interaction, action changes, and emotion modification.
  • World Knowledge Reasoning: Temporal, causal, game, math, and chemical reasoning tasks.
  • Algorithmic Visual Reasoning: Tasks requiring multi-step logical deduction, such as optimal path identification, convex hull identification, knapsack selection, and word discovery in grids.
  • Multi-Image Tasks: Composition, virtual try-on, and multi-image awareness (transferring attributes across images).
  • Complex Tasks: Compound instructions and "Complex Paint" tasks where users provide visual cues (arrows, circles, masks) alongside text.
• Data Construction: Utilizes a hybrid approach. General and Complex tasks use real-world images with instructions generated by LLMs and verified by humans. Dynamic, World Knowledge, and Multi-Image tasks are designed by experts with source images generated via refined prompts. Algorithmic tasks are programmatically generated with ground-truth annotations derived from algorithmic solutions (e.g., Dijkstra, dynamic programming).
• Evaluation Framework: Adopts a fine-grained, multidimensional framework based on three core dimensions:
  • Instruction Awareness: Instruction following and world knowledge awareness.
  • Visual Consistency: Unedited region consistency and identity consistency.
  • Visual Quality: Naturalness, structural fidelity, and artifact severity.
  • Mechanism: Uses an MLLM-as-judge pipeline with structured chain-of-thought reasoning and specific scoring rubrics (1–5 scale) to generate scalar scores and interpretable rationales.

2. EditReward-Compass (Reward Modeling Benchmark)

• Scale: Contains 2,251 preference pairs, each consisting of an editing instruction and two candidate edited images.
• Sampling Strategy: Designed to simulate realistic RL optimization scenarios. It employs a FlowGRPO-inspired strategy with stochastic differential equations to sample candidate outputs from diverse models (both open-source and proprietary). This ensures the evaluation reflects the intra-model candidate distributions found during policy optimization, rather than just cross-model comparisons.
• Annotation: A rigorous two-stage human annotation pipeline involving eight experts ensures high-quality preference pairs with clear quality differences in instruction adherence and visual consistency.

Key Results

The authors evaluated 29 frontier image editing models and 21 reward models.

Image Editing Models

• Proprietary vs. Open-Source: A substantial performance gap exists. The best proprietary model (Nano Banana Pro) achieved an overall score of 3.99, while the strongest open-source model (Qwen-Image-Edit) reached 2.69.
• Task-Specific Weaknesses: While open-source models perform competitively on basic tasks (General, Dynamic Manipulation), they struggle significantly with:
  • World Knowledge Reasoning: (Nano Banana Pro: 3.89 vs. Qwen-Image-Edit: 1.74).
  • Algorithmic Visual Reasoning: Most models score near the baseline (1.0–1.5), indicating a failure to perform deep visual reasoning and faithful execution of algorithmic edits.
  • Multi-Image Editing: Closed-source models show a clear advantage in cross-image understanding and composition.
• Language: Many models exhibit cross-lingual imbalance (better performance in English), though advanced unified models show more balanced capabilities.

Reward Models

• Native MLLMs vs. Specialized Models: Native multimodal large language models (e.g., Qwen3.5-9B, Qwen3.6-27B) outperform existing open-source reward models explicitly trained on preference data (e.g., EditScore, EditReward).
• Scaling and Reasoning: Larger models generally perform better. Enabling "thinking" (chain-of-thought) inference consistently improves reward model performance, with gains of up to 10.56 points for sparse MoE models.
• Evaluation Protocol: The structured prompts used in EditReward-Compass significantly improve evaluation performance over previous baselines (e.g., EditScore prompts), with gains up to 12.93%.

Significance and Claims

The paper claims to provide a comprehensive and human-aligned framework for evaluating frontier image editing systems and reward models. Its significance lies in:

1. Faithful Assessment: By introducing progressively challenging tasks and fine-grained, reasoning-based evaluation, the benchmark reveals persistent weaknesses in current models (specifically in reasoning and world knowledge) that coarse benchmarks miss.
2. RL Alignment: EditReward-Compass bridges the gap between static evaluation and dynamic RL optimization scenarios, offering a realistic testbed for analyzing reward-guided optimization.
3. Future Direction: The results suggest that while shallow perception-level editing is maturing, the field must focus on deeper reasoning, world knowledge integration, and complex multi-image manipulation. Furthermore, native multimodal LLMs appear to be a more promising foundation for reward modeling than specialized, smaller-scale models trained solely on preference data.

The authors acknowledge limitations, noting their reliance on API-based MLLM judges, which may introduce variability. They propose future work to develop dedicated image-editing judge models to ensure more stable and transparent evaluation without proprietary dependencies.