CompBench: Benchmarking Complex Instruction-guided Image Editing

This paper introduces CompBench, a large-scale benchmark featuring fine-grained instructions and an MLLM-human collaborative framework to rigorously evaluate and expose the limitations of current models in complex, instruction-guided image editing tasks.

The Gist

Imagine you have a magical photo editor. You can tell it, "Make the dog wear a hat," and it does. That's the current state of AI image editing. But what if you asked it something trickier? Like, "Move the dog to the left of the tree, but make sure he's hiding behind the bush so only his ears show, and change his hat to a red one that matches the sunset?"

Current AI models usually crash and burn on requests like that. They get confused, mess up the background, or just ignore half your instructions.

This paper introduces CompBench, a new "final exam" designed to test just how good these AI editors really are at handling complex, real-world requests. Here is the breakdown using simple analogies:

1. The Problem: The "Baby Picture" Test

Think of existing benchmarks (tests used to grade AI) like baby pictures. They are simple, clean, and have very few objects.

• The Issue: If you train a chef to only cook scrambled eggs on a white plate, they will look like a 5-star chef. But if you ask them to cook a complex banquet with 10 different dishes, intricate plating, and specific dietary restrictions, they might fail miserably.
• The Reality: Current AI tests only ask for "scrambled eggs" (simple edits like "change the dog to a cat"). They don't test if the AI can handle a "banquet" (complex scenes with many objects, occlusions, and specific spatial relationships).

2. The Solution: CompBench (The "Culinary Olympics")

The authors created CompBench, which is like the Culinary Olympics for AI. Instead of simple eggs, they throw complex challenges at the models.

• The Menu: They have 9 different types of difficult tasks, such as:
  • Multi-Object Editing: "Remove the two zebras in the back but keep the one in the front."
  • Action Editing: "Make the giraffe bend its neck down."
  • Viewpoint Editing: "Move the camera to the right to reveal a building hidden behind a tree."
  • Implicit Reasoning: "What would happen if the dog slipped on the snow?" (This requires the AI to imagine physics and consequences, not just swap pixels).

3. How They Built It: The "Architect and the Builder"

Creating these tests was hard because you can't just ask an AI to make them; the AI would make mistakes.

• The Strategy: They used a Collaborative Framework.
  • The Architect (AI): A smart AI looks at a complex photo and suggests an editing instruction.
  • The Builder (Human): A human expert checks the suggestion. If the instruction is vague or the result looks weird, they fix it.
• The Result: They ended up with over 3,000 high-quality, "perfect" examples where the instruction matches the result exactly. This ensures the test is fair.

4. The "Decoupling" Trick

One of their smartest ideas is Instruction Decoupling.

• The Analogy: Imagine giving a builder a messy note: "Fix the room." The builder might fix the wrong thing.
• The Fix: CompBench teaches the AI to break that messy note into four clear blueprints:
  1. Location: Where is the object? (e.g., "Top left").
  2. Appearance: What does it look like? (e.g., "Red and spotted").
  3. Dynamics: What is it doing? (e.g., "Flying").
  4. Objects: Which object is it? (e.g., "The fish").
     By separating these, the AI understands the instructions much better.

5. The Results: Who Passed the Test?

They tested 15 different AI models on this new exam.

• The Findings: Most models failed the hard questions. They could handle simple edits but fell apart when asked to reason about space or complex interactions.
• The Winners: A few models stood out, specifically Bagel, Qwen-Image-Edit, and FLUX.1 Kontext. These models are like the "Master Chefs" who can actually handle the banquet.
• The Failure Mode: The biggest problem found was Hallucination. When asked to move an object, the AI often kept the object in the wrong place or distorted the background (like stretching a building like taffy).

6. Why This Matters

This paper is a wake-up call. It tells us that while AI image editors are getting better, they are still "babies" when it comes to complex reasoning.

• The Future: To build the next generation of tools, we need to stop training AI on simple tasks and start teaching them to understand spatial logic, physics, and complex instructions.

In a nutshell: CompBench is a tough new gym for AI image editors. It proves that most are currently out of shape, but it also shows us exactly what exercises they need to do to become the super-intelligent editors of the future.

Technical Summary

1. Problem Statement

Existing instruction-guided image editing benchmarks suffer from three critical limitations that hinder the evaluation of real-world applicability:

• Lack of Scene Complexity: Current datasets (e.g., MS COCO-based) rely on oversimplified, object-centric scenes with sparse spatial layouts and minimal occlusion. They fail to represent the dense object interactions and natural clutter found in real-world images.
• Limited Instruction and Task Comprehensiveness: Most benchmarks use atomic, simplistic instructions (e.g., "change dog to cat") that lack the contextual reasoning, multi-object interactions, and dynamic state changes required by actual user requests.
• Deficiencies in Edited Image Quality: Existing benchmarks often contain ground truth data with instruction-alignment inaccuracies or visible visual artifacts (distortions, background inconsistencies), leading to misleading performance evaluations.

Consequently, current models often achieve high scores on simplified benchmarks but fail when faced with complex, reasoning-heavy editing tasks.

2. Methodology: CompBench Construction

The authors introduce CompBench, a large-scale benchmark designed to bridge the gap between simplified benchmarks and real-world complexity.

A. Data Generation Framework

The dataset is constructed using an MLLM-Human Collaborative Framework with tailored task pipelines:

1. Source Data: Utilizes the MOSE dataset, filtered for quality (NIQE metrics) and manually verified.
2. Mask Preprocessing: Multi-object annotations are decomposed into single instances. Discontinuous or heavily occluded masks are discarded or refined using MLLMs (e.g., Qwen-VL) to ensure pixel-level precision.
3. Task Pipelines: Four specialized pipelines generate data for diverse scenarios:
   • Local Editing: Object removal, addition, replacement.
   • Action/Scene Spatial Editing: Modifying dynamics, location, or viewpoint.
   • Complex Reasoning: Implicit contextual edits.
   • Multi-editing: Multi-turn and multi-object interactions.
4. Human-in-the-Loop: MLLMs generate initial instructions, which are then validated and refined by human experts to ensure high-fidelity alignment between the instruction and the edited image.

B. Instruction Decoupling Strategy

To enhance instruction clarity, the authors propose a strategy that disentangles editing intents into four key dimensions:

1. Location: Spatial positioning (e.g., "left of the table").
2. Appearance: Visual attributes (color, texture).
3. Dynamics: Motion states (e.g., "flying," "bending").
4. Objects: Specific entity definitions.
   This structured approach converts ambiguous requests into precise specifications while maintaining natural language expressiveness.

C. Dataset Statistics

• Scale: Over 3,000 high-quality image-instruction pairs.
• Complexity Metrics: CompBench significantly outperforms existing benchmarks in average object count per image (13.58 vs. ~9.96 for the next best), object categories, occlusion rate (98.47%), and out-of-frame object rate (86.38%).
• Task Coverage: 5 major categories covering 9 specific tasks (Local, Multi-turn, Multi-object, Action, Location, Viewpoint, Implicit Reasoning).

3. Key Contributions

1. First Large-Scale Complex Benchmark: CompBench is the first benchmark specifically designed for intricate, real-world instruction-guided image editing, featuring high scene complexity and diverse task types.
2. Novel Construction Framework: The MLLM-human collaborative pipeline with an instruction decoupling strategy ensures high-quality, semantically aligned data that addresses the "instruction-image" gap.
3. Comprehensive Evaluation Protocol: The benchmark introduces rigorous evaluation metrics, including foreground-background decoupling (using LC-T, LC-I, PSNR, SSIM, LPIPS) and multi-perspective scoring (GPT-4o, Qwen2.5-VL, and human annotators) for tasks where automatic metrics are insufficient.
4. Critical Insights: The paper provides a deep analysis of current model failures, identifying specific bottlenecks in reasoning and geometric consistency.

4. Experimental Results

The authors evaluated 15 state-of-the-art models (including InstructPix2pix, MagicBrush, SmartEdit, Bagel, FLUX.1 Kontext, Qwen-Image-Edit, etc.) on CompBench.

• Performance Leaders: Bagel emerged as the top performer, achieving top results in nearly 50% of the metrics across 9 tasks. Qwen-Image-Edit and FLUX.1 Kontext also demonstrated superior performance, particularly in complex reasoning and spatial editing.
• Model Limitations:
  • No Dominance: No single model dominates all tasks; performance varies significantly by task type.
  • Multi-turn Degradation: All models showed a notable decline in background consistency during the second turn of multi-turn editing.
  • HQ-Edit Failure: HQ-Edit demonstrated significantly inferior results across nearly all tasks.
• Key Findings:
  • MLLM Integration is Critical: Top-performing models (Bagel, Qwen-Image-Edit) integrate Multi-modal Large Language Models. Models relying solely on CLIP-text alignment often fail to interpret complex instructions or edit the wrong targets.
  • Planner-Executor Misalignment: Even when MLLMs (Planners) correctly identify targets, diffusion models (Executors) often fail at precise masking in high-density scenes, causing background leakage.
  • Reasoning Bottleneck: Complex reasoning capabilities (e.g., Chain-of-Thought) are essential for high-fidelity edits.
  • Geometric Hallucination: Models frequently produce distorted geometries in action and viewpoint editing tasks, indicating a lack of 3D structural priors.

5. Significance and Future Directions

• Benchmarking Standard: CompBench sets a new standard for evaluating image editing models, moving beyond simple object swaps to test spatial reasoning, context understanding, and dynamic manipulation.
• Guidance for Next-Gen Systems: The results highlight that future image editing systems must:
  • Integrate MLLMs for better instruction interpretation.
  • Improve pixel-level grounding stability in cluttered scenes.
  • Enhance reasoning capabilities through reasoning-aware training.
  • Incorporate 3D structural priors or geometric guidance to prevent hallucinations in spatial tasks.

In conclusion, CompBench reveals that while current models have made progress, they remain inadequate for complex, real-world editing scenarios. The benchmark provides the necessary infrastructure and insights to drive the development of the next generation of robust, reasoning-capable image editing systems.