DLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model

This paper introduces DLEBench, the first benchmark designed to evaluate Instruction-based Image Editing Models on small-scale object editing through a challenging dataset of 1,889 samples and a refined dual-mode evaluation protocol, revealing significant performance gaps in current models.

The Gist

Imagine you have a magical photo editor that can listen to your voice and change things in a picture. You say, "Make that tiny red bird blue," and poof! It happens. This is the dream of Instruction-based Image Editing Models (IIEMs).

But here's the catch: While these AI models are great at changing big things (like turning a whole sky blue or swapping a car for a truck), they often get completely lost when you ask them to fix something tiny, like a speck of dust, a small logo, or a single button on a shirt. They might change the wrong thing, or they might accidentally ruin the rest of the photo while trying to fix the small detail.

This paper introduces DLEBench (DeepLookEditBench), a new "stress test" designed specifically to see how good these AI models are at editing tiny objects.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Needle in a Haystack" Issue

Think of current image editing benchmarks like a game where you have to find and change a giant elephant in a photo. Most AI models can do this easily.
But DLEBench is like asking the AI to find and change a single grain of rice in a bowl of soup.

• The Issue: When the object is small (occupying only 1% to 10% of the image), the AI often gets confused. It might change the wrong grain of rice, or it might spill the whole bowl of soup while trying to pick up one grain.
• The Goal: The authors built a dataset of 1,889 tiny-object challenges to see which AI models can actually handle this "needle in a haystack" task without messing up the rest of the picture.

2. Building the Test: The "Recipe" for Tiny Edits

Creating these test cases was hard because you can't just ask a human to draw 1,889 tiny edits manually; it would take forever.

• The Solution: The team built a semi-automated assembly line.
  • Step 1: They took existing puzzles (visual reasoning questions like "What color is the scarf?") and turned them into editing instructions ("Change the scarf from red to green").
  • Step 2: They used a special "crop-and-edit" trick. Instead of asking the AI to edit the whole huge photo, they cut out a small piece containing just the tiny object, edited that piece perfectly, and then put it back. This created a "Gold Standard" answer (the Reference Image) that humans could use to grade the AI.
  • Step 3: Humans checked the work to make sure the "tiny object" was actually tiny and the instructions made sense.

3. The Grading System: The "Detective" vs. The "Human"

How do you grade an AI's work? Usually, we use other AIs to judge the results (like a robot teacher grading a student). But the authors found that standard "Robot Teachers" (Large Multimodal Models) are blind to tiny details. They might say, "Looks good!" even if the AI changed the wrong object.

To fix this, they created a Dual-Mode Grading System:

• Mode A: The Tool-Driven Detective. Instead of just looking at the picture, the grading AI is given a magnifying glass, a highlighter, and a ruler (digital tools). It can zoom in, compare pixels, and highlight differences. This helps it see the tiny changes it would otherwise miss.
• Mode B: The Oracle-Guided Pro. In this mode, the grader is given the exact coordinates of the tiny object (like a treasure map). It doesn't have to search; it just looks at the specific spot to see if the edit was done right. This removes the confusion of "where is the object?"

4. The Results: The "Big Fish" vs. The "Small Fry"

The authors tested 10 different AI models (both free and paid) on this new test.

• The Shock: The most famous, expensive, "closed-source" models (like the ones from Google or OpenAI) were not the best at this task. In fact, some open-source models actually performed better!
• The Main Finding: Almost all models struggled.
  • Localization Failure: Many models couldn't even find the tiny object. They changed the background instead.
  • Over-Modification: Some found the object but ruined its shape or texture while changing its color.
  • The "Change Count" Nightmare: Asking an AI to change the number of tiny objects (e.g., "Add two more tiny bees") was the hardest task of all. Most models failed miserably.

5. Why This Matters

Imagine you are using an AI to fix a photo of your family. You want to remove a tiny speck of dust from your grandmother's glasses.

• Without DLEBench: The AI might remove the whole face, or change the color of the glasses to purple.
• With DLEBench: We now have a way to measure exactly how good an AI is at these delicate tasks. This forces developers to build better models that can handle the "fine print" of image editing, not just the big picture.

Summary

DLEBench is a new, difficult exam for AI image editors. It proves that while AI is getting smarter at big changes, it is still terrible at small, precise ones. The paper also gives us a better way to grade these AIs, ensuring that in the future, when we ask for a tiny fix, we actually get a tiny fix—and not a disaster.

Technical Summary

1. Problem Statement

While Instruction-based Image Editing Models (IIEMs) have shown significant progress in adhering to text instructions and reasoning, they suffer from a critical blind spot: small-scale object editing.

• The Gap: Current benchmarks predominantly focus on salient objects with large spatial footprints. However, precise local editing (where target objects occupy only 1%–10% of the image area) is essential for refining details and correcting minor errors without regenerating entire images.
• The Challenge: Existing models often fail to localize these small targets, leading to "Localization Failure" (editing the wrong object) or "Over Modification" (altering unintended details).
• Evaluation Limitations: Standard similarity metrics (e.g., CLIP) and even advanced Large Multimodal Models (LMMs) acting as judges struggle with fine-grained visual perception. They often fail to distinguish minute changes in small regions from background noise, leading to unreliable assessments that do not align with human judgment.

2. Methodology

A. Benchmark Construction: DLEBench

The authors introduce DeepLookEditBench (DLEBench), the first benchmark dedicated to small-scale object editing.

• Data Source & Pipeline: They constructed a semi-automated three-stage pipeline to convert visual reasoning datasets (MME-Realworld, Pixel-Reasoner, V*-Bench) into editing tasks:
  1. Metadata Construction: Using counterfactual synthesis (via GPT-4.1), they transform Visual Question Answering (VQA) pairs into editing instructions, target object definitions, and instruction types.
  2. Reference Image Generation: To overcome the difficulty of generating ground-truth edits for small objects on full images, they employ a Crop-and-Edit strategy. This involves:
     • Manual annotation of bounding boxes (bboxes) for precise localization.
     • An Adaptive BBox Expansion strategy: Dynamically adjusting the crop size based on object scale (larger context for smaller objects to maintain global coherence, tighter crops for larger objects) to balance local focus with global context.
  3. Human Verification: Rigorous human inspection ensures semantic consistency and visual feasibility of the generated reference images.
• Statistics: The dataset contains 1,889 samples where target objects occupy 1%–10% of the image area. It covers 7 instruction types (Change Material, Color, OCR, Shape, Removal, Replacement, Count) and includes complex scenarios like partial occlusion and multi-object editing.

B. Evaluation Protocol

To address the misalignment between automated judges and human evaluation, the authors propose a Dual-Mode Evaluation Framework:

1. Criteria:
   • Instruction Following (IF): Evaluates localization accuracy, action correctness, and preservation of non-target attributes.
   • Visual Consistency (VC): Evaluates the preservation of global non-target regions (background stability and absence of anomalies).
   • Refined Rubrics: Instead of vague scores, they define hierarchical failure modes (e.g., Score 1: Localization Failure, Score 2: Wrong Action, Score 3: Over Modification, Score 4: Flawless Execution) to reduce subjectivity.
2. Evaluation Modes:
   • Tool-driven Mode: The LMM acts as an agent invoking external tools (GroundingDINO, Zoom-In, Difference detection, Enhancer) to iteratively locate targets and detect pixel-level differences, compensating for the LMM's weak visual perception.
   • Oracle-guided Mode: Uses human-annotated bboxes to crop/mask images before evaluation. This decouples the evaluation from the localization step, forcing the LMM to focus solely on the edit quality (IF) or background consistency (VC).

3. Key Contributions

1. DLEBench Benchmark: The first dedicated benchmark for small-scale object editing, highlighting a significant gap in current IIEM capabilities.
2. Robust Evaluation Protocol: A novel dual-mode framework (Tool-driven and Oracle-guided) with refined, objective scoring rubrics that significantly outperform standard "LMM-as-a-Judge" baselines in alignment with human judgment.
3. Comprehensive Analysis: A systematic evaluation of 10 representative IIEMs (including closed-source models like Gemini-3-Pro and open-source models like Bagel-Think, OmniGen2, etc.), providing deep insights into their specific failure modes.

4. Experimental Results

The evaluation of 10 IIEMs on DLEBench reveals several critical findings:

• Performance Gaps: Even state-of-the-art models struggle significantly. Gemini-3-Pro achieved the highest average score (65.55), but many open-source models performed poorly (e.g., MagicBrush: 23.16).
• Localization is the Primary Bottleneck: A large portion of failures (Score 1) across models are due to Localization Failure, where models edit the wrong object or fail to detect the small target entirely.
• Instruction Type Difficulty: "Change Count" (modifying the number of small objects) is the most difficult task for all models due to the complexity of isolating and enumerating multiple small entities.
• Scale Sensitivity: Performance is highly correlated with object size. Most models show a sharp decline in performance as the target object size decreases. Some models (e.g., UniWorld-V1, MagicBrush) show consistently poor performance regardless of scale.
• Trade-off between IF and VC: Models that fail to localize often resort to "aggressive modification," altering non-target regions (low Visual Consistency). Conversely, models that adopt a "conservative strategy" (abstaining from editing when uncertain) often preserve global consistency better but score lower on Instruction Following.
• Evaluation Alignment: The proposed Oracle-guided Mode achieved the highest correlation with human judgments, significantly outperforming standard LMM-as-a-Judge baselines (which often hallucinated "Flawless Execution" when they missed subtle deformations).

5. Significance

• Advancing Precision: DLEBench shifts the focus from broad, semantic editing to fine-grained, precise manipulation, a necessary step for real-world applications like photo retouching and error correction.
• Reliable Benchmarking: The paper demonstrates that current evaluation methods are insufficient for small-scale tasks. The proposed dual-mode framework provides a more reliable, reproducible, and human-aligned standard for future research.
• Future Directions: The results indicate that current IIEMs lack the visual acuity required for small-object editing. Future work must focus on improving spatial localization, fine-grained attention mechanisms, and training data that specifically targets small-scale objects. The authors plan to expand the dataset to balance instruction types and improve diversity.

In summary, DLEBench exposes a critical limitation in the current generation of image editing models: their inability to handle small-scale objects with precision. By providing a specialized benchmark and a rigorous evaluation protocol, the paper sets a new standard for developing and assessing the next generation of fine-grained image editing systems.