🤖 AI

EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing

EdiVal introduces an automated, object-centric framework and benchmark for the fine-grained, multi-turn evaluation of instruction-based image editing, addressing current limitations by decomposing images into semantic objects to precisely measure instruction following, content consistency, and visual quality without relying on paired reference images.

Original authors: Tianyu Chen, Yasi Zhang, Zhi Zhang, Peiyu Yu, Shu Wang, Zhendong Wang, Kevin Lin, Xiaofei Wang, Zhengyuan Yang, Linjie Li, Chung-Ching Lin, Jianwen Xie, Oscar Leong, Lijuan Wang, Ying Nian Wu, Mingyua

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyu Chen, Yasi Zhang, Zhi Zhang, Peiyu Yu, Shu Wang, Zhendong Wang, Kevin Lin, Xiaofei Wang, Zhengyuan Yang, Linjie Li, Chung-Ching Lin, Jianwen Xie, Oscar Leong, Lijuan Wang, Ying Nian Wu, Mingyuan Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical photo editor. You tell it, "Put a horse in this field," and it does. Then you say, "Make the horse a deer," and it changes. Then you say, "Make the deer brown," and it changes again.

The problem is: How do you know if the editor is actually doing a good job?

If you just look at the pictures, it's hard to tell if the editor is getting confused, deleting the wrong things, or making the picture look weird. Current ways of testing these editors are like trying to grade a student's essay by only looking at the spelling, or by comparing it to a single "perfect" essay that might have been written by a different student. They miss the big picture.

This paper introduces EdiVal-Agent, a new, super-smart "AI Teaching Assistant" designed to grade these photo editors with extreme precision.

Here is how it works, broken down into simple analogies:

1. The "Lego Master" Approach (Object-Centric)

Most AI editors just look at the whole picture as a blurry blob. EdiVal-Agent is different. It acts like a Lego Master.

Step 1: Decomposition. Before the editor even starts, EdiVal-Agent breaks the image down into its individual "Lego bricks" (objects). It identifies: "That's a yellow metal sign," "That's a brown wooden pole," "That's a blue sky."
Step 2: The Instruction. It then creates a series of editing tasks based on those specific bricks. "Change the pole to gray," "Add a bench next to the sign."
Step 3: The Grading. When the editor finishes, EdiVal-Agent doesn't just look at the whole image. It checks the specific bricks. Did the pole actually turn gray? Is the sign still there? Did the bench appear in the right spot?

2. The Three-Part Report Card

EdiVal-Agent gives the editor a report card with three specific grades, using different tools for each:

Grade 1: Did you listen? (Instruction Following)
- The Analogy: Imagine a strict teacher checking a math homework.
- How it works: If the instruction was "Remove the dog," EdiVal-Agent uses a specialized "object detector" (like a high-tech metal detector) to scan the image. If the dog is gone, it gets a point. If the dog is still there, or if a new dog appeared by mistake, it loses points. It combines this hard data with a "smart brain" (a Vision-Language Model) to check if the meaning of the change is correct.
Grade 2: Did you break anything? (Content Consistency)
- The Analogy: Imagine you are renovating a kitchen. You want to paint the cabinets blue, but you don't want to accidentally knock over the fridge or change the floor tiles.
- How it works: This is the hardest part for multi-turn editing. If you ask for three changes in a row, the editor often accidentally messes up the first two. EdiVal-Agent tracks every single "brick" that wasn't supposed to change. It checks: "Is the yellow sign still yellow? Is the floor still the same?" If the editor accidentally changed the sign to red while trying to fix the pole, EdiVal-Agent catches it immediately.
Grade 3: Does it look good? (Visual Quality)
- The Analogy: Imagine a professional art critic.
- How it works: Sometimes an editor follows instructions perfectly but makes the picture look like a blurry mess or adds weird glowing artifacts. EdiVal-Agent uses a "Human Preference Model" (an AI trained to think like a human art critic) to score how natural and beautiful the final image looks.

3. The "Multi-Turn" Challenge

The paper focuses heavily on Multi-Turn Editing (doing many edits in a row).

The Problem: Think of it like a game of "Telephone." In the first turn, the AI is fresh and accurate. But by the third or fourth turn, it starts getting confused by its own previous work. It might forget what the original image looked like or start hallucinating (making up things that aren't there).
The Discovery: The authors tested 16 different top-tier AI editors. They found that while some models (like Seedream 4.0 and Nano Banana) are like excellent marathon runners who stay consistent even after many turns, others (like Qwen-Image-Edit) are like sprinters who start fast but crash hard after a few turns, losing consistency and quality.

Why This Matters

Before this paper, we were grading photo editors with a ruler made of rubber. We couldn't trust the scores.

EdiVal-Agent gives us a steel ruler.
It helps developers know exactly why their AI is failing (e.g., "It's great at adding things, but terrible at keeping the background stable").
It creates a Benchmark (EdiVal-Bench), a standardized test that all future AI editors must take, ensuring that the next generation of tools will be smarter, more reliable, and better at following complex instructions.

In short: EdiVal-Agent is the ultimate referee for AI photo editing, making sure that when you ask for a brown deer in a forest, you actually get a brown deer in a forest, and not a purple horse in a kitchen.

1. Problem Statement

Instruction-based image editing has advanced rapidly, but reliable evaluation remains a bottleneck. Current evaluation protocols suffer from two main structural issues:

Dependence on Paired References: Many benchmarks compare model outputs against ground-truth edited images. However, the space of valid edits is multimodal (many correct outcomes exist), and a single reference captures only one realization. Furthermore, references are often synthesized by prior models, inheriting their biases and limitations.
Limitations of Zero-Shot VLMs: Existing methods rely heavily on Vision-Language Models (VLMs) to judge edits. However, VLMs struggle with:
- Spatial Reasoning: They are poor at verifying object positions and relations.
- Hallucinations: They often hallucinate object existence, attributes, or counts.
- Sensitivity: They miss subtle, pixel-level changes crucial for content consistency.
- Aesthetic Calibration: They are often miscalibrated regarding generative artifacts and human aesthetic preferences.

These limitations make it difficult to precisely evaluate multi-turn editing, where a model must execute a sequence of instructions while maintaining consistency with previous turns and the original image.

2. Methodology: EdiVal-Agent

The authors propose EdiVal-Agent, an automated, fine-grained evaluation framework grounded in an object-centric perspective. The pipeline consists of three stages:

Stage 1: Decomposition

Input: A base image.
Process: A VLM (e.g., GPT-4o) parses the image to identify clearly visible foreground objects.
Output: A structured JSON list of objects with attributes (material, color, text, count) and a boolean foreground flag.
Validation: Grounding-DINO is used to validate these objects and detect bounding boxes. Only reliable detections are kept to seed the "All Objects Pool."

Stage 2: Instruction Generation

Mechanism: The agent generates diverse, context-aware editing instructions across multiple turns (default $MAX\_TURNS = 3$ ).
Dynamic Object Pools: The system maintains three evolving pools:
1. $P_{all}$ : All objects ever present.
2. $P_{unch}$ : Original objects not yet edited (for consistency checks).
3. $P_{avail}$ : Objects currently editable.
Instruction Types: The system samples from 9 instruction types across 6 categories:
- Subject-centric: Add, Remove, Replace.
- Attribute-centric: Color Alter, Material Alter.
- Relational: Position Change.
- Counting: Count Change.
- Text: Text Change.
- Global: Background Change.
Adaptivity: If a turn requires a background change, the system disables background consistency scoring for subsequent turns and restricts future edits to foreground objects to ensure comparability.

Stage 3: Evaluation

The framework introduces three novel metrics:

EdiVal-IF (Instruction Following): Measures if the instruction was executed.
- Symbolic Tasks (Add, Remove, Replace, Position, Count): Uses an Open-Vocabulary Object Detector (Grounding-DINO) to check for the presence/absence of objects, bounding box counts, and geometric relations (e.g., $x_A < x_B$ ).
- Semantic Tasks (Color, Material, Text, Background): Uses a VLM (Qwen2-VL) on detector-guided object crops to verify semantic changes (e.g., "Is the pole gray?").
- Scoring: The multi-turn success rate is the logical AND of scores across all turns in the chain.
EdiVal-CC (Content Consistency): Measures the preservation of non-target content.
- Calculates semantic similarity between the base image ( $I_0$ $I_{0}$ ) and the current image ( $I_t$ $I_{t}$ ) for:
  - Unchanged Objects: Objects in $P_{unch}$ are cropped and compared using DINOv3 features.
  - Background: The background region (excluding all object bounding boxes from $P_{all}$ ) is compared using pixel-wise L1 or DINO similarity.
- Formula: $EdiVal\text{-}CC = \frac{1}{2} (s_{bg} + \text{avg}(s_{o}))$ .
EdiVal-VQ (Visual Quality):
- Uses HPSv3 (Human Preference Score v3), a model trained on human preferences, to quantify aesthetic quality.
- Reports absolute change ( $\Delta$ ) from the base image to detect "beautification" drift or artifacts.
EdiVal-O (Overall Score):
- The geometric mean of EdiVal-IF and EdiVal-CC to balance instruction following and consistency.

3. Key Contributions

EdiVal-Agent Framework: A fully automated, tool-agnostic pipeline that combines symbolic reasoning (detectors) with semantic reasoning (VLMs) to evaluate multi-turn editing with high precision.
EdiVal-Bench: A new benchmark constructed using the agent, featuring 1,716 instructions across 9 types and 3 turns, evaluated on 572 real-world images. It covers 16 state-of-the-art models (In-Context AR, Flow-Matching, and Diffusion).
Superior Human Agreement: EdiVal-IF achieves 81.3% agreement with human raters, significantly outperforming zero-shot VLMs (75.2%) and CLIP-based baselines (65.4%).
Comprehensive Analysis: The paper provides the first systematic comparison of In-Context Multi-Turn (AR models) vs. Context-Free Multi-Turn (Diffusion/Flow models) and Complex Single-Shot editing.

4. Key Results & Insights

The benchmark evaluated 16 models, including closed-source (GPT-Image-1/1.5, Nano Banana, Seedream 4.0) and open-source (Qwen-Image-Edit, FLUX variants, Step1X-Edit).

Top Performers:
- Seedream 4.0 achieved the best overall performance (Rank 1), balancing strong instruction following, high consistency, and low latency.
- GPT-Image-1.5 significantly improved over GPT-Image-1, particularly in content consistency across turns.
- Nano Banana 2 outperformed its predecessor in instruction following and overall score.
Model-Specific Failure Modes:
- Exposure Bias: Open-source models like Qwen-Image-Edit and Step1X-Edit showed sharp performance degradation in later turns (e.g., Qwen dropped from 78.36 to 41.93 in EdiVal-O from Turn 1 to Turn 3). This suggests they are trained on real images and struggle when conditioned on their own synthetic outputs.
- Consistency Issues: GPT-Image-1 excelled at instruction following but ranked near the bottom in content consistency, often altering untargeted objects.
- Latency vs. Quality: Newer versions often improved robustness at the cost of higher latency (e.g., Nano Banana 2 is ~2.5x slower than Nano Banana).
Multi-Turn vs. Complex Single-Shot:
- Models without exposure bias (e.g., Nano Banana) performed better in Multi-Turn settings (step-by-step refinement).
- Models with strong exposure bias (e.g., Qwen-Image-Edit) sometimes performed better in Complex Single-Shot (concatenated prompts), as they avoided the error accumulation of sequential generation.
Visual Quality Artifacts:
- Luminance Drift: Models like Qwen-Image-Edit exhibited progressive brightening and over-exposure artifacts across turns, detectable via low-level statistics (99.9% luminance quantile) but often missed by VLMs.

5. Significance

Bridging the Evaluation Gap: EdiVal-Agent moves beyond simple pixel-matching or unreliable VLM scoring, providing a robust, interpretable standard for the next generation of image editing models.
Diagnosing Multi-Turn Challenges: The framework explicitly identifies "exposure bias" and "error accumulation" as critical failure modes in current multi-turn editing systems, guiding future research into training strategies (e.g., RLHF with evaluation rewards).
Community Resource: By releasing the code, dataset, and evaluation pipeline, the authors enable the community to rigorously benchmark and improve instruction-based editing, accelerating the transition from research prototypes to practical applications.