🤖 AI

IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection

The paper introduces IMAGAgent, a multi-turn image editing framework that employs a constraint-aware planning, tool orchestration, and multi-expert reflection mechanism to overcome error accumulation and semantic drift, achieving superior performance on the MTEditBench and MagicBrush datasets.

Original authors: Fei Shen, Chengyu Xie, Lihong Wang, Zhanyi Zhang, Xin Jiang, Xiaoyu Du, Jinhui Tang

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fei Shen, Chengyu Xie, Lihong Wang, Zhanyi Zhang, Xin Jiang, Xiaoyu Du, Jinhui Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an artist hired to paint a portrait based on a client's very specific, multi-step instructions.

The Old Way (Existing Methods):
The client says, "First, put a hat on the cat. Then, change the background to a cloudy sky. Finally, make the cat look like it's wearing a tuxedo."

In the old systems, the artist would blindly follow these steps one by one.

Step 1: They put a hat on the cat. But maybe they made the hat too big, squishing the cat's ears.
Step 2: They change the sky. But because they didn't notice the squished ears from Step 1, they accidentally paint the sky over the cat's face.
Step 3: They add the tuxedo. Now the cat looks like a blob with a hat and a suit, but its face is gone.

The problem is error accumulation. The artist doesn't look back to check if the previous step ruined the picture. They just keep going, and by the end, the image is a distorted mess. They also suffer from semantic drift, where they forget the original goal (a cute cat) and just focus on the immediate command, losing the "soul" of the image.

The New Way (IMAGAgent):
The authors of this paper, IMAGAgent, built a super-smart art director to manage this process. Instead of just a painter, you now have a team with a strict workflow: Plan, Execute, and Reflect.

Here is how it works, using our art studio analogy:

1. The Planner (The Architect)

Before the painter touches a brush, the Planner (a Vision-Language Model) looks at the original photo and the client's long, complicated request.

The Magic: It doesn't just say "Do it." It breaks the request down into tiny, safe, atomic steps.
The Rules: It ensures that each step only touches one thing (Target Singularity) and is simple enough to do without confusion (Semantic Atomicity).
Analogy: Instead of saying "Fix the whole room," the Planner says: "1. Move the chair. 2. Paint the wall. 3. Hang the picture." It makes sure you don't try to move the chair while painting the wall, which would be a disaster.

2. The Orchestrator (The Tool Manager)

Once the plan is set, the Orchestrator picks the right tools for the job.

The Magic: It has a toolbox full of different experts: one for finding objects (like a detective), one for cutting things out (like a surgeon), and one for painting (the artist).
The Magic: It dynamically chooses the right tool for the current step. If the task is "find the cat," it uses the detective tool. If the task is "change the cat's fur," it uses the painter.
Analogy: It's like a conductor in an orchestra, making sure the violinist plays when needed and the drummer stays silent, all based on what the picture looks like right now.

3. The Reflection Team (The Quality Control Board)

This is the most important part. After the painter finishes a step, they don't just move to the next one. They show the result to a Panel of Critics (Multi-Expert Reflection).

The Magic: Three different "experts" look at the new image. They check:
- Did we actually follow the instruction?
- Did we accidentally ruin the cat's ears?
- Does it look good?
The Feedback Loop: If the critics say, "Hey, the hat is too big and it's covering the cat's eyes," the system stops. It doesn't move to the next step. It goes back to the painter, says, "Fix the hat size," and tries again.
Analogy: It's like a film director yelling "Cut!" if an actor flubs a line, rather than letting the movie keep rolling with a bad take. The system keeps retrying until the "Quality Score" is high enough.

Why This Matters

The paper introduces a new test called MTEditBench (think of it as a difficult exam for AI artists) with long, complex instructions.

Old AI: Failed the exam because it got confused, forgot the original image, and produced distorted results after a few turns.
IMAGAgent: Passed with flying colors. Because it constantly checks its work, remembers the history of what it changed, and corrects its own mistakes, it can handle long, complex editing chains without the picture falling apart.

In Summary:
IMAGAgent is like upgrading from a robot that blindly follows a list of commands to a highly skilled, self-correcting creative team. It plans carefully, uses the right tools, and constantly critiques its own work to ensure the final result is exactly what the user wanted, without any accidental disasters.

1. Problem Statement

Current multi-turn image editing paradigms suffer from three critical failure modes when handling sequential instructions:

Error Accumulation: Small deviations in early turns propagate through subsequent steps because there is no mechanism to diagnose or correct errors in real-time.
Semantic Drift: As editing progresses, the model often overwrites or violates semantics established in previous turns due to a lack of explicit constraint representation.
Structural Distortion: Long-horizon editing often leads to unintended side effects that corrupt the identity, geometry, and texture coherence of the image.

Existing methods typically operate under a static "one instruction, one edit" regime or rely on rigid, predefined workflows. They lack a closed-loop decision mechanism that explicitly represents, evaluates, and enforces cross-turn constraints, making them unstable for complex, multi-step creative workflows.

2. Methodology: IMAGAgent Framework

IMAGAgent introduces a closed-loop "Plan-Execute-Reflect" framework designed to achieve deep synergy between instruction parsing, tool scheduling, and adaptive correction. The system operates through three core modules:

A. Constraint-Aware Planning Module

Function: Decomposes complex, high-level natural language instructions ( $T_0$ ) into a sequence of atomic, executable sub-tasks ( $\mathcal{A} = \{t_1, t_2, ..., t_n\}$ ).
Mechanism: Utilizes a Vision-Language Model (VLM) as a planner ( $\Psi$ ) that grounds instructions in the visual context of the initial image ( $I_0$ ).
Key Constraints: To ensure executability, the planner enforces three rigorous criteria:
1. Target Singularity: Operations must focus on a single entity or coherent group to avoid confusing downstream tools.
2. Semantic Atomicity: Sub-tasks must be indivisible without losing semantic integrity.
3. Visual Perceptibility: Tasks must result in tangible visual alterations rather than abstract shifts.
Output: A causally consistent, reordered task sequence where prerequisite edits precede dependent ones.

B. Tool-Chain Orchestration Module

Function: Dynamically constructs execution paths to perform the specific sub-tasks.
Mechanism: An agent employs a Chain-of-Thought (CoT) mechanism to select and schedule heterogeneous tools based on the current image state ( $I_{i-1}$ ), the current sub-task ( $t_i$ ), and the historical context ( $\mathcal{C}$ ).
Tool Suite: The system integrates diverse models for:
- Retrieval: Finding reference images (e.g., CLIP-ViT).
- Segmentation/Detection: Precise masking (e.g., SAM, GroundingDINO).
- Editing: Generative modification (e.g., Stable Diffusion, Qwen-Image-Edit, Doubao-Seedream).
Adaptivity: The agent can switch between local and cloud-based tools and adjust parameters dynamically based on the specific visual requirements of the sub-task.

C. Multi-Expert Collaborative Reflection Mechanism

Function: Evaluates intermediate results to suppress semantic drift and trigger self-correction.
Mechanism:
1. Expert Panel: Three distinct VLM experts independently critique the generated candidate image against four dimensions: Semantic Alignment, Perceptual Quality, Aesthetic Assessment, and Logical Consistency.
2. Aggregation: A central Large Language Model (LLM) synthesizes the individual critiques into a unified consensus feedback triplet ( $F_{i}^{(j+1)}$ ), consisting of positive traits to preserve, negative defects to fix, and a quantitative score ( $S$ ).
3. Iterative Refinement:
  - If the score $S \ge \tau_{sr}$ (success threshold), the task is accepted.
  - If $S < \tau_{sr}$ and iterations $< \tau_{it}$ , the system triggers a re-generation loop, using the negative feedback to guide the next attempt.
  - If the iteration limit is reached, a fallback mechanism selects the best historical candidate.

3. Key Contributions

Closed-Loop Framework: Proposed IMAGAgent, the first framework to model multi-turn image editing as a constraint-aware, closed-loop process that explicitly verifies long-horizon visual and semantic constraints.
Novel Modules:
- Constraint-Aware Planning: Ensures sub-tasks are executable and semantically atomic.
- Tool-Chain Orchestration: Enables adaptive scheduling across heterogeneous vision tools.
- Multi-Expert Reflection: Transforms visual feedback into executable correction signals, effectively suppressing error accumulation.
MTEditBench Dataset: Constructed a new benchmark specifically for long-horizon multi-turn editing, containing 1,000 high-quality sequences with depths of at least 4 turns, addressing the lack of deep interaction data in existing datasets like MagicBrush.
Comprehensive Evaluation: Provided systematic quantitative and qualitative assessments demonstrating superior stability and instruction fidelity compared to State-of-the-Art (SOTA) baselines.

4. Experimental Results

The authors evaluated IMAGAgent on MTEditBench and the MagicBrush dataset, comparing it against SOTA models (e.g., ACE++, HQEdit, UltraEdit, VINCIE, OmniGen, GPT-4o).

Quantitative Performance:
- IMAGAgent achieved the highest scores in DINO (visual consistency), CLIP-I (image similarity), and CLIP-T (text-image alignment) across all turns.
- Crucially, while other models showed significant performance degradation as the number of turns increased (semantic drift), IMAGAgent maintained stable scores, proving its ability to preserve structural integrity over long sequences.
Qualitative Results:
- Visual comparisons showed IMAGAgent successfully executing complex multi-step instructions (e.g., removing an object, adding a new one, changing background, and altering style) without the structural distortions or identity loss seen in baselines.
Ablation Studies:
- Removing the Reflection module caused a significant drop in DINO scores, confirming the necessity of the feedback loop for structural stability.
- Removing Constraints led to lower instruction alignment (CLIP-T), showing that rule-based decomposition is vital for complex logic.
- Removing Historical Context caused identity drift, highlighting the need for memory of previous edits.
User Study: In a blind preference test involving 20 experts, IMAGAgent consistently ranked highest in instruction following, visual realism, and overall quality.

5. Significance

IMAGAgent represents a paradigm shift from isolated, single-step image editing to autonomous, multi-turn visual reasoning. By integrating constraint-aware planning with a multi-expert reflection mechanism, it solves the fundamental bottleneck of error accumulation in generative AI.

Practical Impact: It enables reliable, complex creative workflows where users can iteratively refine images without losing control over the final output's structure or semantics.
Scientific Contribution: The introduction of MTEditBench sets a new standard for evaluating long-horizon agent capabilities, moving beyond simple single-turn benchmarks.
Future Direction: The framework lays the groundwork for extending closed-loop reasoning to more complex modalities, such as video editing (temporal consistency) and 3D generation (geometric integrity).

The code and dataset are publicly available, fostering further research into autonomous visual agents.