💻 computer science

Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions

This paper proposes an adaptive task reformulation framework that leverages a multi-modal large language model (MLLM) agent to dynamically analyze and refine poorly formulated image editing instructions into a sequence of optimized operations, thereby significantly improving editing reliability across various models and benchmarks without modifying the underlying generative architecture.

Original authors: Bo Zhao, Kairui Guo, Runnan Du, Haiyang Sun, Pengshan Wang, Huan Yang, Kun Gai, Yixin Cao, Wei Ji

Published 2026-04-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bo Zhao, Kairui Guo, Runnan Du, Haiyang Sun, Pengshan Wang, Huan Yang, Kun Gai, Yixin Cao, Wei Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but slightly literal-minded artist. You ask them to "move the chair to the left," and they do a great job. But if you ask them to "remove the tiny speck of dust on the crown of the statue in the background," they might get confused, accidentally erase the whole statue, or just stare at you blankly.

This is the current state of AI image editing. The models are powerful, but they often fail not because they aren't smart enough, but because the way we ask them to do things is too messy or vague.

This paper introduces a solution called ATR (Adaptive Task Reformulation). Think of ATR not as a new artist, but as a super-smart project manager who stands between you and the artist.

The Problem: The "Bad Brief"

Currently, when you type an instruction into an AI editor, it tries to do it immediately.

The Issue: If your instruction is vague ("make it look better") or the target is tiny ("remove that speck"), the AI gets lost. It's like giving a painter a blurry photo and a scribbled note; even a genius painter will struggle.
The Old Fix: Researchers usually try to build a "bigger" or "smarter" artist (a more powerful AI model) to handle these hard cases. This is expensive and slow.

The Solution: The "Project Manager" Agent

The authors propose keeping the artist the same but adding a Project Manager (an AI Agent) who rewrites your request before the artist sees it.

Here is how this Project Manager works, using three simple strategies:

1. The "Zoom-In" Strategy (Localized Editing)

Scenario: You want to change the color of a tiny button on a jacket.
The Problem: If the AI looks at the whole jacket, the button is too small to see clearly.
The Manager's Move: The manager says, "Wait, don't look at the whole jacket. Let's crop out just the button, zoom in so it fills the screen, change the color, and then paste it back."
Analogy: It's like using a magnifying glass to fix a tiny scratch on a car before painting the whole hood.

2. The "Un-Entangle" Strategy (Spatial Decoupling)

Scenario: You want to remove a chair that is stuck under a table.
The Problem: If the AI tries to "erase" the chair, it might accidentally erase the table legs because they are touching.
The Manager's Move: The manager says, "Let's cut the chair out first, move it aside, fix the empty space where the table was, and then put the chair back if needed."
Analogy: It's like a surgeon carefully separating two tangled wires before cutting one, rather than just hacking at the knot.

3. The "Translator" Strategy (Instruction Rewriting)

Scenario: You say, "Make the scene feel more like a summer vacation."
The Problem: This is too vague. Does the AI add a beach? Change the clothes? Brighten the sun?
The Manager's Move: The manager translates your vague wish into a specific recipe: "Add palm trees, change the sky to blue, and make the lighting warmer."
Analogy: It's like a translator turning a poet's abstract feelings into a clear set of instructions for a builder.

How It Works in Real Life

The system doesn't just pick one strategy; it thinks about your request first.

Analyze: It looks at your image and your text. "Is the target small? Is the instruction vague? Are objects tangled?"
Route: It decides which strategy to use (Zoom, Un-Entangle, or Translate).
Execute: It breaks the big task into small, easy steps, checks the work after every step, and fixes mistakes immediately.
Result: The final image is much better, even though the underlying "artist" AI hasn't changed at all.

Why This Matters

The paper proves that we don't always need bigger, more expensive AI models. Instead, we just need to organize the tasks better.

Before: "Here is a hard job, try to do it." (Result: Failure)
After: "Here is a hard job. Let's break it into three easy steps, zoom in on the tricky part, and check our work." (Result: Success)

The Bottom Line

This paper is a reminder that sometimes, the problem isn't that the worker isn't skilled enough; it's that the instructions were bad. By adding a smart "middleman" that reformulates tasks on the fly, we can make existing AI image editors significantly more reliable, especially for the tricky, detailed jobs that usually cause them to fail.

1. Problem Statement

Instruction-guided image editing has advanced significantly with generative models, yet it frequently fails on tasks that are semantically simple but structurally challenging. The authors argue that these failures are not primarily due to a lack of model capacity, but rather a mismatch between the task formulation (the input image-instruction pair) and the model's effective operating regime.

Common failure modes include:

Small or weakly grounded targets: The model struggles to locate or isolate small objects.
Implicit spatial relations: Instructions relying on relative positions (e.g., "move the chair to the left of the table") often fail if the spatial grounding is ambiguous.
Structural dependencies: Edits requiring the preservation of physics or hidden structures (e.g., removing a chair leg while keeping the seat) often result in broken geometry.
Under-specified instructions: Vague prompts lead to hallucinations or incorrect edits.

Existing solutions typically attempt to fix this by scaling up model parameters or improving training data. This paper posits that the bottleneck lies in how the task is presented to the model, not just the model itself.

2. Methodology: Adaptive Task Reformulation (ATR)

The authors propose ATR, a framework that improves editing reliability without modifying the underlying generative model (backbone). Instead, it uses a Multimodal Large Language Model (MLLM) agent to dynamically reformulate the task and execute it through a structured, feedback-driven process.

The framework consists of three core stages:

A. Query Profiling

Before execution, the agent analyzes the input image ( $I$ ) and instruction ( $T$ ) to generate a structured query profile ( $p$ ). This profile captures:

Edit Target: What object or region is being manipulated?
Key Constraints: Are there structural dependencies, physics requirements, or ambiguity?
Edit Scope: Is the change localized (e.g., removing a small object) or global (e.g., changing the background)?

B. Reformulation Routing

Based on the profile, a Router assigns the task to one of three execution strategies:

Route A (Direct/Rewrite): For tasks that do not require spatial decoupling. The agent decides whether to edit directly or first rewrite the instruction to be more explicit and unambiguous.
Route B (Spatial Decoupling): For tasks involving structural dependencies or complex relocations. The agent decomposes the task into:
- Target isolation (using segmentation tools like SAM).
- Destination estimation.
- Background completion.
- Recomposition of the edited content.
Route C (Localized Editing): For small targets or cluttered scenes. The agent crops a local workspace around the target to increase the effective resolution and signal-to-noise ratio, performs the edit, and then pastes the result back into the original image.

C. Route-Conditioned Agentic Execution

Once a route is selected, a Planner executes a sequence of tool calls in a closed-loop manner.

State Maintenance: The system maintains an intermediate state ( $I_k, T_k, H_k$ ) including the current image, instruction, and history.
Tool Invocation: The agent selects tools (e.g., crop_tool, sam3_tool, smartpaste_tool, refinement_tool) based on the current state.
Feedback & Verification: Intermediate outputs are verified. If an edit fails (e.g., poor boundary blending or incorrect localization), the agent triggers refinement steps or re-localizes the target.
Fallback Mechanism: If the agent fails to converge within a step limit, it falls back to a single-pass edit on the original input to prevent total failure.

3. Key Contributions

Identification of Task Formulation Mismatch: The paper establishes that a significant portion of editing failures stems from poorly conditioned inputs rather than insufficient model capability.
Adaptive Reformulation Framework: They introduce ATR, a model-agnostic framework that profiles queries, routes them to specific strategies (Direct, Decoupling, Localized), and executes them via an agentic loop with intermediate feedback.
Empirical Validation: The method demonstrates consistent improvements across diverse benchmarks and backbone models, proving that better task organization can yield gains comparable to or exceeding larger, more complex models.

4. Experimental Results

The authors evaluated ATR on three benchmarks: ImgEdit (general editing), PICA (physical realism), and RePlan (complex spatial grounding). They tested against baselines including direct models (Qwen-Edit, Nano Banana), reasoning-augmented frameworks (EditThinker), and planning-based methods (RePlan).

Performance Gains:
- On ImgEdit-Hard, Qwen-Edit-ATR achieved a score of 4.13, significantly outperforming the base Qwen-Edit (3.57) and the planning-based RePlan (3.52).
- On PICA, ATR boosted Qwen-Edit's physical realism score from 61.43% to 65.91%.
- On RePlan, ATR improved the consistency score from 2.39 to 4.14, effectively eliminating referential ambiguity.
Efficiency: The method achieves these gains with a low average number of backbone calls (1.35–1.71 per task), outperforming "Best-of-2" sampling strategies.
Ablation Studies:
- Removing the routing logic or the feedback loop significantly degrades performance.
- The combination of Instruction Rewriting, Spatial Decoupling, and Localized Editing covers the majority of failure modes, with no single strategy being universally optimal.

5. Significance and Conclusion

The paper makes a paradigm shift in image editing research by suggesting that task reformulation is a critical, underexplored dimension.

Model-Agnostic Improvement: ATR allows existing, smaller, or less capable models to perform at the level of much larger models (e.g., Nano Banana-ATR approaches Nano Banana Pro) simply by optimizing the input and execution flow.
Robustness: By breaking complex edits into manageable sub-tasks and verifying intermediate steps, the system handles ambiguity and structural constraints far better than direct "one-shot" generation.
Future Direction: The work highlights that the path to more reliable AI editing may lie less in brute-force model scaling and more in intelligent, adaptive orchestration of existing capabilities.

In summary, the authors demonstrate that making image editing easier is achievable by adapting the task to the model's strengths through an intelligent, agentic reformulation process.