🤖 AI

Towards Robust Sequential Decomposition for Complex Image Editing

This paper proposes a robust framework for complex image editing that overcomes the limitations of single-turn and error-prone sequential paradigms by leveraging a unified in-context approach, a large-scale synthetic dataset for training decomposed editing sequences, and a sim-to-real generalization strategy to achieve high-fidelity results on complex, multi-step instructions.

Original authors: Zilai Zeng, Mingdeng Cao, Zijie Li, Xiaochen Lian, Yichun Shi, Peihao Zhu, Chen Sun, Peng Wang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zilai Zeng, Mingdeng Cao, Zijie Li, Xiaochen Lian, Yichun Shi, Peihao Zhu, Chen Sun, Peng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef, and a customer hands you a very complicated order: "Take the spicy sauce off the steak, swap the steak for a fish, move the fish to the center of the plate, add a sprig of rosemary, and then change the plate from white to gold."

If you try to do all of that in one giant, chaotic motion, you might spill the sauce, drop the fish, or forget the rosemary. This is what current AI image editors struggle with when given complex instructions. They try to do everything at once and end up making a mess.

On the other hand, if you try to do it step-by-step, you might fix the first step perfectly, but then mess up the second step because the plate is already moving, and by the third step, the whole dish looks ruined. This is called "error accumulation."

This paper introduces a new way to teach AI how to handle these complex "multi-step" image editing orders without making a mess. Here is how they did it, explained simply:

1. The Problem: The "One-Shot" vs. The "Chain Reaction"

The researchers looked at two common ways AI tries to edit images:

The "One-Shot" Chef: Tries to do the whole order in one go. It often misses steps or gets confused by the long list of instructions.
The "Chain Reaction" Chef: Breaks the order into small steps (Step 1: Remove sauce, Step 2: Swap fish, etc.). The problem is that if Step 1 is slightly wrong, Step 2 builds on that mistake, and by Step 5, the image is unrecognizable.

2. The Solution: A "Smart Assistant" with a Memory

The team built a system that acts like a very organized assistant who doesn't just follow orders but remembers the entire history of what has happened so far.

Instead of just saying, "Now move the fish," the system says, "I see we just removed the sauce and swapped the steak. Now, taking that into account, I will move the fish." This "memory" helps the AI correct itself and stay on track, rather than letting small mistakes pile up.

3. The Training Ground: A Digital Toy Box

You can't easily teach an AI to do complex edits on real photos because it's hard to know exactly what the "perfect" result should look like for every single step.

So, the researchers built a digital toy box (using a 3D software called Blender).

They created virtual rooms with virtual objects (chairs, tables, lights).
They programmed the computer to perform thousands of random edits (e.g., "Move the chair," "Change the wall color").
Because it's a computer simulation, they knew exactly what the result should be at every single step.

This gave them a massive library of "perfect" step-by-step editing examples to train their AI model. It's like giving a student a textbook with the answer key for every single math problem, so they learn the process of solving it, not just the final answer.

4. The "Sim-to-Real" Leap

Once the AI learned how to handle these complex, step-by-step tasks in the "toy box," the researchers wanted to see if it could work on real photos (like a picture of your living room).

They taught the AI a little bit about real photos, but mostly relied on the skills it learned in the toy box. The result? The AI could take a complex real-world instruction (like "Move the lamp, change the rug, and add a plant") and break it down into manageable steps, using its "memory" to ensure the final picture looked exactly right.

5. The Secret Sauce: "Context-Guided" Editing

The paper found that the best way to do this wasn't just to break the task into steps, but to use a specific technique called Context-Guided Sequential Editing.

Think of it like this:

Old Way: "Here is the next step. Do it." (If the previous step was slightly off, the AI gets confused).
New Way: "Here is the next step. Also, remember that the lamp is now on the left, and the rug is blue. Use that information to place the plant correctly."

By constantly reminding the AI of the current state of the image while it performs the next step, the system prevents errors from snowballing.

The Bottom Line

The paper claims that by training an AI on thousands of perfect, step-by-step examples in a virtual world, and then teaching it to "remember" the history of its edits, we can finally get computers to follow complex, multi-part instructions for editing photos. It turns a chaotic, error-prone process into a robust, reliable one, allowing the AI to handle tasks that were previously too difficult to get right.

Technical Summary: Towards Robust Sequential Decomposition for Complex Image Editing

Problem Statement
Recent visual generative models have advanced high-fidelity image editing guided by human instructions. However, these models struggle with complex instructions involving combinatorial editing operations or inter-step dependencies. The paper identifies two canonical paradigms that fail under such complexity:

Single-Turn Editing: Attempts to apply all instructed edits in one pass. This often fails to accurately parse complex instructions, leading to incomplete or incorrect edits.
Sequential Editing: Decomposes tasks into simpler steps but suffers from compounding errors. Inaccuracies introduced in early steps propagate through subsequent ones, degrading the fidelity of the final result.

The core challenge is balancing the benefits of multi-step instruction decomposition against the drawbacks of error accumulation.

Methodology
The authors propose a robust solution within a unified in-context editing framework, where a multimodal model generates interleaved sequences of decomposed instructions and their corresponding edited images, conditioning on the full history of contextual inputs.

Synthetic Data Pipeline: To address the lack of high-quality decomposed data for complex tasks, the authors developed a pipeline in Blender.
- Scene Initialization: Procedurally generates 3D scenes with random objects and materials.
- Editing Chain Composition: Applies a sequence of 10 atomic editing operations (e.g., object addition, removal, translation, rotation, color/material change, viewpoint change, background change).
- Inter-Step Dependency Injection: Introduces complexity by requiring later operations to reference results from earlier edits (e.g., "move the object that was recolored" or "add a book where the chair originally was").
- Ground Truth: Because operations are deterministic, the pipeline produces perfectly accurate decomposed sequences and target images, allowing for the curation of 94,000 high-quality editing sequences (60K independent, 34K dependent).
Model Training: The framework utilizes BAGEL, a unified multimodal model. The model is finetuned with three objectives:
1. Instruction Decomposition: Predicting decomposed instructions between intermediate images given the complex instruction and source image.
2. In-Context Editing: Generating intermediate and target images based on decomposed text-image pairs, using a rectified flow objective.
3. Single-Turn Editing: Retaining the ability to edit from source to target in a single step using the complex instruction.
Inference Paradigms: The authors implement and compare three paradigms during inference:
1. Single-Turn Editing: Direct editing from source to target.
2. Full-Context Sequential Editing (FCSE): Decomposes the task into $K$ steps, where each step attends to all previous text and image contexts.
3. Context-Guided Sequential Editing (CGSE): A novel paradigm that mitigates error accumulation. It computes a base velocity ( $v_d$ ) from the source image and the full sequence of decomposed instructions, then applies a separate "context guidance" term derived from previous editing results. This allows the model to modulate the influence of historical edits via a coefficient ( $\gamma_{ctx}$ ), preventing early errors from corrupting subsequent steps.
Sim-to-Real Transfer: To generalize to real-world images, the model is co-trained on a mixture of the synthetic decomposed sequences and real-world single-turn editing pairs (from Pico-Banana).

Key Results

Synthetic Domain:
- Zero-shot sequential editing causes severe performance degradation in state-of-the-art models (e.g., GPT-4o, Gemini 2.5 Flash Image).
- Finetuning on synthetic data enables robust sequential decomposition. Context-Guided Sequential Editing (CGSE) consistently outperforms single-turn baselines and Full-Context Sequential Editing, particularly on tasks with inter-step dependencies.
- CGSE achieves the best GPT-evaluated scores and maintains robustness in similarity metrics (DINO-I/DINO-D) even as task complexity (chain length and dependency ratio) increases.
- Increasing decomposition steps (up to $K=5$ ) generally yields further improvements.
Sim-to-Real Generalization:
- Co-training with real-world data allows the decomposition skills learned from synthetic tasks to transfer effectively.
- On the Complex-Edit benchmark, the co-trained model with 2-step Context-Guided Editing achieves superior Instruction Following (IF) and Identity Preservation (IP) compared to single-turn baselines and models trained only on synthetic data.
- Human preference studies confirm that the 2-step CGSE approach is preferred for instruction following and identity preservation, though it shows a slight trade-off in perceptual quality compared to single-turn editing on real images.

Significance and Claims
The paper claims that robust sequential decomposition is achievable by training on high-quality, decomposed editing sequences and integrating them with a well-designed inference paradigm that manages error accumulation. Specifically:

Sequential Decomposition is Viable: Contrary to the belief that sequential execution inevitably leads to low-fidelity results due to error accumulation, the authors demonstrate that with proper training and the Context-Guided paradigm, decomposition yields robust improvements even as task complexity increases.
Sim-to-Real Transfer: Skills learned from synthetic tasks (specifically the ability to reason over dependencies and decompose instructions) can be successfully transferred to real-world domains through co-training, enhancing complex image editing across broader domains.
Framework Utility: The in-context editing framework provides a unified perspective to study and compare different editing paradigms, revealing that context-guided strategies are essential for balancing the benefits of decomposition with the risks of error propagation.

The authors conclude that while their work primarily focuses on semantic consistency and instruction following, enhancing the perceptual quality of sequentially edited images in the real world remains an area for future work.