💻 computer science

From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

This paper proposes an experiential framework for open-ended image editing that tightly couples a planner generating atomic task decompositions with an orchestrator selecting tools and regions, using a vision-language judge to provide outcome-based rewards that iteratively refine both components for more coherent and reliable multi-step edits.

Original authors: Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but slightly literal-minded artist. If you tell them, "Put a hat on that man," they will happily add a hat. But if you say, "Make this advertisement feel more friendly to people living in the countryside," they might get confused. They might just add a cow, or they might change the whole picture so much that the original brand disappears. They struggle with big, abstract ideas that require a sequence of small, specific changes.

This paper introduces a new system called Plan2Pix that solves this problem by giving the artist a Project Manager and a Quality Control Inspector. Instead of trying to do the whole job in one giant leap, the system breaks the task down, plans it out, and learns from its mistakes.

Here is how it works, step-by-step:

1. The Project Manager (The Planner)

When you give the system a big, vague instruction like "Adapt this ad for a rural audience," the Planner steps in. Think of the Planner as a chef reading a complex recipe. Instead of just guessing, the Planner writes a checklist.

The Trick: The Planner doesn't just guess the steps. It uses a "checklist" (like a set of rules: "Must keep the logo," "Must change the background," "Must update the text") to generate a step-by-step plan.
Self-Learning: Crucially, the Planner learns by practicing on its own checklists. It doesn't just copy a master chef; it practices making its own lists, which helps it get better at understanding what it is capable of doing. This prevents it from getting confused when it tries to do something new.

2. The Foreman (The Orchestrator)

Once the Planner has the checklist, the Orchestrator takes over. Think of the Orchestrator as a construction foreman on a job site. The Planner says, "We need to replace the background with a farm scene." The Orchestrator has to decide:

Which Tool? Should I use a "paintbrush" tool (for the whole image) or a "stencil" tool (for just one part)?
Where? Exactly which part of the image do I paint?

The Orchestrator doesn't just guess. It looks at the result of its actions and asks a Judge if it did a good job.

3. The Judge (The Reward System)

This is the secret sauce. After the Orchestrator makes a change, a smart AI "Judge" looks at the new image and the original instruction. It asks three questions:

Did you follow the instructions? (e.g., Did you actually make it look rural?)
Did you ruin the original? (e.g., Did you accidentally delete the company logo?)
Does it look good? (Is the image blurry or weird?)

If the Orchestrator gets a high score, it remembers, "Hey, using the 'farm stencil' on the background worked well!" If it gets a low score, it learns, "Okay, don't do that again." Over time, the Orchestrator becomes an expert at picking the right tool for the right job.

4. The Safety Net (Refinement)

Sometimes, the Planner might write a step that is impossible (like "Add a flying unicorn" when the tools can only add real animals). The system has a Safety Net that checks the plan before it starts. If a step can't be done with the available tools, it gets cut out of the plan. This ensures the team never tries to do the impossible.

Why is this better than what we have now?

Old Way: You ask a single AI to "Make this ad rural." It tries to do everything at once. It often fails, changes too much, or misses the point.
Plan2Pix Way: It breaks the job into small, manageable tasks (Change background -> Change text -> Add a barn). It uses the right tool for each small task. It checks its work after every step.

The Result

The paper shows that this "Plan, Execute, Check" method creates images that are much more faithful to the user's complex instructions. The final images look professional, keep the original brand identity intact, and actually solve the abstract problem (like making an ad feel "rural") rather than just making random changes.

In short, Plan2Pix turns a chaotic, one-shot attempt into a structured, learning-based project management system that gets better the more it works.

Technical Summary: From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

Problem Statement

Modern diffusion-based image editing models excel at well-specified, concrete transformations (e.g., "change the car color to red") but struggle with abstract, open-ended, and long-horizon instructions (e.g., "adapt this advertisement for a rural audience"). Such tasks require coordinated changes to imagery, text, layout, and context, often involving multiple subtasks and heterogeneous tools. Prior agent-based approaches attempt to decompose these tasks but rely on handcrafted pipelines or teacher-imitation. These methods fix execution orders and heuristics, failing to train the planner on its own distribution or optimize tool selection based on actual editing outcomes. This leads to distribution shifts, limited generalization, and poor scalability for open-ended instructions.

Methodology

The authors propose an experiential learning framework that decouples long-horizon image editing into two tightly coupled stages: Planning and Orchestration. The system interacts with editing tools and receives outcome-based feedback from a Vision-Language Model (VLM) judge to refine its policies.

1. Stage 1: Checklist-Guided Planner (Self-Supervised)

Given an input image $x$ and a high-level instruction $I$ , a multimodal LLM (Planner) generates a structured, ordered sequence of atomic subtasks $P = \{s_1, \dots, s_T\}$ .

Checklist Construction: To ensure comprehensive coverage without relying on brittle handcrafted templates, the planner is prompted with a checklist $C$ specifying criteria a satisfactory edit must meet (e.g., product substitution, semantic alignment).
Self-Supervised Fine-Tuning: Instead of imitating a teacher model (which causes distribution shift), the planner is fine-tuned via self-distillation. It learns to reproduce plans it generates itself under checklist prompting, conditioned only on $(x, I)$ . This aligns training with the model's native generation distribution, improving stability and generalization.
Inference: At test time, the checklist is removed; the fine-tuned planner directly generates structured multi-step plans from $(x, I)$ .

2. Stage 2: Reward-Driven Orchestrator

Conditioned on the plan $P$ , an Orchestrator (another multimodal LLM) selects the appropriate tool $a_t$ and region $r_t$ for each subtask.

Toolset: The framework utilizes a modular set of tools including analysis tools (SAM-2, DeepSeek-OCR, Qwen-Layered, Qwen-BBox) for region discovery, whole-image editors (Flux Kontext, Qwen-Image-Edit), and region-level editors (Flux Inpaint).
Reward Function: A strong VLM judge evaluates the edited image $\hat{x}$ based on three criteria: Instruction Adherence, Identity Preservation, and Visual Quality.
Policy Optimization: The orchestrator is trained to maximize the expected trajectory reward. Since evaluating full trajectories is computationally expensive, the authors introduce two approximations:
1. Additive Reward Approximation: The trajectory reward is approximated as the sum of independent sub-task rewards.
2. Original-Image Independence: Tool contributions are estimated by evaluating them on the original image rather than intermediate edits.
  These approximations allow for the precomputation of rewards for tool-region pairs, enabling tractable policy optimization.

3. Closed-Loop Refinement and Inference

Plan Refinement: Before training the orchestrator, infeasible subtasks (those with maximum achievable rewards below a threshold $\tau$ ) are pruned. The planner is then retrained on these refined plans to align high-level reasoning with the executable action space.
Verifier-Guided Selection: During inference, the orchestrator proposes top- $k$ tool-region candidates. A lightweight verifier (distilled from the VLM judge) scores these intermediate edits, and the highest-scoring edit is selected for the next step. This reduces error accumulation while maintaining efficiency.

Key Contributions

Long-Horizon Image Editing Framework: The paper casts abstract editing as a coordinated planning-and-orchestration problem, enabling multi-step reasoning beyond single-step generation.
Self-Supervised Checklist-Guided Planning: A structured planner learns decompositions from its own checklist-guided samples, reducing distribution shift compared to teacher imitation.
Experiential Orchestrator: A reward-driven policy learns tool and region selection directly from judged editing outcomes rather than handcrafted rules or static prompts.
Closed-Loop Refinement: The system prunes infeasible subtasks based on orchestration feedback, aligning plans with executable actions and achieving state-of-the-art performance.

Experimental Results

The framework was evaluated on the MadVerse dataset (multilingual advertisements) and standard benchmarks like GEdit-Bench and MagicBrush.

Comparison to Baselines: The method outperforms strong single-step editors (Flux Kontext, Qwen-Image-Edit) and multi-step agent baselines (GenArtist, LayerCraft). While single-step models often fail to execute complex instructions or degrade image identity, the proposed method achieves higher instruction-following scores while maintaining strong identity preservation and visual quality.
User Studies: Randomized A/B testing with human participants showed a consistent preference for the proposed method's outputs over baselines.
Ablation Studies:
- Orchestrator Design: Training the orchestrator to learn an explicit tool-selection policy significantly outperforms prompting a base model or using single-tool sequential baselines.
- Plan Source: Checklist-guided self-distillation yields lower perplexity and better distributional alignment than training on teacher-generated plans.
- Plan Refinement: Filtering infeasible subtasks increases the average maximum achievable reward, confirming improved compatibility between planning and execution.
- Inference Branching: Increasing the number of candidate branches ( $k$ ) during inference consistently improves instruction satisfaction and visual quality.

Significance and Claims

The paper claims that by tightly coupling planning with reward-driven execution, the approach yields more coherent and reliable edits than single-step or rule-based multi-step baselines. The significance lies in moving away from handcrafted pipelines and teacher imitation toward an experiential learning paradigm. By learning from the actual outcomes of editing tools via a learned judge, the system grounds its decisions in empirical performance. This allows the framework to handle abstract, open-ended instructions that require coordinated, multi-step transformations, addressing a critical limitation in current instruction-based image editing systems. The authors emphasize that their method improves through interaction with tools and judged outcomes, offering a scalable path for complex visual reasoning tasks.