💻 computer science

IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment

The paper introduces IEA, a conversational image editing agent trained via a three-stage multitask alignment pipeline that leverages parameterized tools to produce transparent, interpretable edits, achieving superior instruction following and perceptual quality compared to existing generative and tool-calling methods.

Original authors: Zichen Zhu, Yuheng Sun, Mingxuan Zhu, Wenjie Ma, Situo Zhang, Zhexiang Wang, Ziyue Yang, Danyang Zhang, Kunyao Lan, Zihan Zhao, Dingye Liu, Siqi Xiang, Lu Chen, Kai Yu

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zichen Zhu, Yuheng Sun, Mingxuan Zhu, Wenjie Ma, Situo Zhang, Zhexiang Wang, Ziyue Yang, Danyang Zhang, Kunyao Lan, Zihan Zhao, Dingye Liu, Siqi Xiang, Lu Chen, Kai Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Smart Assistant" vs. A "Magic Wand"

Imagine you have a photo that looks a little dull. You want to fix it.

The Old Way (Professional Software): You open a complex program like Photoshop. It's like being handed a full mechanic's toolbox with 500 wrenches. You have to know exactly which wrench to use and how hard to turn it. If you don't, you might break the engine.
The "Magic" Way (Generative AI): You tell an AI, "Make it look like a sunset." It tries to re-paint the entire picture from scratch. Sometimes it works, but often it hallucinates—adding extra fingers, weird textures, or changing the person's face entirely. It's like hiring an artist who ignores your specific instructions and just paints whatever they feel like.
The IEA Way (This Paper): This introduces IEA, a conversational agent that acts like a highly skilled, patient photo editor sitting next to you. You say, "Make the sky warmer," and instead of re-painting the sky, it knows exactly which specific dial to turn (the "Temperature" knob) and by how much. It doesn't guess; it uses real tools.

How IEA Learned Its Job (The Three Stages)

The researchers didn't just tell the AI to "be good." They taught it in three distinct steps, like training a new employee:

Stage 1: The "Shadowing" Phase (Supervised Fine-Tuning)

The Analogy: Imagine a master chef (an expert human editor) is cooking a meal. The new apprentice (the AI) watches closely. The chef says, "I'm adding a pinch of salt and turning the heat down." The apprentice writes this down.
What happened: The researchers took thousands of photos that experts had already fixed. They used a powerful AI to figure out exactly which tools and numbers the experts used. They then taught IEA to mimic these steps. This gave IEA a basic understanding of how to use the "knobs" (brightness, contrast, etc.) without making things up.

Stage 2: The "Tasting" Phase (Reinforcement Learning)

The Analogy: Now the apprentice cooks a dish. A "Taste Tester" (a reward system) tries it.

If the dish tastes closer to the master chef's version, the apprentice gets a gold star.
If the apprentice uses too many ingredients when only one was needed, they get a "ding" (a penalty).
If the apprentice writes a recipe summary that makes sense, they get another star.
What happened: The AI tried editing images on its own. A special scoring system checked:

Likeness: Did the result look like the expert's version?
Usefulness: Did the AI use the right tools, or did it waste time turning knobs that didn't help?
Summary: Could the AI explain why it made those changes in simple words?
This step taught IEA to be efficient and accurate, not just lucky.

Stage 3: The "Practice Marathon" Phase (Synthetic Data)

The Analogy: The apprentice has seen 1,000 meals, but what if a customer asks for something weird? To prepare, the chef creates thousands of fake scenarios: "What if we make it super bright but less colorful?" "What if the user says 'make it pop' but actually means 'increase contrast'?"
What happened: The researchers generated hundreds of thousands of fake editing scenarios. They taught IEA to handle a huge variety of requests, from "make it brighter" to "make it look vintage." This made the AI robust enough to handle almost any request an amateur user might throw at it.

What IEA Can Actually Do

Talk to You: You can chat with it. "The shadows are too dark," or "I want a warmer, nostalgic feel."
Use Real Tools: Instead of magic, it uses 16 specific, real-world editing tools (like Brightness, Contrast, Saturation, Temperature). It turns the dials step-by-step.
Show Its Work: Because it uses real tools, you can see the "edit trace." It's like seeing the exact recipe: "I increased brightness by 30 and lowered contrast by 18." You can see exactly what happened, which makes it trustworthy.
Refine Based on Feedback: If you say, "It's still too dark," IEA doesn't start over. It adjusts the previous settings slightly, just like a human editor would.

The Results: Did It Work?

The researchers tested IEA against two types of competitors:

The "Magic Wand" AIs: These tried to re-generate the whole image.
The "Tool-Calling" AIs: These tried to use tools but weren't trained as well.

The Verdict:

Accuracy: IEA followed instructions better than the others. When users said "make it brighter," IEA actually made it brighter without messing up the rest of the photo.
Quality: In user studies, people preferred IEA's results. The images looked more natural and had fewer weird glitches (artifacts) than the "Magic Wand" AIs.
Understanding: IEA was also better at summarizing what the user wanted, acting like a good listener.

In a Nutshell

This paper presents IEA, a tool that bridges the gap between "I have no idea how to edit photos" and "I need professional results." It does this by teaching an AI to act like a conversational assistant that uses real, interpretable editing tools rather than trying to magically re-draw the picture. It learned by watching experts, practicing with a scoring system, and training on thousands of made-up scenarios to become a reliable, amateur-friendly photo editor.

Technical Summary: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment

Problem Statement

Current image editing tools suffer from a polarization gap: professional software (e.g., Photoshop, Lightroom) offers precise control but requires steep learning curves, while automated "one-click" filters lack fine-grained control and personalization. Meanwhile, generative AI models capable of text-to-image editing often introduce artifacts, implausible details, or stylistic drift away from photorealism, and they operate as "black boxes" that offer little insight into why an edit was made. There is a critical need for an agent that can interpret natural language instructions to perform non-destructive, parameterized edits using standard tools, thereby bridging the gap between amateur intent and professional-quality outcomes while maintaining interpretability.

Methodology

The authors propose IEA (Image Editing Agent), a Vision-Language Model (VLM) driven agent trained via a three-stage multitask alignment pipeline. IEA operates within a simulated image editing environment (mimicking Adobe Lightroom) that supports 16 global adjustment parameters (e.g., brightness, contrast, temperature, saturation).

1. Simulation Environment

The agent interacts with a lightweight, matplotlib-based editor capable of processing images in 50–300ms. It supports 16 tools with integer parameters ranging from -100 to 100. This environment allows for real-time reward calculation and feedback during training.

2. Three-Stage Training Pipeline

The training process is divided into three distinct stages to progressively align the model with human preferences and tool usage:

Stage 1: Policy Initialization (Supervised Fine-Tuning - SFT)
- Data: A high-quality dataset of ~29k samples distilled from the GIER dataset. This involves using a strong VLM (GPT-4.1) to infer missing tool parameters and reasoning steps from expert retouching pairs. A lightweight heuristic parameter search is employed to minimize the sim-to-real gap by refining distilled parameters to match expert outcomes more closely.
- Tasks: The model is trained on two dual tasks:
  1. Image-Edit: Given an image and instruction, predict tool parameters and reasoning.
  2. Image-Summary: Given an original and edited image, summarize the user's intent into a natural language instruction.
- Goal: Initialize the policy with reasonable prior knowledge of tool selection and parameter magnitudes.
Stage 2: Policy Optimization (Reinforcement Learning - GRPO)
- Algorithm: Group Relative Policy Optimization (GRPO) is used to refine the policy.
- Rewards:
  1. Likeness Improvement ( $R_L$ ): Measures how much closer the edited image is to a reference expert edit compared to the original.
  2. Usefulness ( $R_U$ ): Penalizes redundant or ineffective tool calls by checking if removing a specific tool increases the distance to the reference.
  3. Alignment ( $R_A$ ): A lightweight reward model evaluates the semantic consistency and specificity of the summarized instructions against ground truth.
- Goal: Align the agent's parameters with perceptual goals and encourage efficient tool usage.
Stage 3: Policy Generalization (Large-Scale Synthetic Supervision)
- Data: To address the limited coverage of the 16 tools in the distilled dataset, the authors synthesize ~400k samples. This involves sampling tool subsets and parameter values, generating expert-style instructions, and paraphrasing them into natural, amateur-style commands using GPT-4.1.
- New Task: Image-Refine. The model is trained to interpret updated user feedback (e.g., "The brightness is too strong") and incrementally adjust the tool parameters from a previous state.
- Goal: Broaden instruction coverage, enable iterative refinement, and improve generalization across diverse user intents.

Key Contributions

IEA Framework: A VLM-driven agent that analyzes images and instructions to select appropriate editing tools and parameters, summarizes user intents, and refines results based on feedback, operating entirely within a parameterized tool space rather than synthesizing pixels.
Dataset Construction: Creation of a high-quality dataset (~~29k samples) distilled from GIER and a large-scale synthetic dataset (~~400k samples) covering diverse tool interactions and instruction styles.
Three-Stage Training Recipe: A novel pipeline combining SFT for initialization, GRPO with multi-objective rewards (likeness, usefulness, summary alignment) for optimization, and synthetic data for generalization and refinement capabilities.
Interpretability: By generating explicit tool calls and reasoning traces, IEA provides transparent edit paths that can be inspected and debugged, unlike generative models.

Experimental Results

The authors evaluated IEA on the GIER and FiveK datasets across quantitative metrics and user studies.

Quantitative Performance:
- Image-Edit: IEA achieved a lower pixel distance ( $L$ ) to reference images (0.1034) compared to strong baselines like GPT-4.1 (0.150) and Qwen2.5-VL-7B (0.158). It also attained positive likeness improvement ( $R_L$ ) and usefulness ( $R_U$ ) rewards, indicating efficient and effective editing.
- Image-Summary: IEA achieved a higher ROUGE-L score (0.258) and alignment reward ( $R_A$ ) than baselines, demonstrating superior ability to capture user intent.
User Study:
- In a study with 56 participants comparing 50 samples, IEA ranked highest among tool-calling methods for instruction following.
- In overall perceptual quality, IEA outperformed all baselines, including generative methods (e.g., GPT-Image-1, Qwen-Image-Edit). While generative models sometimes excelled in recovering unrecoverable details in extreme cases, they frequently introduced artifacts and stylistic drift. IEA produced stable, natural results by avoiding over-editing and erroneous tool invocations.

Significance and Claims

The paper claims that IEA validates interpretable, tool-centric VLMs as a reliable path to human instruction-guided image retouching. By combining the understanding capabilities of VLMs with the precision of symbolic, controllable tools, the approach offers a solution that is both amateur-friendly (via natural language) and professionally robust (via parameterized, non-destructive editing). The authors position this as a significant step toward bridging the gap between user intent and editing outcomes, offering a transparent alternative to the "black box" nature of generative image editing.

The work acknowledges limitations, including the current restriction to 16 global tools (lacking local/semantic control) and a simplified simulation backend, but establishes a foundation for future work involving richer toolsets, realistic backends, and human-in-the-loop refinement.