ScaleEdit-12M: Scaling Open-Source Image Editing Data Generation via Multi-Agent Framework

The paper introduces ScaleEditor, a fully open-source hierarchical multi-agent framework that generates the largest image editing dataset to date (ScaleEdit-12M), demonstrating that agentic pipelines can achieve commercial-grade data quality and significantly improve model performance without relying on costly proprietary APIs.

The Gist

Imagine you want to teach a robot artist how to paint exactly what you imagine. You tell it, "Change the sky to sunset," or "Remove that ugly sign," or "Turn this photo into a comic book."

For a long time, teaching these robots was like trying to train a dog using only a few scraps of food. The data (the "scraps") was either too small, too messy, or too expensive to get because it required paying huge fees to use "super-robots" (commercial AI) to generate the examples.

ScaleEdit-12M is a new project that solves this by building a massive, free, and high-quality training kitchen using a team of robot helpers.

Here is the simple breakdown of how they did it:

1. The Problem: The "Expensive Chef" Dilemma

Previously, to get good training data, researchers had two bad options:

• Option A: Use a cheap, open-source robot to make the data. But this robot was clumsy, making photos that looked weird or didn't follow instructions well.
• Option B: Pay a "Super Chef" (like GPT-4o) to make the data. The results were delicious, but the bill was so high you could only cook a tiny meal. You couldn't feed the whole world.

2. The Solution: The "Robot Assembly Line" (ScaleEditor)

The authors built a new system called ScaleEditor. Think of this not as one robot, but as a factory assembly line run by a team of specialized robot workers who never get tired and cost nothing.

Here is how their "factory" works in three steps:

• Step 1: The Scavengers (Source Expansion)
  Imagine a team of robots scouring the entire internet, libraries, and even generating new pictures from scratch. They gather over 10 million different images—beaches, cities, people, animals—so the training data isn't just about cats and dogs. It's about everything.
• Step 2: The Editors (Multi-Agent Synthesis)
  This is the magic part. Instead of one robot trying to do everything, they use a Task Router (like a foreman).
  • If the image is a building, the foreman sends it to the "Architecture Robot."
  • If the image has text, it goes to the "Typo-Fixing Robot."
  • If the instruction is complex (like "Make the sun look like a melting clock"), it goes to the "Logic Robot."
    Each specialized robot writes a specific instruction and edits the photo perfectly for that job. They work together to create 12 million unique "Before and After" pairs.
• Step 3: The Inspectors (Quality Control)
  Before the data is used, a team of "Inspectors" (another set of AI robots) checks every single photo. They ask:
  • "Did the robot actually do what the instruction said?"
  • "Does the photo look natural, or did it glitch?"
  • "Is the lighting right?"
    If a photo fails even one check, it gets thrown in the trash. Only the perfect 12 million make the cut.

3. The Result: The "Super-Student"

They took this massive, high-quality dataset (called ScaleEdit-12M) and used it to train two popular open-source AI models (UniWorld-V1 and Bagel).

The Analogy:
Imagine you have a student who is okay at art.

• Before: You gave them a few old, blurry magazines to study. They improved a little.
• After: You gave them a library of 12 million perfect, high-definition art books with step-by-step instructions.
• The Result: The student didn't just get a little better; they became a master.

4. Why This Matters

The paper shows that when they tested these new models:

• They beat almost every other open-source model.
• They performed just as well as models trained on expensive, commercial data (which cost millions to make).
• They got really good at tricky tasks, like fixing text in a sign or changing the physics of an object (e.g., making an egg look cracked).

The Bottom Line

This paper proves you don't need to spend a fortune on "Super Chefs" to train great AI. By building a smart, automated team of robot workers (a multi-agent framework), you can cook up a massive, high-quality dataset for free.

It's like turning a small, messy home kitchen into a world-class, automated restaurant chain that feeds the entire AI community, proving that open-source collaboration can rival expensive, closed-door secrets.

Technical Summary

1. Problem Statement

Instruction-based image editing is a critical capability for Unified Multimodal Models (UMMs). However, constructing large-scale, diverse, and high-quality editing datasets faces two major bottlenecks:

• Proprietary Dependency: High-quality datasets often rely on closed-source commercial models (e.g., GPT-4o) for annotation, which is cost-prohibitive for scaling to millions of samples.
• Quality and Diversity Limitations: Existing open-source datasets often rely on fixed, rule-based synthetic pipelines. These suffer from narrow edit types, limited semantic diversity, and poor quality (artifacts, misalignment), failing to match the performance of commercial-grade data.

The core question addressed is: Can we build a large-scale (10M+), diverse, and high-quality image editing dataset using only open-source tools and cost-effective agentic frameworks?

2. Methodology: The ScaleEditor Framework

The authors propose ScaleEditor, a fully open-source, hierarchical multi-agent framework designed for end-to-end dataset construction. It consists of three primary stages:

A. Source Image Expansion with World-Knowledge Infusion

To overcome the narrow domain coverage of existing datasets, the framework expands the source image pool from standard datasets (COCO, OpenImages, SA-1B) to over 10 million unique images using two strategies:

1. Retrieval-Based Expansion: Uses web-scale search engines with both image-based and text-based queries to retrieve real-world, long-tail visual concepts.
2. Synthesis-Based Expansion: Leverages generative models (MetaCaptioner for detailed captions, Qwen-Image for variant generation) to create semantically coherent variations of source images, densifying the data manifold within specific domains.

• Preprocessing: Includes rule-based filtering (resolution, aspect ratio) and deduplication via perceptual hashing.

B. Adaptive Multi-Agent Editing Synthesis

Instead of rigid pipelines, ScaleEditor employs a dynamic routing and synthesis system:

1. Task Router: A Qwen2.5-VL-72B agent analyzes each image and uses a rejection-based strategy to determine which of the 23 predefined editing tasks are applicable. This allows a single image to support multiple diverse editing workflows.
2. Specialized Agents:
   • Instruction Agents: A modular pool of 24 agents (driven by Qwen2.5-VL-72B) generates specific editing instructions tailored to the task (e.g., "Add a beach umbrella" vs. "Change material to glass").
   • Edit Agents: State-of-the-art open-source editing models (Qwen-Image-Edit, FLUX.1 Kontext, Step1X-Edit, Flux-Text) execute the edits.
3. Specialized Workflows:
   • Text-Aware Editing: A three-stage pipeline using PaddleOCR for text detection, an instruction agent for validation, and specialized text-editing models for precise glyph rendering.
   • Knowledge-Infused Reasoning: Uses an instruction decoupling strategy where a "Reasoning Agent" constructs complex queries with explicit reasoning chains, which are then distilled into executable commands by a "Rewriter Agent."

C. Task-Aware Quality Verification

To ensure commercial-grade quality without proprietary APIs, the framework implements a multi-dimensional verification module powered by Qwen2.5-VL-72B:

• Evaluation Dimensions:
  1. Instruction Following: Does the output match the prompt?
  2. Editing Consistency: Is the semantic/structural coherence preserved?
  3. Generation Quality: Are there artifacts, and is the visual fidelity high?
• Filtering Criteria: Samples must achieve a perfect score (3/3) on Instruction Following and at least 2/3 on Consistency and Quality. This rigorous filtering ensures only high-reliability triplets (Instruction-Image-Edit) are retained.

3. Key Contributions

1. ScaleEditor Framework: A novel, fully open-source, hierarchical multi-agent pipeline that integrates world-knowledge expansion, adaptive task routing, and rigorous quality verification. It demonstrates that agentic workflows can rival commercial data generation.
2. ScaleEdit-12M Dataset: The largest open-source image editing dataset to date, containing 12 million rigorously verified instruction-image-edit triplets. It covers 23 task families (e.g., Addition, Removal, Material Change, Text Replacement, Knowledge-Infused Reasoning) across diverse real and synthetic domains.
3. Performance Validation: Comprehensive experiments showing that fine-tuning foundation models on ScaleEdit yields significant improvements, often surpassing models trained on commercial datasets.

4. Experimental Results

The authors fine-tuned two representative UMMs (UniWorld-V1 and Bagel) on ScaleEdit and evaluated them against benchmarks using both open-source and commercial baselines.

• General Editing Benchmarks (GEdit, ImgEdit):

  • UniWorld-V1: Improved by 35.1% on GEdit and 10.4% on ImgEdit compared to its baseline.
  • Bagel: Improved by 10.0% on GEdit and 7.8% on ImgEdit.
  • Comparison: Models trained on ScaleEdit consistently outperformed those trained on other open-source datasets (OmniEdit, AnyEdit, UltraEdit) and matched or exceeded performance of models trained on commercial datasets (ShareGPT-4o, Pico-Banana).

• Knowledge-Infused Benchmarks (RISE, KRIS-Bench):

  • UniWorld-V1: Achieved a massive 150.0% improvement on RISE and 26.5% on KRIS-Bench.
  • Bagel: Showed gains of 23.0% (RISE) and 26.5% (KRIS-Bench).
  • These results highlight the dataset's unique value in teaching models world knowledge and logical reasoning for editing.

• Ablation Studies:

  • Equal-Scale Comparison: Even when resampled to 1M, ScaleEdit outperformed other datasets, proving superior data quality, not just scale.
  • Component Analysis: Disabling the Task Router or Quality Filtering led to performance drops, confirming the necessity of adaptive routing and rigorous verification.
  • Judge Reliability: Qwen2.5-VL-72B was validated as a reliable surrogate for GPT-4o in quality verification, showing high alignment with human preferences.

5. Significance

• Democratization of High-Quality Data: ScaleEdit proves that large-scale, commercial-grade image editing datasets can be constructed without relying on expensive proprietary APIs, making advanced editing capabilities accessible to the open-source community.
• Scalability and Cost-Effectiveness: The multi-agent framework offers a scalable, cost-efficient alternative to manual annotation or commercial API synthesis.
• Advancement of UMMs: The dataset significantly boosts the performance of unified models in both basic editing (color, object removal) and complex reasoning tasks (knowledge-infused edits), bridging the gap between current open-source capabilities and commercial systems.

In conclusion, ScaleEdit-12M and the ScaleEditor framework represent a paradigm shift in data generation for image editing, demonstrating that open-source agentic pipelines can achieve quality and scale previously thought to be exclusive to commercial entities.