💻 computer science

SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision

To overcome the lack of high-quality benchmarks for pixel-level sketch-based image editing, this paper introduces the SI-Data dataset and the SI-Edit framework, which leverage multimodal large language models to synthesize instruction-guided quadruplets and achieve precise, semantically aligned local deformations through a collaborative spatial-semantic learning approach.

Original authors: Weixin Ye, Wei Wang, Hongguang Zhu, Xuecheng Nie

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Weixin Ye, Wei Wang, Hongguang Zhu, Xuecheng Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to give a very specific, tiny instruction to a magical artist who can paint anything you want. You might say, "Make that flower stem bend into a smooth S-shape." But here's the problem: the artist is a bit of a daydreamer. They hear "S-shape" and might bend the stem in the wrong spot, or they might turn the whole plant into a snake instead of just curving the stem. This is the challenge of image editing using artificial intelligence. Scientists have built powerful "generative models"—think of them as digital artists trained on millions of pictures—that can change images based on text. But when you ask for a tiny, precise change, like bending a single leaf or stretching a flame, these digital artists often get confused. They might change the wrong part of the picture, or they might misunderstand exactly how you want it to change.

To fix this, researchers have tried two main tricks. One is to give the artist a sketch, like drawing a line on the screen to show exactly where to bend. The other is to give text instructions, like writing "bend it here." But each trick has a flaw. If you only give a sketch, the artist doesn't know what to do (do you want to copy the leaf, move it, or just bend it?). If you only give text, the artist doesn't know where to do it. This paper, SI-Edit, tries to solve this by teaching the artist to listen to both the sketch and the text at the same time, acting like a super-precise digital sculptor who can reshape an image down to the very last pixel.

The Problem: The "Lost in Translation" Artist

Imagine you are playing a game where you have to describe a drawing to a friend who can only draw what you say. If you say, "Make the tree branch curve," your friend might curve the wrong branch, or they might draw a whole new tree. This is what happens with current AI tools that only listen to text instructions. They are great at big changes, like "turn the sky blue," but terrible at tiny, precise ones. They lack a "spatial anchor"—a way to know exactly which pixel to move.

On the other hand, if you only give a sketch (a line drawn on the screen), the AI knows where to draw but not what to do. Is that line meant to move a leaf? Copy it? Or just bend it? Without a clear instruction, the AI might guess wrong, leading to weird results like a leaf that suddenly turns into a copy of itself, or a stem that vanishes.

The authors of this paper realized that to get perfect, pixel-level control, you need both: a map (the sketch) and a destination (the text). But there was a big hurdle: no one had ever built a training manual for an AI that combined these two things perfectly.

The Solution: A New Training Manual and a New Artist

To fix this, the team first had to build a massive, high-quality training dataset called SI-Data. Think of this as a giant cookbook for the AI. Instead of just showing the AI "before" and "after" pictures with a text description, they created quadruplets—sets of four things that go together:

The original image (the starting point).
The target image (what it should look like after the edit).
A sketch (a simple line showing the exact shape change).
An instruction (the text telling the AI what to do).

They didn't draw these by hand; that would take forever. Instead, they used a clever automated pipeline. They took beautiful photos, asked a smart AI (Qwen3-VL) to invent a specific editing instruction (like "stretch the tent"), and then used another AI (Nano Banana Pro) to actually make the change. Then, they used a mathematical trick called optical flow to automatically draw the "sketch" by seeing how the pixels moved from the original to the new image. This gave them 6,500 perfect examples of how to combine a sketch and a text instruction to make a precise edit.

How SI-Edit Works: The "Same Position" Trick

With this new dataset, they built a new editing tool called SI-Edit. This tool is designed to be a "collaborative" artist. It doesn't just look at the text or the sketch; it looks at them together.

The paper introduces two main tricks to make this work:

The "Task-Trigger" Token: They added a special secret code word, written as <sk>, to the beginning of every instruction. Think of this like a bell that rings before the artist starts painting. When the AI hears "bell" (<sk>), it knows, "Oh, I need to pay extra attention to the sketch lines I'm about to see." This helps the AI understand that the sketch isn't just a decoration; it's a strict rule.
Same Position Encoding (SPE): This is the most important part. Imagine you are tracing a picture on a piece of glass. You have the original picture underneath and your sketch on top. If you don't line them up perfectly, your drawing will be off. The AI usually treats the sketch and the original image as two separate things, which causes them to get "drifted" apart. SI-Edit forces the AI to treat the sketch and the original image as if they are in the exact same spot in its brain. It overlays the sketch on the image and tells the AI, "These two things are at the same coordinates." This stops the AI from getting confused about where the bend should happen.

What They Found: Precision at the Pixel Level

The team tested SI-Edit against other popular tools like SketchEdit, MagicQuill, and FramePainter. The results were clear: SI-Edit was much better at following the rules.

Geometric Precision: When measuring how closely the AI followed the sketch lines, SI-Edit had a score of 4.292 (lower is better), while the next best tool, MagicQuill, scored 7.434. This means SI-Edit's bends and curves were much closer to what the user actually drew.
Semantic Understanding: When checking if the AI actually did what the text said (like "stretch" vs. "copy"), SI-Edit scored 0.0174 on a metric called $\Delta$ CS, beating all other methods.
User Preference: When real people looked at the results, they preferred SI-Edit over the others 81.3% to 94.8% of the time, depending on what they were looking for (visual consistency, accuracy, or image quality).

The paper also showed that this tool could do more than just bend things; it could even change a person's pose, like making a martial artist shift their stance, while keeping their face and clothes looking exactly the same.

The Bottom Line

The authors suggest that by combining a clear map (the sketch) with a clear destination (the text), and by teaching the AI to treat them as one unified guide, we can finally get image editing tools that are precise enough to handle tiny, complex changes without messing up the rest of the picture. They didn't just build a better tool; they built the first public dataset that teaches AI how to do this, opening the door for future researchers to create even smarter digital artists. The paper concludes that while the problem of "spatial ambiguity" (not knowing where to edit) and "semantic blindness" (not knowing what to edit) was a major roadblock, this collaborative approach successfully clears the path for pixel-perfect editing.

Technical Summary: SI-Edit

Problem Statement

Despite rapid advancements in generative diffusion models, achieving pixel-level precision in sketch-based image editing remains a significant challenge, particularly for fine-grained local deformations. The paper identifies two primary bottlenecks in existing approaches:

Linguistic Bottleneck (Spatial Ambiguity): Text-only instructions (e.g., "bend the stem") lack precise spatial anchors. Without explicit geometric constraints, models often produce semantic drift, inaccurate deformations, or ignore instructions entirely.
Semantic Blindness (Intent Ambiguity): Sketch-only inputs provide geometric constraints but lack explicit semantic intent. A single sketch (e.g., a new contour) can correspond to multiple operations (e.g., copying, moving, or deforming), leading to unintended results.

Furthermore, the field suffers from a data scarcity issue. Existing datasets primarily offer text-image triplets or region-level masks, lacking the high-quality, publicly available quadruplets (Source Image, Target Image, Local Geometric Sketch, Semantic Instruction) necessary to jointly supervise geometric constraints and semantic intent.

Methodology

The authors propose SI-Edit, a collaborative framework designed to resolve both spatial and semantic ambiguities, supported by a newly constructed dataset, SI-Data.

1. SI-Data: A Quadruplet Dataset

To address the lack of training data, the authors introduce SI-Data, a high-quality dataset containing 6.5K quadruplets $(I_{src}, I_{tgt}, S_k, Inst)$ . The construction pipeline is automated and leverages Multimodal Large Language Models (MLLMs):

Source Collection: High-resolution images from Unsplash.
Instruction Generation: Qwen3-VL is prompted to generate spatially explicit editing instructions (specifying location and geometric operation like "bend" or "stretch").
Target Synthesis: Nano Banana Pro executes the instructions on the source images to create target images.
Sketch Extraction: Dense optical flow (via SEA-RAFT) is computed between source and target images. Motion boundaries are extracted using Sobel operators to generate binary sketches that serve as fine-grained spatial anchors.
Augmentation: The pipeline generates multiple linguistically diverse instructions for each edit to ensure robustness.

2. SI-Edit Framework

Built upon the FLUX.1 Kontext architecture (a Latent Diffusion Transformer), SI-Edit employs a collaborative spatial-semantic tuning strategy to fuse instructions and sketches:

Task-Trigger Token ( $\langle sk \rangle$ ): A learnable token is prepended to the text instruction. This token acts as a semantic anchor, conditioning the attention mechanisms across both text and image streams to steer the generation toward sketch-guided editing.
Same Position Encoding (SPE): To prevent spatial drift, the source image and the sketch-overlay image are processed independently but assigned identical 3D Rotary Positional Embeddings (RoPE). This ensures that the model maintains strict spatial correspondence between the source features and the sketch constraints at the latent-token level.
Fine-Tuning: The model is fine-tuned using Low-Rank Adaptation (LoRA) on the attention layers of both Double-Stream and Single-Stream blocks, optimizing for the rectified-flow matching objective.

Key Contributions

SI-Data: The first dataset providing quadruplets of (Source, Target, Sketch, Instruction) specifically designed for joint spatial and semantic supervision of geometric deformations.
SI-Edit: A collaborative framework that integrates semantic instructions with precise geometric constraints via a learnable $\langle sk \rangle$ token and the SPE strategy, effectively resolving the ambiguity inherent in single-modality approaches.
Evaluation Protocol: A comprehensive set of metrics designed to measure structural fidelity and semantic adherence, including Localized Chamfer Distance (LCD), Mask IoU, CLIP Score improvement ( $\Delta$ CS), and a novel Background Preservation Score (BPS).

Experimental Results

The authors evaluate SI-Edit against state-of-the-art baselines, including SketchEdit, MagicQuill, and FramePainter.

Quantitative Performance: SI-Edit outperforms all baselines across all metrics. Notably, it achieves a Localized Chamfer Distance (LCD) of 4.292 (lower is better), significantly outperforming SketchEdit (10.957) and FramePainter (7.979). It also achieves the highest Mask IoU (0.362) and Background Preservation Score (0.648), indicating superior alignment with sketches and better preservation of unedited regions.
Qualitative Results: Visual comparisons show that SI-Edit successfully executes complex local deformations (e.g., bending tree branches, stretching flames, enlarging objects) while maintaining realistic textures and object identity. In contrast, baselines often suffer from semantic blindness (blurry artifacts) or hallucinate incorrect textures.
User Study: In a pairwise comparison of 60 test cases, SI-Edit was preferred over all baselines in terms of visual consistency, editing accuracy, and image quality (preference rates ranging from 81.3% to 94.8%).
Ablation Studies: The study confirms that placing the learnable $\langle sk \rangle$ token at the Beginning-of-Sequence (BOS) significantly improves semantic alignment ( $\Delta$ CS) compared to End-of-Sequence (EOS) placement or fixed prompts.

Significance and Claims

The paper claims that SI-Edit represents a significant step toward reliable, pixel-level local image editing. By introducing a collaborative guidance mechanism that unifies semantic intent with geometric constraints, the work addresses the core limitations of current single-modality methods.

The authors emphasize that the release of SI-Data and the accompanying evaluation protocol provides a transparent benchmark intended to facilitate future research in sketch-instruction guided image editing. The work demonstrates that precise local editing requires the collaborative use of both modalities: instructions specify what should change, while sketches indicate where and how the change should occur geometrically.