InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning

This paper presents InterCoG, a novel text-vision interleaved chain-of-grounding reasoning framework that enhances fine-grained image editing in complex multi-entity scenes by explicitly deducing target locations through text-based spatial reasoning before performing visual grounding and outcome specification, supported by a new dataset and auxiliary training modules to ensure spatial precision.

The Gist

Imagine you are an art director giving instructions to a very talented, but slightly literal-minded, digital painter.

The Problem: The "Which One?" Confusion
You point to a crowded photo of a street scene and say, "Change the shirt of the woman standing between the two kids to green."

Older AI models often get confused. They might think, "Which woman? There are ten of them!" or "Which kids? Are they the ones on the left or right?" Because they struggle to understand the relationships between objects (like "between," "next to," or "holding hands"), they might paint the wrong person's shirt green, or accidentally change the color of the whole street. They are great at painting, but bad at figuring out exactly where to paint.

The Solution: InterCoG (The "Detective Painter")
The paper introduces a new framework called InterCoG. Think of InterCoG not just as a painter, but as a detective who is also a painter. Instead of jumping straight to painting, InterCoG follows a strict, step-by-step detective process before touching the canvas.

Here is how it works, using a simple analogy:

1. The Textual Detective (The "Mental Map")

First, InterCoG reads your instruction and looks at the photo, but it doesn't touch the image yet. It acts like a detective writing a report in its notebook.

• It thinks: "Okay, I see a group of people. There is a woman in the middle. She is holding a child's hand. She is flanked by two other children. That must be the target."
• Analogy: It's like reading a map and saying, "The treasure is under the big oak tree, next to the red bench," before you even start digging. It figures out the logic of the scene using words.

2. The Visual Detective (The "Highlighter")

Once the detective is sure about the location, it doesn't just guess. It pulls out a highlighter and draws a box and a mask over the specific person it identified in step 1.

• It does: It highlights the woman in red on the screen.
• Analogy: This is like putting a "Target Acquired" sticker on the exact person you want to edit. It proves to itself, "Yes, this is the one I was talking about."

3. The Rewriter (The "Clarifier")

Now that it knows exactly who it is editing, it rewrites your original instruction to be super precise.

• It changes: "Change the shirt of the woman between the kids" becomes "Change the blue shirt of the specific woman highlighted in the red box to green."
• Analogy: It's like a translator making sure the painter understands the exact nuance of the request, removing any ambiguity.

4. The Painter (The "Execution")

Finally, with the target locked in, the logic confirmed, and the instruction clarified, the painter goes to work. Because it knows exactly where to look, the result is perfect. The woman's shirt turns green, and the kids next to her stay exactly the same.

Why is this special?

Most other AI models try to guess the location immediately. If the scene is complex (like a crowd, a busy market, or a forest), they get lost.

InterCoG is special because it interleaves (mixes) thinking and seeing. It doesn't just "think" then "do." It thinks, checks its work visually, thinks again, and then does. It's like a chef who tastes the soup, adds a pinch of salt, tastes it again, and then serves it, rather than just guessing the recipe once and hoping for the best.

The "Training School" (GroundEdit-45K)

To teach this AI to be such a good detective, the researchers built a massive training school called GroundEdit-45K.

• They didn't just show the AI pictures and say "fix this."
• They showed the AI pictures and forced it to write out the detective's report (the reasoning steps) before fixing the picture.
• They also created a "test exam" (GroundEdit-Bench) where the AI has to find hidden targets in tricky situations.

The Result

The paper shows that InterCoG is much better at these tricky, "find the needle in the haystack" tasks than previous models. It doesn't just change pixels; it understands the story of the image and edits the right part of the story without messing up the rest.

In short: InterCoG is an AI that refuses to guess. It thinks, verifies, and then acts, ensuring that when you ask for a specific change in a crowded room, it changes exactly what you meant, and nothing else.

Technical Summary

1. Problem Statement

While unified multi-modal models have shown promise in general image editing, they struggle with fine-grained editing in complex, multi-entity scenes. Existing methods often fail when:

• Targets are not visually salient: They cannot distinguish between similar objects (e.g., "the second statue from the right" vs. "the first").
• Spatial reasoning is required: Instructions rely on implicit relationships (e.g., "the woman between two children") rather than explicit object names.
• Localization is imprecise: Current approaches often rely on coarse geometric cues (pre-extracted regions, coordinate prediction) or shallow semantic understanding, leading to edits that affect the wrong entities or alter the surrounding context.

The core challenge is bridging the gap between high-level semantic reasoning (understanding what to edit based on complex instructions) and low-level spatial grounding (precisely locating where to edit in pixel space).

2. Methodology: InterCoG Framework

The authors propose InterCoG, a novel Text-Vision Interleaved Chain-of-Grounding framework. Instead of treating reasoning and generation as separate stages, InterCoG interleaves them to create a coherent reasoning trajectory.

Core Workflow

The framework operates through a four-step interleaved process:

1. Textual Grounding (Reasoning): The model analyzes the image and instruction to perform spatial reasoning purely in text. It deduces the target's identity and location using relational details (e.g., "the person standing behind the car").
2. Visual Grounding (Localization): Guided by the textual reasoning, the model generates a visual output highlighting the target with a bounding box and a semi-transparent mask in pixel space. This acts as a "visual thought" to confirm the location.
3. Instruction Rewriting: The model rewrites the editing description to explicitly specify the intended outcome based on the confirmed target.
4. Final Generation: The model executes the final edit, ensuring the modification is semantically faithful and spatially consistent with the grounded target.

Key Technical Components

• Backbone: Built upon Bagel, a state-of-the-art unified multi-modal model, utilizing its understanding and generation experts.
• Auxiliary Training Modules:
  • Multimodal Grounding Reconstruction Supervision: A shared mask decoder is trained to reconstruct binary masks from textual, visual, and editing features. This forces the model to learn intrinsic grounding features and align text/visual representations.
  • Multimodal Grounding Reasoning Alignment: To prevent "shortcut reasoning" (where the model ignores its own reasoning steps), the authors introduce alignment losses:
    • Text-Vision Alignment: Aligns the textual reasoning features (target location description) with the visual features of the editing region.
    • Vision-Vision Alignment: Aligns the visual grounding reasoning features with the final editing features.
  • These losses ensure that "what is thought is what is done."

3. Key Contributions

1. InterCoG Framework: The first framework to explicitly interleave textual reasoning and visual grounding for image editing, enabling precise localization in complex, non-salient scenarios.
2. GroundEdit-45K Dataset: A new dataset containing 45,000 high-quality editing samples. Unlike previous datasets, it features:
   • Grounding-First Strategy: Targets are selected from complex scenes where standard grounding models fail.
   • Chain-of-Grounding Annotations: Each sample includes detailed text-vision reasoning chains (Scene Description, Target Localization, Editing Description, Post-Edit Description).
3. GroundEdit-Bench: A rigorous evaluation benchmark with 100 human-annotated cases specifically designed to test grounding-aware editing capabilities.
4. Two Auxiliary Training Modules: Novel alignment and reconstruction losses that significantly improve spatial localization accuracy and reasoning interpretability.

4. Experimental Results

The authors evaluated InterCoG against state-of-the-art methods (including GPT-Image-1, Bagel, GoT, InstructPix2Pix, and Qwen-Image-Edit) on GroundEdit-Bench and SmartEdit.

• Quantitative Performance:
  • Editing Grounding Accuracy (EGA): InterCoG achieved 0.88, significantly outperforming the next best (Bagel-SFT at 0.70 and GPT-Image-1.5 at 0.83).
  • Editing Score (ES): Achieved 3.97 (on a 1-5 scale), surpassing all competitors.
  • Human Ratings: Achieved 9.2/10, demonstrating superior user satisfaction.
• Qualitative Results: Visual comparisons show InterCoG successfully handles complex instructions (e.g., "remove the person with their back to the camera near the stone wall") where other models either edit the wrong person or fail to localize the target.
• Ablation Studies:
  • Removing the interleaved reasoning drops EGA to 0.70, proving the necessity of combining text and visual grounding.
  • Removing reasoning alignment leads to inconsistencies between the model's internal reasoning and the final output.
• Efficiency: While InterCoG has slightly higher latency (~2m 16s vs. 1m 50s for Qwen-Image-Edit) due to the reasoning steps, it uses significantly less peak GPU memory and offers a massive gain in precision.

5. Significance

InterCoG represents a paradigm shift in instruction-based image editing by treating spatial grounding as a reasoning task rather than a simple detection task.

• Solves the "Where" Problem: It effectively addresses the limitation of current models that can identify "what" to edit but fail to pinpoint "where" in complex scenes.
• Interpretability: The chain-of-grounding approach provides a transparent reasoning path, making the model's decisions more explainable and debuggable.
• Real-World Applicability: By handling implicit spatial relations and multi-entity scenes, InterCoG moves image editing closer to human-level understanding, making it suitable for professional and complex real-world applications where precision is critical.

In conclusion, InterCoG demonstrates that interleaving text-based reasoning with visual grounding is a powerful strategy for achieving spatially precise, semantically faithful, and fine-grained image editing.