🤖 AI

Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

This paper introduces "Analysis-by-Proxy," a framework that reveals how Vision-Language Models used as condition encoders for image editing fail to propagate crucial localization signals in a single forward pass, as these spatial representations remain hidden in intermediate layers rather than reaching the predefined conditioning points used by current pipelines.

Original authors: Yoav Baron, Sara Dorfman, Roni Paiss, Daniel Cohen-Or, Or Patashnik

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yoav Baron, Sara Dorfman, Roni Paiss, Daniel Cohen-Or, Or Patashnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart" Model vs. The "Clumsy" Editor

Imagine you have a brilliant, highly educated architect (the Vision-Language Model, or VLM). This architect is amazing at looking at a photo and a written instruction like, "Remove the bee on the very top of the sunflower," and instantly knowing exactly where that bee is. If you ask the architect to point it out, they get it right 89% of the time.

However, when you hire this architect to work as a foreman for a construction crew (the Image Editing Pipeline), things go wrong. The foreman tells the crew, "Go fix the bee," but the crew ends up painting the wrong flower, removing the whole sunflower, or hallucinating a bee that wasn't there. The crew only gets it right 57% of the time.

The Mystery: Why does the architect know the answer perfectly, but the construction crew fails so badly?

The Paper's Hypothesis: The "One-Way Street" vs. The "Conversation"

The authors discovered that the problem isn't that the architect is bad at finding bees. The problem is how the architect is being asked to do the job.

The Architect's Superpower: The architect was trained to have a conversation. If you ask a question, they think, answer, think again, and refine their answer. This "back-and-forth" (called autoregressive generation) helps them dig deep into their memory to find the exact location of the bee.
The Construction Job: In image editing, the architect is forced to work in silence. They are given the photo and the instruction, and they must spit out a single set of instructions immediately. They aren't allowed to "think out loud" or generate a conversation.

The paper argues that when you force the architect to work in this "silent, one-shot" mode, the specific details about where the bee is get buried deep inside their brain. The final instructions they send to the crew are too vague, even though the architect actually knows the answer.

The Solution: The "Spy" (Analysis-by-Proxy)

Since the architect won't talk to us directly in this silent mode, the researchers invented a Spy, which they call a Proxy.

The Setup: Instead of asking the architect to write a long story (which they can't do in this mode), the researchers put a tiny, super-smart detective (the Proxy) inside the architect's brain.
The Mission: The detective's only job is to look at the architect's internal thoughts (the hidden layers of the neural network) and try to guess the coordinates of the bee.
The Discovery: The detective found that the "bee location" information wasn't in the architect's final thoughts (the last layer). It was hidden in the middle layers, like a secret note passed between friends in a classroom.
- Analogy: Imagine the architect is a library. The final layer is the "Summary Book" at the front desk—it's too general. The middle layers are the specific shelves where the detailed maps are kept. The construction crew was only reading the Summary Book, but the map was actually on Shelf 15.

The "Dynamic" Twist: It's Not Always Shelf 15

The researchers found something even more interesting. The location of the "secret note" changes depending on the question.

If you ask about a bee on a sunflower, the map is on Shelf 15.
If you ask about a bee on a different flower, the map might be on Shelf 18.

The standard editing tools were like a librarian who always checked Shelf 20, no matter what. That's why they kept failing. The "secret" moves around based on the specific image and text.

How They Fixed It

The researchers used their Spy (the Proxy) to find the secret note in the middle layers, read the coordinates, and then physically drew a box around the bee on the photo before handing it to the construction crew.

The Result: When the construction crew saw the box drawn by the Spy, they stopped guessing. They knew exactly where to work. The editing became accurate, and they stopped painting the wrong flowers.

The Takeaway

This paper teaches us that just because a smart AI model knows where something is, it doesn't mean the tools using that model can see it.

Old Way: Ask the smart model for a final summary and hope it's detailed enough. (It usually isn't).
New Way: Use a lightweight "spy" to peek inside the model's middle layers, find the specific details that are hidden there, and give those details directly to the editing tool.

By doing this, the researchers proved that the "clumsy" editing failures weren't because the AI was stupid; it was because the editing pipeline was looking in the wrong place for the answer.

Technical Summary: Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

Problem Statement

Vision-Language Models (VLMs) have become the standard backbone for conditioning diffusion-based image editing models due to their advanced multimodal reasoning capabilities. However, a significant performance gap exists: while standalone VLMs demonstrate strong localization capabilities when allowed to autoregressively generate text, image editing pipelines that utilize VLMs as condition encoders frequently fail to localize edits accurately, particularly in complex, multi-entity scenes.

Current editing pipelines typically treat the VLM as a non-generative encoder, processing inputs in a single forward pass and extracting conditioning signals from a predefined, input-independent subset of layers (often the final layer). The authors hypothesize that this operational mode restricts the VLM from fully exposing its spatial understanding, which is optimized for autoregressive generation. Consequently, crucial localization signals may remain hidden within intermediate representations or fail to propagate to the layers selected for conditioning, leading to mislocalization, hallucination, or incorrect attribute application in the final edited image.

Methodology: Analysis-by-Proxy

To investigate the internal mechanisms of VLMs under single-pass constraints and recover hidden spatial signals, the authors introduce Analysis-by-Proxy, a framework designed to probe VLM representations without relying on autoregressive text generation.

1. The Proxy Model

The framework employs a lightweight, interpretable Q-Former as a proxy for the downstream Diffusion Transformer (DiT). Unlike the DiT, which operates with complex generative objectives, the Q-Former is trained on a tractable auxiliary task: predicting the bounding box coordinates $(x_1, y_1, x_2, y_2)$ of a target object directly from the VLM's hidden states.

Architecture: The Q-Former utilizes learnable queries that cross-attend to a subset of VLM hidden states $\{H^{(l)}\}$ and self-attend.
Training Objective: The proxy is optimized using a combination of Generalized Intersection over Union (GIoU) and L1 loss to predict bounding boxes.
Ground Truth: The ground truth bounding boxes are obtained by prompting the VLM in its native autoregressive mode, ensuring the proxy learns to decode the spatial information the VLM inherently possesses.

2. Investigating Signal Distribution

Using this framework, the authors conduct a granular analysis of where localization information resides within the VLM:

Layer-wise Analysis: By training independent proxies on hidden states from specific layers, they determine which layers contain the most decodable spatial signals.
Attention Decomposition: The authors analyze the attention weights of the Q-Former queries to identify "decisive" attention heads. These heads act as dynamic routers, concentrating attention mass on specific VLM layers and specific tokens depending on the input prompt.
Token-Level Analysis: Within the identified dominant layers, the authors examine which tokens carry the spatial signal, finding that it is sparsely encoded in semantically significant nouns and adjectives defining the target edit.

3. Recovering and Integrating Signals

To validate that these hidden signals can improve editing, the authors propose a two-stage inference pipeline:

Signal Extraction: A single VLM forward pass extracts intermediate hidden states, which the trained Q-Former uses to predict the target bounding box.
Conditioning: The predicted bounding box is visually overlaid onto the source image. This image (with the box) is encoded into the DiT's latent space via a VAE.
Editing: The DiT is conditioned on both the original text instruction and the explicit spatial marker (the box), guiding it to localize the edit precisely.

Key Contributions

Identification of the Localization Gap: The paper empirically demonstrates a 31.5% performance drop in localization accuracy when VLMs are used as single-pass condition encoders compared to their autoregressive capabilities (89.0% vs. 57.5% on a curated multi-entity dataset).
Analysis-by-Proxy Framework: A novel method for analyzing VLMs in non-generative settings. It reveals that spatial knowledge is not lost but is diluted in final layers and dynamically distributed across intermediate layers and specific tokens.
Discovery of Signal Sparsity: The study uncovers that localization signals are highly concentrated. They peak in intermediate layers (specifically layers 15–24 in the Qwen2.5-VL backbone) and are encoded almost exclusively within a few semantically critical tokens, rather than being uniformly distributed.
Practical Recovery Mechanism: The authors demonstrate that distilling these intermediate signals into explicit spatial bounding boxes significantly improves the localization performance of existing editing pipelines, resolving common failure modes like modifying the wrong object or excessive removal.

Results

Quantitative Evaluation: Using a Vision Question Answering (VQA) metric via Gemini 2.5 Pro to assess semantic editing success and LPIPS to measure background preservation, the proposed method achieved the highest mean VQA score among state-of-the-art baselines (including FLUX.2, FIBO-Edit, and Qwen-Image-Edit).
Baseline Comparison: The method outperformed the standard Qwen-Image-Edit pipeline and other conditioning strategies (e.g., norm-averaging across layers). Notably, an autoregressive "Full Description" baseline performed well, corroborating the hypothesis that sequential decoding helps surface spatial cues that are dormant in single-pass modes.
Qualitative Improvements: Visual results show that the proxy-guided approach successfully corrects localization errors, ensuring edits are applied strictly to the intended target objects in complex scenes, whereas baseline methods often failed to distinguish between similar entities.

Significance and Claims

The paper claims to provide a principled foundation for understanding and designing conditioning architectures in text-guided image editing. By revealing that standard extraction strategies (relying on final layers or fixed layer configurations) are fundamentally suboptimal for spatial tasks, the work argues for a shift in how information is extracted from conditioning models.

The authors assert that their findings establish a deeper understanding of VLM internal mechanisms in single-pass regimes. They propose that future conditioning designs should move away from fixed-layer extraction toward input-dependent strategies that dynamically select the most informative representations (layers and tokens) for a given task. This approach aims to balance the efficiency of single-pass inference with the rich spatial reasoning capabilities inherent in VLMs, ultimately enabling more precise and reliable image editing tools.