💻 computer science

Faithful Grounded Visual Reasoning via Learned Proxy-Tokens

The paper introduces Composer, a Multimodal Large Language Model that replaces traditional textual coordinates with learned proxy-tokens to bridge the semantic-spatial gap in visual grounding, thereby achieving significantly improved grounding accuracy while maintaining competitive answer performance.

Original authors: Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo, Angelique Loesch

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tom Hodemon, Mohamed Chaouch, Aboubacar Tuo, Angelique Loesch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, but slightly mysterious, robot to look at a photo and answer a question about it. For example: "Is the blue backpack on the right?"

Current "smart" robots (called Multimodal Large Language Models) are great at giving the right answer, but they work like a black box. You ask a question, and they spit out an answer. You have no idea how they found that answer. Did they actually look at the backpack? Or did they just guess based on the word "backpack" appearing often in their training data?

The Problem: The "Fake Map"

To make these robots more honest, researchers tried teaching them to point to where they are looking. Usually, they do this by asking the robot to write down coordinates, like a text string saying "x=100, y=200."

The paper argues this is like asking a human to navigate a city by reading a list of random numbers instead of looking at a map. The robot treats these numbers as just another word in a sentence. Because there is no real connection between the number "100" and the actual pixels of the image, the robot often hallucinates. It might say, "I see the backpack at coordinates 100, 200," even if there is no backpack there. It's lying about where it is looking, even if the final answer happens to be right.

The Solution: "Composer" and the Magic Index Cards

The authors introduce a new model called Composer. Instead of using random numbers as coordinates, Composer uses Learned Proxy-Tokens.

Think of the image as a giant library of tiny picture fragments (pixels).

Old Way: The robot tries to describe a location by shouting a random number. It's like trying to find a book in a library by guessing a random page number.
Composer's Way: The robot is given a set of Magic Index Cards. Each card has a unique name (a "token") that is permanently glued to a specific part of the image.

When the robot needs to talk about the backpack, it doesn't guess numbers. It simply picks up the specific "Index Card" that covers the backpack. It's like the robot has a direct, physical handle on the image. It can say, "I am looking at Card #42," and because Card #42 is physically linked to the backpack in the robot's memory, it cannot lie about what it sees.

How It Works: Building a Chain of Evidence

The robot doesn't just jump to the answer. It builds a step-by-step story, like a detective solving a case:

Find the object: "I'm looking at the backpack (Card #42)."
Check the location: "Is Card #42 on the right side of the room? Yes."
Check the color: "Is Card #42 blue? Yes."
Conclusion: "Yes, the blue backpack is on the right."

Because the robot uses these "Index Cards" (tokens) instead of random numbers, the steps in its story are tightly connected to the actual image. If the backpack isn't blue, the robot can't just guess "blue" and hope for the best; the "Card" it is holding tells the truth.

The New Test: "ComposerGCoT"

The researchers realized that just checking if the final answer was right wasn't enough. A robot could get the right answer for the wrong reasons (like a student guessing the right answer on a math test without showing work).

So, they built a special test dataset called ComposerGCoT. This test doesn't just ask, "Did you get the answer right?" It checks:

Did you follow the right steps? (Reasoning Consistency)
Did you actually look at the right part of the picture? (Grounding Accuracy)

The Results

When they tested Composer against the old "coordinate" models:

Accuracy: Composer got the final answers right just as often as the old models.
Honesty: Composer was 9% better at actually pointing to the correct spot in the image.

The Big Takeaway

The paper concludes that by giving the robot "addressable" handles (proxy-tokens) to grab onto parts of the image, we can stop it from making up fake locations. This makes the robot's reasoning faithful—meaning its explanation matches what it actually sees. This is a big step toward building AI that we can trust to tell us not just what it thinks, but why it thinks it, based on real evidence.

Technical Summary: Faithful Grounded Visual Reasoning via Learned Proxy-Tokens

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant success in Visual Question Answering (VQA) but operate under a "black-box" paradigm, mapping inputs directly to answers without exposing decision-making processes. This lack of transparency hinders deployment in critical domains.

To address this, Grounded Visual Reasoning (GVR) approaches attempt to couple textual rationales with visual grounding information, typically in the form of textual bounding box coordinates $(x, y, w, h)$ . However, the paper argues that current coordinate-based methods suffer from a fundamental "semantic-spatial gap." By treating spatial locations as arbitrary text strings, these methods lack a learnable semantic link to the underlying visual features. Consequently, models often "hallucinate" coordinates that do not correspond to actual image evidence, effectively decoupling the reasoning process from visual perception. Furthermore, existing evaluation protocols predominantly rely on final answer accuracy, which masks these reasoning inconsistencies and allows models to guess correctly based on language biases rather than visual evidence.

2. Methodology: The Composer Framework

The authors introduce Composer, an MLLM designed to enforce faithful, step-by-step grounded reasoning by bridging the semantic-spatial gap.

2.1. Visual Grounding via Learned Proxy-Tokens

Unlike traditional methods that regress textual coordinates, Composer employs a novel mechanism based on learned proxy-tokens.

Mechanism: The image is decomposed into $N$ visual tokens $V = \{v_1, v_2, \dots, v_N\}$ . A corresponding set of $N$ learnable proxy-tokens $R = \{<r_1>, \dots, <r_N>\}$ is added to the LLM's vocabulary.
Indexing: The input sequence interleaves these tokens ( $S = [r_1, v_1, r_2, v_2, \dots, Q]$ ), establishing a strict 1-to-1 correspondence between each proxy-token and its visual feature.
Set-Based Operations: Visual regions are treated as discrete, addressable sets of proxy-tokens. This allows the model to perform implicit set-based operations (intersection, union, subset verification) directly on the visual latent space. For example, spatial reasoning (e.g., "is object A to the left of object B?") is reduced to checking the intersection of their respective proxy-token sets ( $S_A \cap S_{left} \neq \emptyset$ ).
Recursive Reasoning: The model can refer back to proxy-tokens generated in previous reasoning steps, effectively "shrinking" the visual search space to relevant indices.

2.2. Training Strategy

Composer is trained using a two-stage data strategy:

Alignment Pretraining: The model is equipped with fine-grained localization capabilities using diverse vision-language tasks (captioning, referring expression comprehension, and visual spatial reasoning). Bounding box annotations from datasets like COCO and RefCOCO are mapped to the proxy-token vocabulary.
Multi-step Reasoning (ComposerGCoT): To address the lack of intermediate supervision in standard datasets, the authors synthesized ComposerGCoT. This dataset is derived from GQA by distilling scene graphs and functional programs into discrete, visually grounded reasoning chains. Each step is annotated with intermediate answers and visual grounding, resulting in 163K reasoning chains for 55K images.

2.3. Structured Output for Evaluation

To enable rigorous assessment, Composer outputs structured reasoning chains using lightweight XML-style tags (e.g., <objectRecognition>, <spatialLocation>, <attributeColor>) encapsulated within a <thought> block. This format allows for the independent verification of the logical validity and spatial precision of each reasoning step, rather than just the final answer.

3. Key Contributions

Composer Model: An MLLM that utilizes learned proxy-tokens to index the image latent space, replacing arbitrary coordinate regression with discrete, semantically manipulable pointers.
ComposerGCoT Dataset: A specialized, synthesized dataset providing fine-grained supervision for every step of the reasoning chain, enabling holistic assessment of reasoning consistency and grounding fidelity.
Evaluation Protocol: A framework that assesses models across three axes: answer accuracy, reasoning consistency (task sequence accuracy), and visual grounding accuracy (IoU).

4. Experimental Results

The authors evaluated Composer against a coordinate-based baseline (B-Composer) and their proxy-token variant (P-Composer) on the ComposerGCoT validation set.

Reasoning Consistency: There is a strong correlation between correct reasoning paths and correct final answers. For P-Composer, 76.08% of correct reasoning paths led to correct answers, suggesting the reasoning steps drive the prediction rather than serving as post-hoc justifications.
Grounding Accuracy: P-Composer significantly outperforms B-Composer in visual grounding precision.
- Overall IoU: Improved by +2.7 points (57.9% vs. 55.2%).
- Strict IoU@0.95: Improved by +9.0 points (15.4% vs. 6.4%).
Capability Breakdown: P-Composer achieved the highest accuracy in spatial positioning (84.8%) and spatial relationships (79.2%), while performance was lower on attribute color (66.5%), likely due to the open-ended nature of color attribution compared to constrained spatial choices.
Answer Accuracy: Composer achieved performance parity with its coordinate-based counterpart in final answer accuracy, demonstrating that the new grounding mechanism does not compromise task performance.

5. Significance and Claims

The paper claims that discrete learned proxy-tokens capture spatial semantics more effectively than typical textual coordinates. By establishing an explicit, learnable link between linguistic reasoning and visual features, Composer promotes faithful interpretability.

The authors assert that this approach represents a promising path toward trustworthy and reliable MLLMs. The work demonstrates that it is possible to maintain high answer accuracy while significantly improving the fidelity of visual grounding, thereby addressing the "semantic-spatial gap" that currently plagues grounded visual reasoning systems. The study emphasizes that rigorous evaluation must go beyond final answer accuracy to diagnose whether models are "right for the right reasons."