💻 computer science

GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning

GeoWeaver is a pre-reasoning framework that enhances spatio-temporal reasoning in vision-language models by adaptively allocating fine-grained geometric evidence to individual visual tokens, thereby establishing geometry as a fundamental representational prerequisite rather than a late-fusion auxiliary signal.

Original authors: Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Deshui Miao, Xingsen Huang, Yameng Gu, Xin Li, Haijun Zhang, Ming-Hsuan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blurry Glasses" Issue

Imagine you are trying to give someone directions in a busy city. You have a map (the visual data) and a voice (the language model).

Current AI models are great at recognizing what things are (e.g., "That's a red car," "That's a tree"). However, they often struggle with where things are relative to each other (e.g., "The car is 5 feet to the left of the tree," or "If I walk forward, will I hit the wall?").

The paper argues that current AI tries to solve this by adding geometry (spatial math) as an afterthought. It's like giving the AI a pair of glasses that only show the names of objects, and then asking it to do complex math about distances while it's trying to speak. The AI gets confused because it has to figure out the shape of the room and the sentence structure at the same time.

The Solution: GeoWeaver (The "Architect's Blueprint" Approach)

The authors propose a new method called GeoWeaver. Instead of adding geometry at the end, they build it into the foundation before the AI starts thinking or speaking.

Think of it like this:

Old Way: You hand a builder a pile of bricks (visual tokens) and a list of instructions. The builder has to figure out the structural integrity of the wall while trying to write the instructions for the next room.
GeoWeaver Way: Before the builder even touches the bricks, you give them a 3D blueprint specifically tailored to that exact spot. The bricks are now "grounded" in the blueprint. When the builder starts writing instructions, the structure is already solid and clear.

How It Works: The "Smart Librarian" System

The paper describes a specific process to make this happen, which they call Token-Adaptive Geometric Grounding. Here is the analogy:

The Multi-Level Library (The Geometry Bank):
The AI has access to a "library" of geometric information. This library isn't just one book; it's a collection of books ranging from "close-up details" (like the texture of a wall) to "wide-angle views" (like the layout of the whole room). This library is built by a frozen, pre-trained geometry expert (called VGGT) that the main AI doesn't need to relearn.
The Smart Librarian (The Router):
In the old days, the AI would read every single book in the library for every single word it wanted to say. This is slow and confusing.
GeoWeaver uses a Smart Librarian. When the AI looks at a specific part of an image (a "visual token"), the Librarian asks: "What kind of geometry do you need right now?"
- If the AI is looking at a sharp corner, the Librarian hands it the "close-up detail" book.
- If the AI is looking at the whole room layout, the Librarian hands it the "wide-angle" book.
The Tailored Mix (Sparse Routing):
The Librarian doesn't just give one book; it gives a tiny, perfect mix of 2 or 3 specific books that are most relevant to that specific spot. It ignores the rest. This prevents the AI from getting overwhelmed by irrelevant information.
The Result (Grounded Tokens):
The visual "bricks" (tokens) are now infused with this specific geometric evidence. They are no longer just "a picture of a fridge"; they are "a fridge located 3 feet to the right of the sink."

Why This Matters (The Results)

The paper tested this on several "spatial reasoning" tests (like estimating distances, counting objects, or planning a route).

The Analogy: Imagine a student taking a math test.
- Old AI: Tries to memorize the formula while solving the problem.
- GeoWeaver: Has the formula written clearly on the desk before the test starts.
The Outcome: GeoWeaver consistently beat other top AI models on these spatial tests. It got better at guessing distances, understanding directions (left/right/back), and planning routes.
The Bonus: Crucially, it didn't lose its ability to do other things. It didn't forget how to answer general questions or understand videos. It just got better at the "spatial" part without breaking the rest of its brain.

The Core Takeaway

The paper claims that geometry shouldn't be an extra signal added later; it should be the foundation.

Just as you can't build a stable house without a solid foundation, you can't have reliable spatial reasoning without visual tokens that are "grounded" in geometric reality before the language model starts its reasoning process. GeoWeaver provides that foundation by letting every part of the image pick its own perfect geometric evidence from a library of options.

Technical Summary: GeoWeaver

Problem Statement

Current Vision-Language Models (VLMs) struggle with reliable spatio-temporal reasoning, particularly in tasks involving relative distance estimation, object layout understanding, and tracking spatial relations in dynamic scenes. While recent approaches attempt to enhance spatial intelligence by incorporating geometric information through structural branches, 3D-aware supervision, or late-stage fusion, they typically treat geometric cues as a shared signal across all visual tokens.

The authors identify a critical limitation in existing paradigms: different visual tokens require different geometric evidence depending on their specific spatial roles (e.g., local boundaries vs. global layout). Current methods often fail to address this heterogeneity, treating geometry as an auxiliary signal injected late in the pipeline (either pre-fusion or within the LLM reasoning stack). This forces the decoder to jointly resolve visual-geometric alignment and high-level relational reasoning, entangling perceptual grounding with cognitive reasoning and reducing reliability.

Methodology: GeoWeaver

GeoWeaver proposes a pre-reasoning geometric grounding framework that treats geometry as a representational prerequisite rather than a late-fusion signal. The core philosophy is to "ground-then-reason," shaping visual representations before they enter the language model.

1. Architecture Overview

The framework consists of three primary stages:

Semantic Visual Branch: Utilizes a pretrained multimodal LLM (specifically Qwen3.5 in the implementation) with a native vision encoder to extract semantic visual tokens ( $V$ ).
Frozen Geometry Branch: Employs a frozen geometry encoder (VGGT) to extract explicit geometric priors from the same input frames. Instead of using only the final representation, GeoWeaver constructs a Multi-Level Geometry Bank by collecting hidden states from deeper layers (e.g., layers 12–23 of a 24-layer VGGT). This bank serves as a reservoir of structural evidence ranging from local details to global abstractions.
Token-Adaptive Geometric Grounding: This is the core innovation. Before language decoding, visual tokens query the geometry bank to retrieve relevant evidence.

2. Token-Adaptive Evidence Allocation

The grounding process involves four specific steps to ensure each visual token receives the most relevant geometric abstraction:

Normalization & Alignment: Geometry features from different layers are normalized and spatially merged (2×2) to match the resolution and hidden dimension of the visual tokens.
Query-Conditioned Routing: A lightweight router predicts preference weights over the selected geometry layers for each individual visual token ( $v_i$ ). This acknowledges that different spatial regions require different levels of geometric abstraction.
Compact Evidence Selection: To avoid diluting token-specific grounding with redundant cues, the model employs sparse routing. It selects only the top- $K$ (default $K=2$ ) most relevant geometry layers for each token, masking the rest.
Residual Grounding: The selected geometric evidence is aggregated and injected into the visual tokens via a residual connection ( $v'_i = v_i + W_o g_i$ ). The projection matrix $W_o$ is initialized to zero, allowing the model to start as an identity mapping and gradually learn to reshape representations during fine-tuning.

3. Training Objective

The model is trained using standard next-token prediction (autoregressive loss) on downstream spatio-temporal reasoning tasks. The geometry encoder remains frozen, and the grounding module is jointly optimized with the base multimodal model. No auxiliary geometric reconstruction or contrastive losses are introduced; the model learns to allocate and exploit geometric evidence purely through end-to-end supervision.

Key Contributions

Representational Diagnosis: The authors identify the lack of geometry-grounded visual tokens as a key bottleneck in current VLMs, arguing that spatio-temporal reasoning depends on whether tokens are geometrically grounded before entering the language model.
GeoWeaver Framework: A novel framework that allocates token-specific geometric evidence from a multi-level geometry bank prior to language decoding. This shifts visual-geometric alignment from the reasoning stack to the perceptual interface.
Token-Adaptive Mechanism: The introduction of a sparse, query-conditioned routing mechanism that allows different visual tokens to retrieve distinct geometric abstractions, avoiding uniform exposure to all geometry layers.
Empirical Validation: Extensive experiments demonstrating that pre-reasoning grounding yields superior performance without catastrophic forgetting of general multimodal capabilities.

Experimental Results

GeoWeaver was evaluated on multiple spatial reasoning benchmarks, including VSI-Bench, ReVSI, SPAR-Bench, ViewSpatial, BLINK, 3DSRBench, and EmbSpatial.

VSI-Bench: GeoWeaver-5B achieved an average score of 72.8, outperforming the strongest open-source baseline (SpatialStack-5B) by 5.3 points. GeoWeaver-10B reached 75.05. Significant gains were observed in geometry-sensitive categories like object counting, size estimation, relative direction, and route planning.
ReVSI: In frame-aware settings, GeoWeaver-10B achieved 57.5% accuracy with 32 frames, outperforming prior spatial VLMs. The results suggest that effective geometric grounding is more critical than simply increasing the number of input frames.
SPAR-Bench: GeoWeaver-5B achieved a new state-of-the-art score of 69.2, a massive improvement of +26.8 points over the previous best, particularly in medium- and high-level reasoning tasks (position matching, distance inference, spatial imagination).
General Capabilities: Evaluations on MMBench, Video-MME, and TempCompass showed that GeoWeaver maintains or slightly improves general multimodal performance compared to baselines, confirming that the geometric grounding does not cause catastrophic forgetting.

Significance and Claims

The paper claims that the primary benefit of GeoWeaver stems not from simply injecting more geometric data into the model, but from reshaping the visual representation before language reasoning begins. By converting raw multi-level geometric cues into geometry-grounded visual tokens, the framework allows the language model to operate on "geometry-ready" representations.

The authors argue that geometric information yields the greatest benefit not as a late-fusion auxiliary signal, but as a fundamental prerequisite that shapes the representational foundation. This approach successfully decouples low-level visual-geometric alignment from high-level relational reasoning, leading to more robust spatio-temporal intelligence while preserving the model's general multimodal competence. The work suggests that future progress in spatial intelligence requires addressing the representational bottleneck at the perceptual interface rather than solely optimizing the reasoning stack.