💬 NLP

v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning

The paper introduces v1, a lightweight extension for multimodal language models that enables active visual referencing through a semantic point-and-copy mechanism, allowing the model to dynamically retrieve and re-ground relevant image patches during reasoning to overcome the limitations of static visual encoding.

Original authors: Jiwan Chung, Junhyeok Kim, Siyeol Kim, Jaeyoung Lee, Min Soo Kim, Youngjae Yu

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiwan Chung, Junhyeok Kim, Siyeol Kim, Jaeyoung Lee, Min Soo Kim, Youngjae Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle, but the pieces are hidden inside a giant, complex picture. In the world of artificial intelligence, there are special "multimodal" brains that can look at pictures and read text at the same time. Usually, when these AI models try to solve a math problem involving a diagram, they take one quick glance at the image, memorize the general vibe, and then try to solve the rest of the problem using only words. It's like looking at a map for five seconds, closing your eyes, and then trying to drive to a new city without ever looking back at the map. The problem is, as the driving instructions get longer and more complicated, the AI starts to forget exactly where the turns are or what the landmarks look like. It loses its "grounding," meaning it drifts away from the visual evidence it needs to be right.

This paper introduces a new way for these AI brains to think, called v1. Instead of just taking one look and then guessing, v1 teaches the model to actively point at the specific parts of the picture it needs to remember, copy that information, and paste it back into its thinking process whenever it gets stuck. It's like having a super-smart student who, while solving a long math problem, keeps reaching out to tap the diagram with a finger, saying, "Wait, let me check this angle again," and then bringing that specific visual detail right back into their notes to help them finish the calculation. The researchers found that this simple "point-and-copy" trick helps the AI stay focused on the right parts of the image, leading to much better answers on difficult visual math tests.

The Problem: The "One-Glance" Trap

Most current AI models that can see and read are like students who are told to study a diagram for a split second and then close their eyes. They encode the image into their memory once and then try to reason through the problem using only text. The researchers noticed a flaw in this approach: as the reasoning gets longer and more complex, the model starts to lose focus. It's as if the student's eyes are glazing over; they stop paying attention to the important details in the picture. The paper shows that as the AI writes out more steps to solve a problem, its attention to the actual image fades away, and it starts making mistakes because it's no longer "looking" at the evidence it needs.

The Solution: Point-and-Copy

To fix this, the team created v1, a lightweight upgrade for these AI models. Think of v1 as giving the AI a magical pointer and a photocopier. When the AI is thinking through a problem and realizes it needs to check a specific part of the image—like an angle in a triangle or a bar on a graph—it doesn't just guess. Instead, it uses a "pointing" mechanism to select that exact patch of the image.

Once it points, it "copies" the visual information from that spot and pastes it directly back into its stream of thought. This means the AI can revisit the visual evidence as many times as it needs, keeping the picture fresh in its mind while it does the math. Crucially, this isn't about generating new images or drawing new pictures; it's about re-accessing the original image data with surgical precision. The model learns to say, "I need to look at the red bar again," points to it, copies its data, and then continues its reasoning with that fresh visual context.

How They Taught the AI

You can't just tell an AI to "look harder"; it needs to be trained on how to do it. The researchers built a massive dataset called v1g, containing 300,000 examples of math problems where the reasoning steps are explicitly linked to specific parts of the image. They used a clever, automated process to create this data:

They took existing reasoning paths from other AI models.
They used a powerful language model to break those paths down, identifying where the AI should have looked at the image.
They added "grounding" annotations, which are like digital sticky notes that say, "When you mention this angle, you must look at this specific region of the image."

This dataset taught the v1 model that when it's stuck or needs to verify a step, it should reach out, point to the right spot, and copy the information.

The Results: Smarter, Focused Reasoning

The team tested v1 on several challenging math benchmarks, including MathVista, MathVision, and MathVerse. These tests are full of diagrams, charts, and geometric shapes that require careful visual inspection.

The results were impressive. Even though v1 is a relatively small model (7 billion parameters), it outperformed many larger models and other specialized reasoning models.

On the MathVision benchmark, the base model without the pointing feature scored 23.6.
When the pointing feature was added during training but not used during the test, the score rose slightly to 25.3.
But when the model was allowed to use the point-and-copy feature during the test, the score jumped to 34.5.

This shows that the ability to actively revisit the image is the key driver of success. The model didn't just get better at math; it got better at looking while doing math. In one example, while other models misidentified the tallest bar in a chart, v1 correctly pointed to the specific bar, copied its data, and calculated the right percentage. In another task involving a pathfinding puzzle with hexagons, v1 successfully pointed to the correct shapes to verify the route, whereas others got lost.

What This Means

The paper suggests that the future of multimodal reasoning isn't just about making models bigger or smarter at text generation. It's about giving them the ability to dynamically access visual information when they need it. By letting the model "point and copy," the researchers found a way to keep the visual evidence aligned with the reasoning steps, preventing the model from drifting away from the truth.

The researchers are careful to note that this isn't a magic wand that solves every problem. Sometimes the model still points to the wrong area, or it might point too many times, or it might struggle with very abstract concepts that don't fit neatly into a box. However, the core finding is clear: active visual referencing helps. It turns the AI from a student who glances at the map once and hopes for the best, into a navigator who constantly checks the map, ensuring every turn is correct.

Technical Summary: v1 - Learning to Point Visual Tokens for Multimodal Grounded Reasoning

Problem Statement
Current Multimodal Large Language Models (MLLMs) typically encode an image once at the beginning of inference, storing the visual information in a frozen key-value cache. Subsequent reasoning occurs purely in the text space. The authors identify a critical limitation in this architecture: visual grounding decay. As reasoning chains lengthen, models progressively lose focus on relevant visual regions. Empirical analysis (Section 3) demonstrates that as generation steps increase, the cumulative attention to all image tokens declines, and the relative attention to semantically important regions diminishes. This makes it difficult for models to re-ground intermediate reasoning steps in the original visual evidence, leading to errors in complex tasks like mathematical problem solving where localized cues (e.g., specific angles or graph bars) must be revisited.

Methodology
The paper introduces v1, a lightweight extension to autoregressive MLLMs designed to enable active visual referencing during inference. The core mechanism is point-and-copy:

Architecture: v1 augments the standard vocabulary with "pointer tokens." At each decoding step, the model computes two distributions simultaneously:
- A generation distribution over the standard vocabulary ( $P_{gen}$ ).
- A pointing distribution over the continuous input image patch embeddings ( $P_{ptr}$ ).
- The output logits are a concatenation of these two spaces. If the model selects a pointer index, it does not generate a new text token; instead, it copies the embedding of the corresponding image patch and injects it as the next input token.
Mechanism: This allows the model to dynamically retrieve and reuse specific visual evidence. Unlike methods that generate new visual tokens (requiring heavy pretraining), v1 reuses existing input embeddings. The pointing mechanism uses a learned linear projection to attend to image patch positions, effectively allowing the model to "point" to a region and "copy" its representation back into the reasoning stream.
Training Data (v1g): To train this behavior, the authors constructed v1g, a dataset of 300,000 multimodal mathematical reasoning traces. The dataset creation pipeline involves:
- Oversampling reasoning traces from a pretrained MLLM (QvQ).
- Using an LLM (Gemini-2.0-flash) to decompose text-only reasoning into discrete visual queries and retrieval steps.
- Grounding these references to specific image regions using a relative attention mechanism within Qwen2.5-VL to generate bounding boxes for abstract or symbolic elements (e.g., angles, chart bars).

Key Contributions

v1 Model: A lightweight MLLM extension that mitigates visual grounding decay through a point-and-copy mechanism, enabling dynamic visual reference without modifying the core architecture or requiring external tools.
v1g Dataset: A large-scale training set containing 300K reasoning traces with fine-grained, interleaved visual grounding annotations.
Empirical Findings: Extensive experiments demonstrating that dynamic visual access at inference time significantly improves multimodal mathematical reasoning capabilities.

Results
The authors evaluated v1 on three established benchmarks: MathVista, MathVision, and MathVerse.

Performance: The 7B-scale v1 model consistently outperformed comparable baselines, including general-purpose MLLMs (Qwen2-VL, InternVL2.5) and reasoning-specialized models (LLaVA-CoT, TVC, Mulberry).
Comparison: Notably, v1-7B narrowed the performance gap with much larger 72B-scale models, particularly on MathVision, which demands precise visual grounding.
Ablation: Removing the pointing capability during inference reduced performance significantly (from 34.5 to 25.3 on MathVision-mini), confirming that the gains are driven by the active re-access mechanism rather than just the training data.
Efficiency: The method incurs minimal computational overhead. While copied tokens increase sequence length, the wall-clock time overhead is modest (~7% compared to text-only baselines), and the model adds only two lightweight linear heads.

Significance and Claims
The paper claims that v1 demonstrates the efficacy of dynamic visual access as a core component of multimodal reasoning. By allowing models to actively revisit input images via a simple point-and-copy mechanism, the approach addresses the inherent limitation of "one-shot" visual encoding. The authors emphasize that this method provides a lightweight, interpretable alternative to generating new visual tokens or relying on external grounding tools. The results suggest that enabling models to explicitly re-ground intermediate reasoning steps in visual evidence leads to more accurate and robust performance in tasks requiring long-horizon, multi-step visual reasoning. The authors release the code, model, and data to facilitate further exploration of this direction.