💻 computer science

Semantic-Enriched Latent Visual Reasoning

This paper introduces Semantic-Enriched Latent Visual Reasoning (SLVR), a two-stage framework that enhances latent visual reasoning by enriching representations with fine-grained attribute semantics and aligning them with diverse queries via Multi-query Group Relative Policy Optimization, supported by the newly constructed SLV-Set dataset and SV-QA benchmark.

Original authors: Tianrun Xu, Yue Sun, Qixun Wang, Jingyi Lu, Yuan Wang, Tianren Zhang, Longteng Guo, Fengyun Rao, Jing Lyu, Feng Chen, Jing Liu

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianrun Xu, Yue Sun, Qixun Wang, Jingyi Lu, Yuan Wang, Tianren Zhang, Longteng Guo, Fengyun Rao, Jing Lyu, Feng Chen, Jing Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, but instead of looking at the whole picture, you have to describe the pieces to a friend who is sitting in a dark room.

The Problem: "Thinking with Images" vs. "Thinking in the Dark"
Current AI models that look at pictures and answer questions usually do one of two things:

The "Cut-and-Paste" Method: They physically crop the image to zoom in on a specific part (like cutting a photo out of a magazine) and then think about it. This is slow and clunky.
The "Visual Memory" Method: They compress the image into a tiny, invisible "thought bubble" (a latent representation) and reason inside that bubble. This is faster, but the paper argues these bubbles are too empty. They remember what the object looks like (colors, shapes) but forget what it means (is it a "running" dog or a "sleeping" dog?).

The Solution: SLVR (Semantic-Enriched Latent Visual Reasoning)
The authors propose a new system called SLVR. Think of SLVR as teaching the AI to build a much richer, more detailed "thought bubble" that contains not just the visual image, but also a detailed description of the object's personality and actions.

They do this in two stages, like training an actor for a play:

Stage 1: The "Character Sheet" Training
Imagine you are training an actor to play a specific character. Instead of just telling them, "You are a man in a red shirt," you give them a detailed Character Sheet.

The Paper's Approach: The AI is shown a region of an image (like a person holding a camera) and is forced to fill out a "Character Sheet" for that region. This sheet lists specific attributes: What color is the shirt? Is the person standing or sitting? What are they holding? Is there text on the object?
The Result: The AI learns to compress the image into a "thought bubble" that is packed with these specific details (semantics), not just a blurry picture.

Stage 2: The "Interview" Training
Now, imagine that same actor is being interviewed by two different journalists at the same time.

Journalist A asks: "What is this person doing?"
Journalist B asks: "What color is their jacket?"
The Paper's Approach: The AI is given the same "thought bubble" (the same region of the image) and asked to answer both questions. The system uses a special training technique (called M-GRPO) to ensure the "thought bubble" is consistent. It forces the AI to realize that the same mental image must be able to answer both questions correctly without changing its mind or getting confused.
The Result: The AI learns that its internal "thought bubble" is a stable, reliable source of truth that can handle different types of questions about the same thing.

The New Dataset: SLV-Set
To teach the AI this way, the authors built a massive new library of training data called SLV-Set.

It contains 400,000 detailed "Character Sheets" (attribute descriptions) for different parts of images.
It contains 800,000 pairs of questions where two different questions are asked about the exact same part of an image.
They also created a test called SV-QA to see if the AI can handle these "interviews" where the same image is questioned from different angles.

The Results
When they tested this new method:

The AI became much better at answering questions about specific parts of an image.
It was more consistent. If you asked it two different questions about the same object, it gave answers that made sense together (unlike older methods that might get confused).
It performed better than previous "fast" methods (latent reasoning) and was competitive with the slower "cut-and-paste" methods, but without the heavy computing cost.

In Summary
The paper claims that by forcing AI to learn detailed "attribute sheets" for image parts and then testing those parts with multiple different questions at once, the AI builds a smarter, more stable internal understanding of images. It's like upgrading from a blurry snapshot in your mind to a high-definition 3D model with a full biography attached to every object.

Technical Summary: Semantic-Enriched Latent Visual Reasoning (SLVR)

1. Problem Statement

Multimodal latent-space reasoning aims to replace computationally expensive explicit "thinking with images" (e.g., iterative cropping, tool invocation) by performing visual reasoning directly within a compact latent space. However, existing approaches face two primary limitations:

Lack of Semantic Richness: Current methods typically rely on visual supervision alone, causing latent representations to encode appearance-level cues while failing to explicitly preserve fine-grained semantic content (e.g., object properties, states, interactions) necessary for diverse reasoning tasks.
Single-Task Optimization: Latent training is often driven by a single query or task formulation. There is no unified mechanism to align how a single enriched latent representation should support diverse query types and reasoning granularities grounded in the same visual region.

Consequently, existing latents lack the structural semantic information required to robustly handle region-level reasoning tasks where multiple semantic perspectives must be addressed simultaneously.

2. Methodology

The authors propose Semantic-Enriched Latent Visual Reasoning (SLVR), a two-stage learning framework designed to enrich latent representations with attribute-level visual semantics and align them with diverse reasoning objectives.

2.1. Data Construction (SLV-Set)

To support the framework, the authors constructed SLV-Set, comprising two complementary components based on Visual-CoT:

Attribute-Level Semantic Dataset: Approximately 400K region-level attribute annotations. Using Qwen3-VL-235B, the system generates structured semantic profiles for key regions, covering appearance (color, shape, material), actions, interactions, spatial attributes, and text. These are encoded into 4096-dimensional semantic embeddings to serve as fine-grained supervision signals.
Multi-Query Dataset: Approximately 800K question-answer pairs where two semantically distinct questions are grounded in the same visual region, probing different semantic aspects.

Additionally, SV-QA is introduced as a benchmark (591 paired samples) to evaluate latent reasoning robustness under semantic variation, extending existing benchmarks (V*, HRBench-4K, HRBench-8K) by querying the same region from multiple perspectives.

2.2. Stage 1: Structured Latent Learning

In the first stage, the model learns region-centric latent representations that explicitly integrate visual evidence with attribute-level semantics.

Input Construction: An image and question are processed. A region of interest (ROI) is encoded into a visual latent segment ( $H_{vis}$ ). A special token <sem> is inserted to generate a single semantic latent ( $z_{sem}$ ).
Visual Alignment: The visual latent tokens are aligned with the vision encoder's features using a mean squared error (MSE) loss to preserve fine-grained visual details.
Semantic Alignment: The semantic latent ( $z_{sem}$ ) is projected and aligned with the pre-computed attribute-level semantic embedding ( $e$ ) using MSE loss. This forces the latent to capture structured attributes (e.g., "holding a camera," "green jacket").
Objective: $L_{stage1} = L_{vis} + L_{sem}$ .

2.3. Stage 2: Multi-query Group Relative Policy Optimization (M-GRPO)

The second stage refines the learned latents to ensure they support diverse reasoning objectives consistently.

Mechanism: Given multiple questions ( $q_1, q_2$ ) grounded in the same region, the model generates corresponding latent pairs. M-GRPO performs policy updates to optimize these latents jointly.
Reward Components:
1. Answer Correctness: Rewards correct answers for each query (evaluated by an external judge).
2. Latent Consistency: Penalizes divergence between the latents generated for different queries ( $q_1$ vs. $q_2$ ) grounded in the same region, ensuring the latent representation remains stable across semantic variations.
3. Stability Regularization: Anchors the optimized latents to the distribution learned in Stage 1 using a margin-based formulation to prevent excessive drift.
Optimization: The framework uses a Group Relative Policy Optimization (GRPO) objective, updating the policy over groups of region-grounded queries to encourage representations that are simultaneously effective for multiple semantic queries.

3. Key Contributions

SLVR Framework: A two-stage learning paradigm that enriches latent representations with fine-grained attribute-level visual semantics and aligns them with diverse reasoning demands via M-GRPO.
SLV-Set Dataset: A large-scale resource containing ~400K region-level attribute annotations and ~800K multi-query QA samples, providing the necessary supervision for semantic enrichment and latent alignment.
SV-QA Benchmark: A new evaluation benchmark designed to test latent reasoning robustness under semantic variation by querying the same visual region from multiple semantic perspectives.
Empirical Validation: Extensive experiments demonstrating that SLVR improves reasoning robustness and semantic consistency compared to existing latent reasoning baselines and "think-with-image" approaches.

4. Experimental Results

Experiments were conducted on standard VQA benchmarks (OKVQA, GQA, VizWiz, ChartQA, TextVQA, AI2D) and the proposed SV-QA benchmark.

Standard Benchmarks: SLVR-7B outperformed prior latent reasoning methods (e.g., LVR) and remained competitive with "think-with-image" baselines (e.g., GRIT, DeepEyes). Notable gains were observed on OKVQA (+11.2% over LVR) and VizWiz (+24.7%).
SV-QA Benchmark: SLVR achieved superior performance on the "Both Correct" metric (joint correctness of two distinct questions on the same region) across V*, HRBench-4K, and HRBench-8K. For instance, on HRBench-8K, SLVR improved the "Both Correct" score by +4.0% over LVR.
VisualPuzzles: SLVR achieved the best overall accuracy (34.2%) compared to LVR (29.4%), with significant improvements in Deductive (+5.0) and Spatial (+7.4) reasoning categories.
Ablation Studies: Results confirmed that Stage 1 semantic supervision and the M-GRPO consistency constraints are critical. Text-only baselines, even with multi-query training, lagged behind latent reasoning variants in maintaining consistent grounding across multiple semantic perspectives.

5. Significance and Claims

The paper claims that SLVR addresses the critical gap in current multimodal reasoning by moving beyond purely visual supervision to semantic-enriched latent representations. The authors argue that:

Explicit visual operations (cropping) are computationally inefficient, while purely visual latents lack semantic depth.
By explicitly supervising latents with attribute-level semantics and enforcing consistency across multiple queries, models can achieve more robust and semantically consistent reasoning without the overhead of explicit visual manipulation.
The proposed framework enables latent representations to flexibly support diverse tasks and reasoning granularities grounded in the same visual evidence, offering a scalable alternative to explicit "thinking with images."

The work is positioned as an advancement in the field of machine learning, specifically in enhancing the efficiency and semantic fidelity of multimodal latent reasoning.