💻 computer science

Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

This paper introduces GAS, a training framework that enhances zero-inference-overhead visual understanding in Multimodal Large Language Models by using a decoupled generation branch with Next Embedding Prediction as auxiliary supervision to refine shared visual representations without compromising the final inference architecture.

Original authors: Zhongbin Guo, Jiahao Xie, Dongling Xiao, Qianle Wang, Ruiqi Lu, Xiaomin He, Wanxuan Sun, Cheng Yang

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhongbin Guo, Jiahao Xie, Dongling Xiao, Qianle Wang, Ruiqi Lu, Xiaomin He, Wanxuan Sun, Cheng Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant student how to understand a complex painting. Traditionally, you'd show them the art and ask them to write a description of it. They learn to talk about the painting, but they never actually have to draw it. This is how most modern "Multimodal Large Language Models" (MLLMs) work today: they are fantastic at describing images, answering questions about them, and even solving puzzles based on what they see. However, because they are only trained to speak about the image, they sometimes miss the tiny, precise details—like exactly where a specific object is, or how many items are hidden in a crowd. They get the "gist" but might fumble the fine print.

Now, imagine a different approach: what if, to help them understand better, you also asked them to recreate the painting? It sounds like a lot of extra work, and you might worry that trying to draw would distract them from learning how to describe. But what if you could use the act of drawing purely as a secret training exercise? You could force their brain to pay attention to every pixel and spatial relationship to get the drawing right, but then, once the training is done, you could throw away the drawing tools entirely. The student would be left with a super-powered ability to understand the image, having learned from the drawing practice, but without ever needing to draw again. This is the core idea behind a new method called GAS (Generation as Auxiliary Supervision).

The Problem: Talking vs. Seeing

Current AI models are like art critics who have never held a brush. They are trained to predict the next word in a sentence about an image. While this makes them great conversationalists, it leaves a gap in their vision. They don't have a direct "muscle memory" for visual details like exact shapes, boundaries, or spatial relationships. The paper argues that by only training them to talk, we are missing out on a massive reservoir of visual learning. Previous attempts to fix this by making models that both talk and draw (Unified Models) often failed because the "drawing" part was too heavy, slowing the model down, or because the way they drew (using different types of digital "pixels") didn't match the way they thought, causing confusion rather than improvement.

The Solution: The "Ghost" Drawing Class

The researchers behind GAS came up with a clever trick. They built a special training framework where the AI learns to "draw" (or more accurately, predict the next piece of an image's digital representation) alongside its normal job of understanding.

Here is how they did it, using a metaphor of a two-story house:

The Shared Foundation (The Trunk): Both the "Understanding" AI and the "Drawing" AI share the same lower floors. This is where the raw visual information is processed.
The Split (The MoT Architecture): As the information goes up the stairs, the house splits. One path leads to the "Understanding" room (where the AI answers questions), and a parallel path leads to the "Drawing" room.
The Secret Weapon (NEP): In the Drawing room, the AI isn't trying to create a pretty picture for a gallery. Instead, it uses a technique called Next Embedding Prediction (NEP). Think of this as the AI trying to guess the next "digital Lego block" that makes up the image, based on the instructions it received. It's not trying to be an artist; it's trying to be a precise architect of visual data.
The Magic Transfer: The act of trying to predict these precise visual blocks forces the shared foundation (the lower floors) to become incredibly detailed and sharp. It learns to hold onto visual information much better than before.
The Zero-Overhead Twist: This is the best part. Once the training is finished, the entire "Drawing" room is demolished. The AI keeps the super-sharp foundation it learned, but it no longer has the extra room or the tools to draw. When you ask it a question later, it answers just as fast as before, but with much better vision. The paper confirms this adds zero inference overhead—meaning it doesn't make the AI slower or heavier to use.

What They Found

The team tested this on models with 2 billion and 4 billion parameters. They didn't just guess; they ran extensive experiments to see if this "drawing practice" actually helped the "understanding" skills.

Better at the Details: The models trained with GAS got significantly better at tasks that require precise vision, like counting objects, figuring out spatial relationships (e.g., "is the lamp closer than the TV?"), and understanding complex diagrams. For example, on a counting test, the 2B model jumped from 87.7 to 90.1, and the 4B model hit 90.8.
It Works at Any Stage: They tried this on AI models that were brand new, models that had already been pre-trained, and models that were already fine-tuned. In every case, the "drawing practice" helped. It was most helpful for brand-new models, but even strong, already-smart models got a boost.
Not All Drawing is Equal: They discovered that the type of drawing task mattered. Simply asking the AI to "draw a cat" didn't help much. But asking it to draw based on complex instructions—like "segment this specific part of the image" or "edit this object based on a logical reason"—forced the AI to think deeply. These "cognitive" drawing tasks were the ones that really supercharged the understanding.
The "Ghost" Branch is Key: They proved that if they didn't separate the drawing path from the understanding path (using their special "Mixture-of-Transformers" design), the AI actually got worse at understanding. The drawing task would confuse the AI. The separation was crucial to let the visual learning "soak into" the foundation without messing up the thinking process.

Why It Matters

The paper suggests that we don't need to build massive, slow AI systems that can both talk and draw to get better vision. Instead, we can use the process of generation as a secret training tool. By forcing the AI to predict visual details in a specific, structured way during training, and then discarding that ability afterward, we get a model that sees the world with much sharper eyes but runs just as fast as before.

The researchers found that this approach works across different model sizes and training stages, suggesting it's a reliable way to boost visual intelligence. They also showed that the AI didn't lose its ability to reason with just text; in fact, it slightly improved there too. The paper concludes that generation-guided training is a practical, efficient route to stronger multimodal understanding, solving the problem of how to make AI "see" better without making it "slower."

Technical Summary: GAS (Generation as Auxiliary Supervision)

Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant success in visual understanding tasks (e.g., captioning, VQA) by encoding visual inputs into text tokens for autoregressive reasoning. However, this paradigm relies solely on text-only supervision (Next Token Prediction), which introduces two critical limitations:

Expressiveness Bottleneck: Natural language cannot precisely encode fine-grained visual details such as exact spatial relationships, pixel-level boundaries, or physical dynamics. Consequently, visual representations in deeper LLM layers often attenuate, losing geometric and structural information.
Underutilized Data: Existing architectures cannot effectively consume visually intensive data (e.g., segmentation masks, image editing) for training, forfeiting a vast reservoir of fine-grained supervision.

While Unified Multimodal Models (UMMs) attempt to jointly train understanding and generation, they face unresolved tensions:

Inference Overhead: Most UMMs retain generation parameters at inference time, incurring latency and memory costs.
Objective Conflict: Generation objectives in UMMs are often optimized for synthesis quality rather than comprehension, leading to performance plateaus or degradation in understanding tasks due to conflicting optimization pressures.
Lack of Mechanism: Existing work often reports aggregate gains without clarifying how generation benefits understanding or which intermediate representations are most amenable to such supervision.

Methodology: GAS Framework

The authors propose GAS (Generation as Auxiliary Supervision), a training framework that reinterprets visual generation not as a deployment goal, but as auxiliary supervision to enrich visual representations for understanding. The framework consists of three core components:

1. Next Embedding Prediction (NEP)

Instead of using discrete visual tokenization or diffusion objectives (which create disjoint representation spaces), GAS formulates generation as autoregressive prediction of continuous visual embeddings within the exact same representation space consumed by the LLM backbone.

Target: The model predicts target image embeddings ( $z_{tgt}$ ) derived from a frozen Vision Transformer (ViT) and a visual-language projector.
Loss: The generation loss is computed as the cosine distance between predicted and ground-truth embeddings in the continuous space, prioritizing structural representation enrichment over photorealistic pixel synthesis.
Stabilization: An Exponential Moving Average (EMA) of the projector weights is used to generate stable target embeddings, preventing supervision drift during joint training.

2. Mixture-of-Transformers (MoT) Architecture

To resolve the conflict between generation and understanding objectives, GAS employs a decoupled architecture:

Shared Trunk: The lower layers of the model (up to a split layer $l_{split}$ ) are shared between the understanding and generation branches.
Parallel Branches:
- The Understanding Branch continues through upper layers, optimized solely by text cross-entropy loss.
- The Generation Branch consists of independent transformer layers initialized from the backbone, optimized solely by the NEP loss.
Mechanism: Gradients from the generation loss flow back through the shared trunk, refining the intermediate hidden states ( $H_{l_{split}}$ ) with fine-grained visual details. These enriched states are then fed into the upper understanding layers, which adapt to exploit these features without being directly perturbed by generation gradients.
Zero Inference Overhead: Crucially, the entire generation branch (parameters and vision head) is discarded after training. The deployed model is identical in architecture and cost to a standard understanding-only MLLM.

3. Systematic Data Construction

The authors constructed a dataset of approximately 10 million generation samples across five categories and 15 subtasks, generated via an automated, annotation-free pipeline:

Categories: Grounding, Segmentation, Image Editing, Visual Chain-of-Thought (Visual-CoT), and Text-to-Image (T2I).
Design Principle: Tasks are selected based on their latent cognitive dependency with understanding capabilities. For instance, Visual-CoT requires generating intermediate visual evidence to support reasoning, forcing the model to develop precise spatial and logical capabilities that text-only training misses.
Pipeline: Uses open-vocabulary tagging (RAM++), detection (Grounding DINO), and LLM-based instruction synthesis to create high-quality, difficulty-controlled data without manual annotation.

Key Contributions

Zero-Overhead Transfer: The combination of continuous NEP supervision and a removable MoT generation branch allows the transfer of generative learning to the deployed model with strictly zero inference overhead.
Task Correlation Hypothesis: The paper demonstrates that generation tasks with stronger latent correlations to understanding (e.g., grounding, segmentation) yield the largest gains. Combining diverse tasks provides complementary improvements, particularly on perception and counting benchmarks.
Systematic Analysis: Through extensive ablations and representation-level diagnostics, the paper clarifies when and why generation helps:
- It is beneficial across all training stages (from scratch, pretraining, and SFT).
- It requires specific architectural isolation (MoT) and objective design (NEP); simply adding generation data or using a shared backbone degrades performance.
- It enhances visual information retention in deep layers and sharpens attention mechanisms toward task-relevant regions.

Results

Evaluated on 2B and 4B parameter backbones (based on Qwen3-VL), GAS demonstrates consistent improvements over understanding-only baselines:

Spatial & Counting: Significant gains on CountBenchQA (e.g., +2.4 points on 2B scale) and CV-Bench, directly attributed to grounding and segmentation tasks.
Reasoning: Improvements on MathVista, DynaMath, and VisuLogic, linked to Visual-CoT and editing tasks.
Generalization: GAS outperforms or matches larger unified models (e.g., BAGEL 7B+7B, Emu3 8B) on multiple benchmarks while maintaining a smaller parameter count and zero inference cost.
Text Reasoning: The framework preserves and slightly improves pure-text reasoning capabilities (ZebraLogic, MMLU-Redux), indicating that generation supervision does not erode language reasoning.

Significance and Claims

The paper claims that generation-guided training is a practical route to stronger multimodal understanding without the deployment penalties of unified models. Its primary significance lies in:

Decoupling Training from Inference: Proving that generation can serve purely as a training-time regularizer to refine visual representations, rather than a required inference capability.
Mechanistic Insight: Providing evidence that the benefits arise from enriched spatial representations and improved information retention in deep layers, rather than just increased data volume.
Scalability: Demonstrating that automated, annotation-free data synthesis can effectively scale generation tasks to improve understanding, offering a blueprint for future MLLM development that leverages the vast potential of visual data without compromising inference efficiency.

The authors conclude that while generation and understanding objectives can conflict, a carefully designed auxiliary supervision framework (GAS) can harness the former to significantly enhance the latter, achieving state-of-the-art visual understanding with the efficiency of standard text-generation models.