Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

1. Problem Statement

Current Unified Multimodal Models (UMMs) typically rely on pretrained vision encoders (e.g., CLIP for understanding, VQ-VAE for generation) to convert raw images into latent representations. This modular approach introduces several critical limitations:

• Representation Misalignment: Separate encoders for understanding and generation create a disconnect between the two tasks, preventing fully end-to-end optimization.
• Inductive Biases: Pretrained encoders impose fixed input resolutions and limit access to fine-grained, low-level visual details, hindering performance on tasks requiring precise perception.
• Architecture Complexity: The reliance on multiple modular components (encoders, connectors, decoders) complicates the training pipeline and prevents the model from learning a truly unified representation directly from raw pixels.

The authors ask: Can we build a unified multimodal model that performs both understanding and generation directly from raw pixels, completely discarding pretrained vision encoders?

2. Methodology

The paper introduces Tuna-2, a native unified multimodal model that operates entirely in pixel space. The methodology is developed through a progressive simplification of existing architectures:

A. Architectural Evolution

The authors propose a three-stage evolution to reach the final encoder-free design:

1. Tuna (Baseline): Uses a standard modular design with a VAE encoder for generation and a representation encoder (e.g., SigLIP) for understanding.
2. Tuna-R (Intermediate): Removes the VAE but retains a pretrained representation encoder (SigLIP 2). It uses a patch embedding layer to convert images into tokens for the LLM decoder, performing pixel-space generation via flow matching.
3. Tuna-2 (Final): Completely removes the pretrained representation encoder. It replaces complex encoders with simple patch embedding layers that directly tokenize raw image pixels. The entire system is a single, monolithic Transformer decoder processing text and visual tokens jointly.

B. Pixel-Space Generation

Instead of latent diffusion, Tuna-2 employs rectified flow matching directly in pixel space:

• Objective: The model predicts the clean image $x_1$ from a noisy sample $x_t$ constructed via a linear schedule: $x_t = tx_1 + (1-t)x_0$.
• Loss Function: It minimizes the velocity prediction error ($v_\theta$), where $v_\theta = \frac{x_\theta - x_t}{1-t}$.
• Inference: Uses an Euler solver to denoise images step-by-step.

C. Masking-Based Feature Learning

To address the difficulty of learning robust representations in high-dimensional pixel space (which is prone to superficial shortcuts), the authors introduce a masking scheme:

• Mechanism: During training, random image patches are masked and replaced with a learnable mask token.
• Dual Role:
  • For Generation: Forces the model to reconstruct clean patches from partially observed noisy inputs, creating a harder denoising task.
  • For Understanding: Acts as a regularizer, forcing the model to perform reasoning under partial visual observation, preventing over-reliance on specific visual cues.

D. Training Pipeline

The model is trained fully end-to-end in two stages:

1. Stage 1 (Pretraining): Joint training on image captioning and text-to-image generation (70% understanding, 30% generation data) using a 7B parameter backbone (Qwen2.5-7B-Instruct).
2. Stage 2 (SFT): Supervised fine-tuning on image instruction-following, editing, and high-quality generation datasets.

3. Key Contributions

1. Tuna-2 Architecture: The first native UMM to achieve state-of-the-art (SOTA) performance in both understanding and generation by completely discarding pretrained vision encoders, relying solely on pixel embeddings.
2. Empirical Validation of Encoder-Free Design: Demonstrates that while pretrained encoders help early convergence, an encoder-free monolithic architecture surpasses them in multimodal understanding, particularly on fine-grained tasks, after sufficient pretraining.
3. Pixel-Space Scalability: Proves that pixel-space flow matching can be scaled to large unified models, competing with latent-space approaches for high-fidelity image generation and editing.
4. Masking Strategy: Introduces a masking-based learning scheme that stabilizes training in high-dimensional pixel space and enhances representation robustness.

4. Experimental Results

The authors evaluated Tuna-2 against various baselines (including Tuna, Tuna-R, Show-o2, BAGEL, and generation-only models like FLUX.1).

• Multimodal Understanding (Table 1):
  • Tuna-2 achieves SOTA performance among 7B-scale native UMMs across general benchmarks (GQA, MMVet, MMMU) and pixel-centric benchmarks (V*, CountBench, VisuLogic).
  • It significantly outperforms Tuna-R on fine-grained perception tasks, suggesting that removing the encoder's inductive bias allows for better learning of low-level visual details.
• Image Generation (Table 2 & 3):
  • Tuna-2 is competitive with SOTA unified models on GenEval and DPG-Bench.
  • While Tuna-R (with encoder) slightly leads in generation metrics initially, Tuna-2 catches up after SFT.
  • Diversity: Tuna-2 generates significantly more diverse images than Tuna-R and Tuna, as judged by LLMs (GPT-5.4, Claude Opus).
• Image Editing & Reconstruction (Table 4 & 5):
  • Tuna-2 performs strongly on image editing (ImgEdit), outperforming many unified baselines.
  • In image reconstruction tasks, Tuna-2 achieves reconstruction quality (PSNR/SSIM) comparable to specialized VAEs like FLUX.1, proving pixel-space representations can faithfully encode visual information.
• Attention Analysis (Figure 7):
  • Visualizations show Tuna-2 has more accurate and stable attention maps, correctly focusing on relevant visual regions even when linguistic priors are misleading (e.g., identifying a glass cup in a football context), whereas encoder-based models often rely on text priors or salient distractors.

5. Significance and Conclusion

The paper fundamentally challenges the prevailing paradigm that pretrained vision encoders are necessary for high-performance multimodal AI.

• Paradigm Shift: It demonstrates that "end-to-end pixel-space learning" is a viable and superior path for developing stronger visual representations, especially for tasks requiring fine-grained perception.
• Simplicity: By stripping away complex modular encoders, Tuna-2 simplifies the architecture to a single Transformer, reducing engineering overhead and enabling true joint optimization.
• Future Direction: The results suggest that future native UMMs should prioritize direct pixel modeling over latent-space compression, as the latter may introduce information bottlenecks that limit the model's ability to perceive and generate high-fidelity visual details.

In summary, Tuna-2 proves that a simple, encoder-free architecture trained directly on raw pixels can outperform complex, encoder-based systems in understanding while maintaining competitive generation capabilities, offering a scalable blueprint for the next generation of multimodal models.