Technical Summary: Representation Forcing for Bottleneck-Free Unified Multimodal Models

1. Problem Statement

Unified Multimodal Models (UMMs) aim to integrate perception (text output) and generation (pixel output) within a single transformer backbone. However, current state-of-the-art UMMs rely on a structural bottleneck: they utilize a separately pretrained, frozen Variational Autoencoder (VAE) for image generation. In this paradigm, images are compressed into latent spaces before diffusion and recovered via a fixed decoder.

This reliance creates two primary issues:

1. Optimization Mismatch: The latent space is optimized for reconstruction rather than the specific objectives of the unified model.
2. Quality Ceiling: The lossy compression of the VAE imposes a hard upper bound on generation quality that further training of the UMM cannot overcome.

A naive alternative is to generate directly in pixel space. While feasible for standalone models, applying this to UMMs results in a significant quality gap. Without the structural guidance provided by a latent space, the model struggles to simultaneously learn high-level semantic structure (object identity, layout) and low-level details (textures) from raw pixels.

2. Methodology: Representation Forcing (RF)

The paper proposes Representation Forcing (RF), a technique that eliminates the need for external VAEs by making representation prediction a native capability of the model. The core philosophy is to ground high-level visual representations in the decoder itself, using them as intermediate tokens to guide pixel generation.

2.1 Core Mechanism

RF operates by forcing the decoder to autoregressively predict visual representations before generating pixels. These representations act as an explicit structural scaffold between text prompts and pixel outputs.

• Source of Representations: Instead of an external latent space, RF utilizes features from the model's own understanding encoder (which is jointly trained). These features capture high-level structure such as object identity and spatial layout.
• Discretization: To make these features predictable under a next-token prediction objective (compatible with language modeling), the encoder's continuous features are discretized into a sequence of visual representation tokens via online vector quantization.
  • An Exponential Moving Average (EMA) copy of the encoder provides stable targets.
  • Features are quantized against a learnable codebook of prototypes.
  • This process yields one token per spatial patch, preserving the image's layout.

2.2 Architecture and Training

The model follows a Mixture-of-Transformers (MoT) architecture (based on BAGEL), where all tokens share self-attention layers but are routed to modality-specific feed-forward experts.

• Unified Sequence: The model processes a unified sequence: [Text Tokens] → [Representation Tokens] → [Pixel Patches].
• Attention Patterns:
  • Text and Representation tokens follow causal (autoregressive) attention.
  • Pixel patches follow bidirectional attention among themselves but attend causally to preceding text and representation tokens.
• Training Objective: The model is trained end-to-end with a combined loss:
  $$\mathcal{L} = \mathcal{L}_{LM} + \mathcal{L}_{Rep} + \mathcal{L}_{FM}$$
  • $\mathcal{L}_{LM}$: Cross-entropy for text next-token prediction.
  • $\mathcal{L}_{Rep}$: Cross-entropy for representation token prediction (predicting the discretized encoder features).
  • $\mathcal{L}_{FM}$: Flow-matching loss for pixel generation (using $x$-prediction with velocity loss).
• Inference: The process is two-staged. First, the decoder autoregressively predicts the full sequence of representation tokens from the text prompt. Second, conditioned on both the text and the predicted representation tokens, the decoder performs iterative denoising to synthesize the image directly in pixel space.

3. Key Contributions

1. Proposal of Representation Forcing (RF): A method that closes the quality gap in pixel-space UMMs by training the decoder to predict visual representations as intermediate tokens. This eliminates the need for any pretrained VAE.
2. Dual Benefit: RF improves both image generation and understanding. It enables pixel-space models to match VAE-based generation quality while outperforming them in understanding tasks.
3. End-to-End Unification: The work advocates for UMMs where perception and generation share a single, end-to-end learned representation space, rather than coordinating separate pretrained components.

4. Experimental Results

The authors conducted controlled experiments comparing four variants: Pixel-space (w/o RF), Pixel-space (w/ RF), VAE-based (w/o RF), and VAE-based (w/ RF).

4.1 Image Generation

• Performance: The pixel-space model with RF (RF-Pixel) achieves a GenEval overall score of 0.84 (0.88 with LLM rewriter), matching state-of-the-art VAE-based unified models like BAGEL and Show-o2.
• Quality Gap Closure: Without RF, the pixel-space model scores only 0.25 on GenEval, producing images with distorted objects and incoherent compositions. With RF, the score jumps to 0.76, demonstrating that RF provides the necessary structural guidance.
• Comparison: RF-Pixel matches the VAE-based baseline across standard benchmarks (GenEval, DPG-Bench) while preserving rich textural details.

4.2 Image Understanding

• Performance: Pixel-space models with RF (Pixel+RF) generally outperform their VAE-based counterparts (VAE+RF).
• Specifics: Pixel+RF improves performance on 6 out of 8 understanding benchmarks (including MMMU, MME, and BLINK), whereas VAE+RF improves on 5. Notably, Pixel+RF shows substantial gains on general visual understanding tasks (e.g., +4.3 on MMMU), suggesting that removing the VAE bottleneck allows for tighter integration of perception and generation.

4.3 Ablation Studies

• Prediction vs. Alignment: RF (decoder prediction) significantly outperforms REPA (auxiliary feature alignment), achieving 0.76 vs. 0.43 on GenEval. Direct conditioning via self-attention is more effective than implicit feature alignment.
• Discrete vs. Continuous: Discrete token prediction (0.76) vastly outperforms continuous regression (0.26). Discretization prevents error accumulation in autoregressive prediction and enforces the separation of high-level structure from low-level details.
• Encoder Choice: Using DINOv3 (self-supervised) as the understanding encoder yields better results than SigLIP2, likely because DINOv3's features contain richer spatial and structural information.

5. Significance and Claims

The paper claims that Representation Forcing represents an effective step toward fully end-to-end, bottleneck-free Unified Multimodal Models.

• Native Capability: By turning representations from perception outputs into generation targets, RF grounds understanding and generation in a single representation space without relying on external latent spaces.
• Feasibility of Pixel-Space UMMs: The results demonstrate that pixel-space generation is not only feasible but potentially superior to VAE-based generation for unified modeling, provided that structural guidance is supplied via representation forcing.
• Future Direction: The authors view this as a move toward models where all multimodal capabilities are acquired directly from raw inputs within a single model, rather than inherited from independently trained components.

Note: The authors acknowledge limitations, including the use of a pretrained LLM backbone rather than training from scratch, and the current focus on still images rather than video or temporal modalities.