Technical Summary: Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space

Problem Statement

Embedding models are foundational to modern AI systems, enabling semantic search and retrieval-augmented generation (RAG). While recent advances in Large Language Models (LLMs) and Vision Language Models (VLMs) have accelerated the development of embedding models, a critical gap remains regarding Multimodal Diffusion Language Models (dLLMs).

Although dLLMs (e.g., LLaDA, Dream) have emerged as competitive alternatives to autoregressive models—offering advantages like bidirectional attention and parallel generation for global context modeling—it is currently unknown whether Multimodal Diffusion VLMs can serve as effective multimodal embedding models. Specifically, it is unclear how their embeddings compare to those of state-of-the-art autoregressive VLMs across diverse tasks such as classification, visual question answering (VQA), and information retrieval.

Methodology

The authors present the first systematic study converting Multimodal dLLMs into embedding models and comparing them against autoregressive counterparts.

Models Evaluated

The study selects four state-of-the-art models representing two paradigms:

• Autoregressive VLMs: LLaVA-1.6 and Qwen2.5-VL. These use causal attention and next-token prediction objectives.
• Diffusion VLMs: LaViDa and MMaDA. These utilize bidirectional attention and token unmasking (denoising) objectives.
  • LaViDa: Focuses on multimodal understanding using a diffusion language backbone (Dream-7B) and a vision encoder (SigLIP).
  • MMaDA: Unifies multimodal understanding and text-to-image generation using a unified diffusion architecture (LLaDA-8B + MAGVIT-v2 tokenizer).

Experimental Setup

Following the VLM2Vec framework, the authors apply contrastive fine-tuning to these pretrained models.

• Tasks: Evaluation spans three meta-tasks:
  1. Classification: Mapping image/instruction inputs to class labels (e.g., ImageNet-1K, N24News).
  2. Visual Question Answering (VQA): Mapping image/instruction/question inputs to answers (e.g., DocVQA, ScienceQA).
  3. Information Retrieval: Ranking candidates based on cross-modal relevance (e.g., MSCOCO, VisualNews, CIRR).
• Training Strategy: Models are fine-tuned using an InfoNCE loss with in-batch and hard negatives.
  • Autoregressive Embeddings: Derived from the last-token representation.
  • Diffusion Embeddings: Derived via mean pooling of contextualized token representations to leverage bidirectional context.
• Data Scale: Fine-tuning is conducted on subsets of datasets (capped at 20k samples per task) to assess data efficiency, covering a total of 32 datasets (19 in-domain, 13 out-of-domain).

Key Results

1. General Performance Gap

Contrary to the hypothesis that bidirectional attention would yield superior embeddings, Diffusion VLMs generally underperform Autoregressive VLMs across all three meta-tasks.

• LaViDa: Shows a modest performance gap compared to autoregressive baselines, with average drops of 3.5 points on classification, 2.5 points on VQA, and 4.4 points on retrieval.
• MMaDA: Exhibits a substantial performance degradation, lagging by over 20 points across all tasks compared to autoregressive models.

2. Domain Generalization

While Diffusion VLMs generally underperform in-domain, LaViDa demonstrates stronger out-of-domain (OOD) generalization.

• On classification tasks, LaViDa underperforms LLaVA-1.6 by 9.0 points in-domain but surpasses it by 2.0 points on OOD tasks.
• Similar trends are observed in VQA and retrieval, suggesting LaViDa may be more resilient to domain shifts despite lower absolute performance.

3. Task-Specific Strengths

Different models exhibit distinct inductive biases:

• LLaVA-1.6: Excels in tasks requiring world knowledge and commonsense reasoning (e.g., OK-VQA).
• Qwen2.5-VL: Performs best on structured document and chart understanding (e.g., DocVQA, ChartQA).
• LaViDa: Achieves top performance on compositional reasoning tasks involving objects and relations (e.g., GQA).

4. Cross-Modal Alignment Analysis

The primary cause of the performance gap is identified as insufficient image-text alignment in diffusion-based models.

• t-SNE Visualization: On the MSCOCO image-to-text retrieval task, LaViDa shows significant separation between paired image and text embeddings even after extensive fine-tuning (166k samples). In contrast, LLaVA-1.6 achieves tight alignment early in training.
• Retrieval Disparity: LaViDa performs relatively well on image-to-image retrieval (e.g., CIRR) but struggles significantly on cross-modal retrieval (e.g., MSCOCO-i2t), confirming weaker cross-modal alignment.

5. Data Efficiency and Architecture

• Data Efficiency: Autoregressive models (specifically LLaVA-1.6) demonstrate higher data efficiency, recovering most of their performance with smaller datasets compared to diffusion models.
• Vision Encoder Impact: An ablation study on MMaDA reveals that unfreezing the vision encoder during fine-tuning yields negligible performance gains. This suggests the bottleneck lies in the language backbone's alignment capabilities rather than the visual encoder's parameters.

Significance and Contributions

The paper makes three primary contributions to the multimodal embedding community:

1. Introduction of a New Class: It defines and evaluates Diffusion VLM embedding models, a new category derived from Multimodal Large Diffusion Language Models.
2. Systematic Comparative Analysis: It provides the first comprehensive benchmark comparing Diffusion and Autoregressive VLM embeddings across 32 datasets and three meta-tasks.
3. Empirical Insights: It reveals that despite the theoretical advantages of bidirectional attention, Diffusion VLM embeddings currently underperform autoregressive counterparts. The study attributes this to weaker multimodal understanding and less effective cross-modal alignment, particularly in models optimized for generative tasks (like MMaDA).

The authors conclude that while Diffusion VLMs show promise in specific areas like OOD generalization and compositional reasoning, their current alignment limitations hinder their effectiveness as general-purpose multimodal embedding models. The findings suggest that future research should focus on improving cross-modal alignment within diffusion architectures rather than solely relying on architectural shifts.