Zamba2-VL Technical Report

Problem Statement

Vision-language models (VLMs) have become the standard interface for multimodal reasoning, typically relying on a Transformer-based Large Language Model (LLM) backbone. However, the self-attention mechanism in Transformers incurs quadratic computational costs during the prefill phase and requires a Key-Value (KV) cache that grows linearly with sequence length. This becomes a severe bottleneck for multimodal models, where a single high-resolution image or video clip can generate thousands to hundreds of thousands of tokens. While input-side mitigations like visual token compression exist, they do not alter the fundamental scaling behavior of the backbone.

State-space models (SSMs), specifically Mamba and Mamba2, offer linear-time sequence processing and a constant-size recurrent state, theoretically solving these efficiency constraints. However, prior SSM-based VLMs have struggled with tasks requiring precise in-context lookup (e.g., visual grounding and fine-grained retrieval), where attention mechanisms remain superior. Furthermore, existing strong SSM-based VLMs are often distilled from Transformer models or built on smaller SSM backbones, leaving open the question of whether a hybrid SSM-Transformer architecture can be competitive when trained from scratch at scale.

Methodology

The authors present Zamba2-VL, a family of VLMs built on the Zamba2 hybrid architecture. Zamba2 interleaves a Mamba2 state-space backbone with a small number of shared Transformer blocks. The Transformer blocks are augmented with LoRA projectors to provide in-context retrieval capabilities, while the Mamba2 layers handle the bulk of computation with linear time complexity and a fixed recurrent state.

Architecture:

• Backbone: Zamba2 (1.2B, 2.7B, and 7B parameter variants).
• Vision Encoder: The Vision Transformer (ViT) from Qwen2.5-VL, chosen for its use of 2D Rotary Position Embeddings (RoPE) and native dynamic-resolution processing, which preserve fine-grained spatial structure.
• Connector: A standard two-layer MLP adapter that projects visual features into the language embedding space. It pools 2x2 windows of patch embeddings, reducing vision tokens by a factor of four.
• Positional Embeddings: The model retains the specific positional embedding schemes of the base Zamba2 models (1D RoPE for 1.2B/7B, no rotary embedding for 2.7B) rather than adopting a multimodal RoPE variant, to avoid excessive compute requirements.

Training Pipeline:
The training follows a three-stage curriculum:

1. Alignment: The MLP adapter is trained on low-resolution image-caption data (LLaVA-ReCap-558K) while the backbone and vision encoder remain frozen.
2. Pretraining: The full model is trained on a heterogeneous mixture of ~30B tokens. This stage deliberately upsamples document understanding and OCR data to compensate for the Zamba2 backbone's initial lag in text-heavy tasks compared to pure Transformers.
3. Instruction Tuning: The model is fine-tuned on ~20B tokens of curated instruction-following data, emphasizing multi-turn conversation, grounding (pointing and bounding boxes), and reasoning.

Key Contributions

1. First Open Hybrid SSM-Transformer VLM: Zamba2-VL is presented as the first open family of VLMs built on a strong hybrid SSM-Transformer backbone, demonstrating that such architectures are viable and competitive substrates for multimodal modeling at scales from 1.2B to 7B.
2. Architectural and Data Design: The paper details the specific architectural choices (connector design, vision-token integration) and data composition strategies (specifically the up-sampling of document/OCR data) required to extend a hybrid backbone into a capable VLM.
3. Efficiency-Accuracy Trade-off: The work characterizes the performance-efficiency profile of these models, showing they achieve competitive accuracy against leading Transformer-based VLMs while inheriting the inference speed advantages of the Zamba2 backbone.

Results

The authors evaluated Zamba2-VL (1.2B, 2.7B, 7B) against leading open-weight Transformer-based VLMs (Molmo2, Qwen3-VL, InternVL3.5, PerceptionLM) across image understanding, reasoning, OCR, grounding, and counting benchmarks.

• Accuracy: Zamba2-VL is broadly competitive with Transformer baselines of comparable scale.
  • Counting: The models show exceptional performance in visual counting. Zamba2-VL-1.2B scored 62.5 on PixMoCount, nearly double the score of InternVL3.5-1B (32.8). The 7B model achieved 85.3, placing it at the top of its band.
  • Document/OCR: The models are competitive on DocVQA, ChartQA, and TextVQA, a result attributed to the deliberate up-sampling of document data during pretraining.
  • Reasoning: The models lag slightly behind the strongest Transformer baselines on heavy reasoning benchmarks (e.g., MMMU, MathVista), a profile the authors attribute to a training pipeline focused on capability acquisition rather than RL-style reasoning specialization.
• Efficiency: The most significant finding is the inference speed. Zamba2-VL delivers roughly an order-of-magnitude lower Time-to-First-Token (TTFT) compared to Transformer baselines at matched parameter scales. This efficiency gap is most pronounced at the 1.2B and 2.7B scales, which are critical for on-device and edge deployment.
• Ablation Studies:
  • Vision Encoder: The Qwen2.5-VL ViT significantly outperformed CLIP and SigLIP-2, largely due to its 2D RoPE and dynamic resolution handling.
  • Backbone: When compared to Llama-3.x references under matched training conditions, the hybrid Zamba2 backbones matched or exceeded Transformers on most image-understanding tasks but lagged on text-dense document tasks (which were later mitigated by data up-sampling). The performance gap between Zamba2 and Llama narrowed significantly as scale increased, with the 7B model being nearly indistinguishable from the Llama-3.1-8B reference.

Significance

The paper claims that Zamba2-VL demonstrates that hybrid SSM-Transformer LLMs are a practical alternative to dense Transformers for VLM backbones, particularly in latency- and memory-constrained settings. The results suggest that the "hybrid recipe"—using sub-quadratic recurrence for the bulk of computation and a small amount of attention for recall—can match the capability of dense Transformers at scale while retaining substantial inference efficiency advantages. The authors release the models and code to provide a foundation for researchers and practitioners to build upon, specifically targeting scenarios where serving costs and latency are binding constraints.