ZAYA1-VL-8B Technical Report

Based on the technical report provided, here is a detailed summary of the ZAYA1-VL-8B paper.

Problem Statement

Vision-Language Models (VLMs) have evolved rapidly, yet several challenges persist before they can be considered fully mature. These include model hallucinations, high computational costs for high-resolution inputs, and difficulties in efficient deployment. A specific architectural limitation identified in the paper is that standard VLMs often route vision and language tokens through identical model parameters, forcing shared representations to serve two modalities with fundamentally different statistical properties. This can create interference, particularly in Mixture-of-Experts (MoE) architectures where expert routing is learned primarily from language data. Additionally, the standard use of causal attention masks for image tokens is suboptimal because image patches lack inherent left-to-right ordering, arbitrarily preventing earlier patches from attending to later ones and limiting the capture of global visual structure. Furthermore, the ecosystem relies heavily on proprietary models or synthetic data distilled from them, limiting reproducibility and visibility into the ingredients driving performance.

Methodology

The authors present ZAYA1-VL-8B, a compact VLM built upon their in-house MoE language model, ZAYA1-8B-A1B. The model comprises 9.2B total parameters with 1.4B active parameters (including the vision encoder). The architecture integrates four main components: a decoder-only LLM, a vision encoder (based on Qwen2.5-VL), an image preprocessor, and a vision adapter.

The paper details two key architectural innovations designed to address the limitations of parameter sharing and causal masking:

1. Vision-Specific LoRA Adapters: Instead of adding new experts to the MoE backbone (which increases model size and data requirements), the authors integrate Low-Rank Adaptation (LoRA) adapters into the linear weights of the attention and MLP modules within the LLM. These adapters are activated only when vision tokens are passed into the LLM. They are trained alongside the standard weights during the main training phase, starting from zero with a weight decay term to prevent divergence from text pathways. This increases modality-specific capacity without substantially growing the model size.
2. Bidirectional Attention over Image Tokens: The authors remove the causal mask for vision tokens within the LLM attention layers. This allows full bidirectional attention among all visual tokens, restoring spatial symmetry and aligning the attention pattern of visual tokens inside the LLM with that of the vision encoder. Text tokens retain standard causal masking.

Training Pipeline:
The training follows a multi-stage curriculum:

• Alignment: Training only the vision adapter on low-resolution image captioning data (LLaVA-ReCap-558K) with the LLM frozen.
• Pretraining: Joint training of all parameters on 30 million multimodal samples. This stage introduces a new chat template and progressively increases the image resolution cap from 0.8MP to 6.3MP.
• Embedding Expansion: Expanding the LLM embedding layer to accommodate special tokens for grounding tasks (bounding boxes and points) while keeping the rest of the model fixed.
• Instruction Tuning: End-to-end training on 20 million curated text and multimodal samples, introducing grounding data in the form of bounding boxes and pointing.

The training pipeline utilizes a fully streaming data pipeline with efficient sequence packing and carefully designed attention masking (including conversation masking) to handle heterogeneous sequence lengths and maximize GPU utilization.

Key Contributions

1. Demonstration of MoE Viability: The paper demonstrates that their custom MoE LLM (ZAYA1-8B-A1B) is capable of powering a strong VLM, showing that the benefits of MoEs in language seamlessly transfer to VLM tasks.
2. Novel Architectural Mechanisms: The introduction of vision-specific LoRA adapters and bidirectional attention over image tokens as lightweight mechanisms to address modality interference and the limitations of causal masking, offering a new approach for extending MoE LLMs to VLMs.
3. Efficiency and Performance: ZAYA1-VL-8B demonstrates strong performance per inference FLOP and sample efficiency in terms of training tokens.
4. Open Source Release: The model, training checkpoints, and inference code are made publicly available to the research community.

Results

ZAYA1-VL-8B was evaluated across a diverse suite of benchmarks covering document understanding, perception, reasoning, counting, and grounding.

• General Benchmarks: The model achieves performance competitive with leading base models such as Molmo2-4B and InternVL3.5-4B. It surpasses models including Qwen2.5-VL-3B, PLM-3B, and MolmoE-8B across image understanding, reasoning, and counting benchmarks (e.g., AI2D, ChartQA, DocVQA, CountBenchQA).
• Grounding Benchmarks: On Point-Bench, ZAYA1-VL-8B achieves an average score of 58.0, matching MolmoE-8B-A1B and outperforming smaller dense models like Qwen3-VL-2B. On RefCOCO, it achieves an overall average of 84.3, placing it competitively among models with significantly more active parameters.
• Ablation Studies: Experiments confirmed that bidirectional attention for image tokens improves document and image understanding tasks, while vision-specific LoRA adapters yield clear performance improvements over baselines without adapters.

Significance

The paper claims that ZAYA1-VL-8B demonstrates that a well-designed MoE backbone paired with a data-efficient pipeline can power a highly capable VLM at a fraction of both the active compute and the data budget of comparable models. By open-sourcing the model, the authors aim to provide a practical foundation for the research community, offering full visibility into the ingredients that drive strong performance. The work highlights that careful data curation and targeted architectural innovations can yield robust general-purpose visual understanding without the need for massive parameter counts or proprietary data distillation. Future directions mentioned include incorporating multimodal RoPE directly into the LLM backbone and scaling up the training corpus for video understanding.