Technical Summary: Gemma 4 Technical Report

Problem Statement

The rapid evolution of large language models has created a critical demand for open-weight models that possess strong multimodal understanding, advanced reasoning capabilities, and high computational efficiency. Existing solutions often struggle to balance performance on complex tasks with the memory and compute constraints required for on-device deployment. Furthermore, handling long contexts typically leads to memory explosions in KV caches, and traditional multimodal architectures often rely on heavy, separate encoders that fragment memory and increase latency.

Methodology

Gemma 4 introduces a new generation of open-weight, natively multimodal language models ranging from 2.3B to 31B parameters, including both dense and Mixture-of-Experts (MoE) architectures. The technical approach focuses on several key architectural and methodological innovations:

1. Architectural Innovations

• Thinking Mode: Inspired by prior work (OpenAI, 2024), Gemma 4 models incorporate a "thinking mode" that generates a reasoning trace before producing a final response. This is designed to enhance performance in reasoning-heavy domains such as mathematics and coding.
• Encoder-Free Architecture: A unified, encoder-free architecture is introduced for the 12B model. Instead of using separate, frozen vision and audio encoders, this model projects raw 40ms audio chunks and image patches directly into the LLM embedding space via lightweight projection modules. This alleviates the need for separate encoders and reduces memory fragmentation.
• Long-Context Efficiency: To address memory explosions in KV caches, the models utilize a specific ratio of local sliding window to global self-attention (5:1 for most models, 4:1 for the 2.3B model). The architecture employs $p$-RoPE (Barbero et al., 2025) for positional encoding on global layers and standard RoPE on local layers. Additionally, it reuses keys as values in global layers and implements KV cache sharing (Shazeer, 2019; Kayyam et al., 2026), reducing the global KV cache footprint by up to 37.5%.
• Multi-Token Prediction (MTP): An autoregressive MTP drafter head (Li et, al., 2024) is integrated to facilitate speculative decoding (Leviathan et al., 2023), improving inference speed. The drafter uses a separate embedder and cross-attends to the main model's KVs, eliminating the need for MTP prefill.

2. Training and Quantization

• Pre-training: Models are trained on a diverse dataset including web documents, code, images, and audio, with a cutoff date of January 2025. The tokenizer (Gemma Team, 2025) features a 262k vocabulary with split digits and byte-level encodings.
• Quantization-Aware Training (QAT): Quantized versions of the models and encoders are provided using mobile quantization (mix of int2/int4 weights with int8 activations) and Q4_0 blockwise quantization. This significantly reduces memory footprints and latency with minimal quality impact.
• Infrastructure: Training leverages TPUv5p and TPUv6e clusters with ZeRO-3 optimizer sharding and Slice-Granularity Elasticity to handle localized failures efficiently.

3. Modality Handling

• Vision: Smaller models (E2B, E4B) use a 150M Vision Transformer (ViT) encoder, while larger models use a 550M encoder. The unified 12B model replaces the encoder with a single large matrix multiplication layer and 2D coordinate-based positional embeddings.
• Audio: E2B and E4B models utilize a 305M audio encoder based on the Universal Speech Model (Zhang et al., 2023), processing 40ms chunks. The unified 12B model discards the encoder entirely, projecting raw 16kHz audio vectors directly into the embedding space.

Key Contributions

• Performance Leap: Gemma 4 establishes a significant performance improvement across STEM, multimodal, and long-context benchmarks compared to its predecessor, Gemma 3.
• Efficiency: The suite offers improved inference speed, memory efficiency, and compute efficiency through MTP, QAT, and optimized attention mechanisms.
• Scalability: The models are designed to serve a variety of on-device hardware, with the 2.3B model matching the performance of the 27B Gemma 3 model while using 10x fewer parameters.
• Open Access: All models are released under an Apache 2.0 license, empowering developers to customize and extend capabilities.

Results

• Human Evaluation: On the Arena (Chiang et al., 2024) leaderboard, Gemma 4 31B ranks as the top open dense model (Elo 1451), rivaling much larger open MoE models (e.g., GLM 5.1, MiMo V2.5 Pro) and approaching the performance of leading closed models.
• Reasoning and STEM: In "thinking mode," Gemma 4 31B achieves 89.2 on AIME 2026 (no tools) and 85.2 on MMLU Pro, significantly outperforming Gemma 3 27B (88.3 and 82.6 respectively).
• Multimodal: Gemma 4 models demonstrate superior performance on vision benchmarks (MMMU Pro, MATH-Vision) and audio tasks (CoVoST, FLEURS ASR). Notably, the 12B encoder-free model achieves competitive audio-text performance without a dedicated encoder.
• Long-Context: The models show substantial gains in long-context retrieval and reasoning, with the 31B model achieving 96.8% accuracy on RULER at 32k context and 96.4% at 128k, outperforming Gemma 3 27B (91.1% and 66.0% respectively).
• Safety: Rigorous evaluations indicate major improvements in content safety across all categories compared to previous Gemma models, with minimal policy violations and low rates of unjustified refusals.

Significance

The paper claims that Gemma 4 represents a "leap in performance" for the Gemma family, establishing a scalable foundation for edge deployment and advanced reasoning. By integrating thinking modes, unified encoder-free architectures, and long-context optimizations, Gemma 4 rivals larger, frontier open models in human-rated tasks while maintaining the efficiency required for diverse hardware environments. The release of these models under an open license aims to democratize access to state-of-the-art multimodal and reasoning capabilities, fostering further research and application development in the AI ecosystem.