Technical Summary: ERNIE 5.0

Problem Statement

Current large-scale foundation models often rely on late-fusion architectures, where a pre-trained language backbone is augmented with modality-specific decoders or generators. While effective for individual modalities, these designs decouple multimodal generation from understanding, relying on non-autoregressive objectives that hinder deep cross-modal integration. Furthermore, existing approaches often struggle with the "ability seesaw" problem, where optimizing for one modality degrades performance in others. Additionally, the deployment of trillion-parameter models faces significant challenges regarding computational costs, memory constraints, and the need for flexible inference configurations without retraining or post-hoc compression. Finally, scaling reinforcement learning (RL) to unified multimodal models introduces instability due to sparse rewards, entropy collapse, and training-inference discrepancies, particularly within ultra-sparse Mixture-of-Experts (MoE) architectures.

Methodology

1. Unified Autoregressive Architecture

ERNIE 5.0 is a natively autoregressive foundation model designed from scratch to integrate text, image, video, and audio under a single framework.

• Unified Objective: All modalities are trained under a unified Next-Group-of-Tokens Prediction objective. Text generation uses Next-Token Prediction (NTP) augmented with Multi-Token Prediction (MTP). Vision and audio generation are formulated as group-of-tokens prediction tasks (Next-Frame-and-Scale Prediction for vision; Next-Codec Prediction for audio) to align with the autoregressive paradigm.
• Ultra-Sparse MoE with Modality-Agnostic Routing: The backbone utilizes an ultra-sparse MoE architecture (activation rate < 3%). Crucially, the expert routing is modality-agnostic, meaning routing decisions are conditioned on unified token representations rather than explicit modality identifiers. This allows tokens from different modalities to be dispatched to a shared pool of experts, promoting cross-modal knowledge generalization without heuristic modality-specific allocation.
• Visual Modeling:
  • Tokenization: Uses a causal 3D convolutional tokenizer (inflated from a 2D image tokenizer) supporting Next-Frame-and-Scale Prediction (NFSP).
  • Understanding: Employs a Dual-Path Hybrid Representation that fuses CNN (perceptual) and ViT (semantic) features via an Attention-Based Patch Merger before quantization, preserving fine-grained details lost in standard compression.
  • Generation: Uses NFSP to predict tokens across spatial scales and temporal frames. A cascaded diffusion refiner is employed post-backbone to enhance high-resolution details without conflicting with the autoregressive loss.
• Audio Modeling:
  • Tokenization: Utilizes a Residual Vector Quantization (RVQ) tokenizer distilled from Whisper, separating high-level semantics (first token) from residual acoustic details.
  • Generation: Introduces Next-Codec Prediction (NCP), a depth-wise autoregressive architecture where hierarchical audio codes are predicted across transformer layers, feeding back embeddings to condition subsequent levels.

2. Elastic Training Paradigm

To address deployment flexibility, ERNIE 5.0 introduces a novel Elastic Training strategy during pre-training. Instead of training a static model and compressing it later, the model learns a family of sub-models simultaneously within a single run.

• Elastic Depth: Randomly varies the number of active transformer layers (75% full depth, 25% reduced depth).
• Elastic Width: Randomly varies the total number of experts in MoE layers (80% full width, 20% reduced width).
• Elastic Sparsity: Randomly varies the routing top-k (number of activated experts per token).
  This allows the extraction of sub-networks with varying capacity-efficiency trade-offs that maintain functional integrity without retraining.

3. Post-Training and RL Optimization

The post-training pipeline combines Supervised Fine-Tuning (SFT) with Unified Multimodal Reinforcement Learning (UMRL). To stabilize RL on ultra-sparse MoE models, three key techniques are employed:

• Unbiased Replay Buffer (U-RB): Extends existing methods (like APRIL) by enforcing a data-ordering constraint. It prevents long-tail queries from stalling batches and ensures an unbiased data distribution, avoiding the non-stationary difficulty shifts that hinder convergence.
• Mitigated Entropy Collapse: Addresses rapid entropy collapse in early RL stages using Multi-granularity Importance Sampling Clipping (MISC) to correct training-inference mismatches and Well-learned Positive Sample Mask (WPSM) to prevent over-optimization on easy queries.
• Adaptive Hint-based RL (AHRL): Introduces partial "think skeletons" (hints) for hard queries with sparse rewards, gradually reducing hint availability as the model improves to bridge the gap between exploration and task completion.

4. Infrastructure

The training infrastructure utilizes Hybrid Parallelism (Tensor, Pipeline, Expert, Data, and Context parallelism) with fine-grained memory control (FP8 mixed precision, dynamic activation offloading) to handle trillion-parameter scales. A Disaggregation Architecture separates tokenizers from the MoE backbone to optimize parallelization for heterogeneous workloads. FlashMask is used to accelerate flexible attention patterns across modalities.

Key Contributions

1. First Trillion-Parameter Unified Autoregressive Model: ERNIE 5.0 represents the first production-scale realization of a unified model supporting both multimodal understanding and generation for text, image, video, and audio within a single autoregressive framework.
2. Modality-Agnostic Expert Routing: Demonstrates that a single shared routing mechanism can effectively specialize experts for different modalities and tasks without explicit modality identifiers, eliminating the need for heuristic modality-specific designs.
3. Elastic Training: Proposes a "Once-For-All" pre-training paradigm that produces a family of deployable sub-models (varying depth, width, and sparsity) from a single checkpoint, significantly reducing the computational cost of training multiple model sizes.
4. Stable Multimodal RL: Introduces a suite of techniques (U-RB, MISC, WPSM, AHRL) that guarantee efficient and stable post-training for unified foundation models under ultra-sparse MoE architectures.

Results

Extensive evaluations on text, vision, and audio benchmarks demonstrate that ERNIE 5.0 achieves strong and balanced performance:

• Language: Matches or outperforms specialized baselines (e.g., DeepSeek V3.2, Kimi K2) on knowledge, reasoning, coding, and multilingual tasks. Post-training significantly enhances instruction following and agent capabilities.
• Vision: Achieves competitive results on visual reasoning, document understanding, and VQA benchmarks. In generation, it produces high-aesthetic images and videos with superior semantic alignment (surpassing Veo3 on VBench-Semantic) compared to specialized models.
• Audio: Delivers state-of-the-art or competitive performance in automatic speech recognition (ASR), voice chatting, and general audio understanding across diverse linguistic and acoustic environments.
• Elasticity: Reducing routing sparsity to 25% yields over 15% decoding speedup with minor accuracy loss. A fully elastic variant using only 53.7% of activated parameters and 35.8% of total parameters retains competitive performance (average score 75.17 vs. 75.55 for the full model).

Significance and Claims

The paper claims that ERNIE 5.0 establishes a scalable and efficient foundation for next-generation unified multimodal models. By unifying understanding and generation under a single autoregressive objective with modality-agnostic routing, it overcomes the architectural fragmentation of late-fusion approaches. The introduction of elastic training offers a principled alternative to post-hoc compression, enabling flexible deployment across diverse hardware constraints without sacrificing performance. Furthermore, the successful stabilization of RL for trillion-parameter multimodal models provides a critical pathway for enhancing reasoning and alignment in future foundation models. The authors emphasize that these results validate the feasibility of native multimodal integration and elastic pre-training as a viable strategy for scaling foundation models.