Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) face significant challenges in unifying perception (understanding) and generation (creation) within a single architecture. Key issues include:

• Representational Disparities: Difficulty in effectively fusing visual, auditory, and linguistic modalities due to mismatched data structures.
• Modality Imbalance: Models often excel in one modality (e.g., text) but struggle to transfer fine-grained knowledge (e.g., spatial relationships, object attributes) from understanding tasks to high-precision generation tasks.
• Efficiency vs. Capacity: Scaling models to handle long-context, multi-turn, and multi-modal interactions often leads to prohibitive computational costs.
• Generation Artifacts: Discrete token-based approaches in audio and image generation often suffer from quantization artifacts, loss of fidelity, and poor controllability (e.g., identity preservation, text rendering).

2. Methodology

Ming-Flash-Omni is built upon Ling-Flash-2.0, a highly sparse Mixture-of-Experts (MoE) architecture. The model features 100 billion total parameters, with only 6.1 billion active per token, striking a balance between massive capacity and inference efficiency.

Core Architectural Innovations

• Unified Two-Stage Pipeline: The model follows a perception-to-generation pipeline but introduces synergistic training to align objectives.
• Sparse MoE Backbone: Utilizes a dual balancing scheme to stabilize training and improve efficiency while supporting long-context modeling (up to 24K–64K tokens).
• Time-Interleaved VideoRoPE: An enhanced positional encoding that interleaves timestamps with video frames, allowing precise temporal alignment for visual tokens and better capture of video dynamics.

Multimodal Specific Enhancements

A. Unified Speech (Understanding & Generation)

• Continuous Acoustic Latents: Replaces discrete speech tokens with continuous acoustic latents (via a VAE-based tokenizer) to eliminate quantization artifacts and improve naturalness.
• Context-Aware ASR: Leverages surrounding linguistic context to resolve homophones and dialects, significantly improving transcription accuracy in noisy or ambiguous scenarios.
• Joint Generation: Enables single-channel generation of speech, sound effects, and music with fine-grained vocal control (timbre, emotion, style) without requiring complex disentanglement modules.
• Low-Frame-Rate Tokenizer: Operates at 12.5 Hz to compress sequence lengths, mitigating error propagation in autoregressive generation for long audio.

B. Unified Vision (Understanding & Generation)

• Single-Stream Diffusion Transformer: Unifies text, images, and other modalities into a single token space with full cross-modal attention. This replaces dual-stream architectures, reducing "copy-paste" artifacts and ensuring global semantic coherence.
• Generative Segmentation: Reframes image segmentation as a generative editing task (e.g., "color the banana purple" instead of just "segment the banana"). This tightly couples perception and generation, using edit quality as direct supervision for visual understanding.
• Identity & Text Preservation:
  • ID Preservation: Concatenates VAE-encoded reference features with noisy latents and uses learnable query tokens to maintain structural consistency in unedited regions.
  • High-Fidelity Text: Integrates a specialized Glyph-byT5 encoder to render text with pixel-level control, ensuring accurate in-image text generation.

Training Strategy

• Two-Stage Pipeline:
  1. Perception: Progressive pre-training (8K → 24K context), followed by two-stage instruction tuning and Reinforcement Learning (RL) using Group Relative Policy Optimization (GRPO).
  2. Generation: Freezes the perception MLLM and optimizes the image generator (Flow Matching + RL) and audio generator.
• Reinforcement Learning: Uses a multi-reward framework (realism, instruction adherence, aesthetics) to refine controllability without overfitting.
• Infrastructure: Employs Sequence Packing for data heterogeneity and Flexible Parallelization (independent DP/PP/TP configurations) for model heterogeneity, achieving a 58–69% speedup over standard setups and a 28% Model FLOPs Utilization (MFU).

3. Key Contributions

1. Sparse Unified Architecture: A 100B parameter MoE model that achieves AGI-like multimodal intelligence with high computational efficiency.
2. Generative Segmentation Paradigm: A novel approach that bridges the gap between understanding and generation by treating segmentation as a semantic editing task, enabling pixel-level control.
3. Continuous Audio Generation: A shift from discrete tokens to continuous latents for speech and sound, enabling joint generation of speech, music, and SFX with superior fidelity and dialect support.
4. Advanced Controllability: Significant improvements in identity preservation for image editing and high-fidelity in-image text rendering.
5. Context-Aware & Dialect-Aware ASR: State-of-the-art performance in recognizing speech within complex contexts and across diverse Chinese dialects.

4. Results

Ming-Flash-Omni was evaluated on over 50 public and in-house benchmarks, demonstrating competitive or superior performance against leading models (e.g., Gemini 2.5 Pro, Qwen3-Omni, GLM-4.6-V).

• Vision-to-Text:
  • Achieves 74.9 on MMStar and 85.6 on MMVet, outperforming Qwen3-Omni and matching Gemini 2.5 Pro.
  • Sets a new high on OCRBench (891) and leads open models on Wiki Knowledge (64.6).
  • State-of-the-art on MVBench (77.5) for video understanding.
• Text-to-Image Generation:
  • Scores 86.98 on DPG-Bench, surpassing SD3-Medium (84.08) and approaching Qwen-Image (88.32).
  • Achieves 0.99 on GenEval, demonstrating exceptional controllability in position and color.
• Image Editing & Segmentation:
  • Outperforms specialized editing models (e.g., Z-Image-Edit) in perceptual quality and semantic consistency.
  • Achieves 72.1 on RefCOCO, significantly outperforming other unified MLLMs like Nano-banana (15.7).
• Audio:
  • ContextASR: Achieves the best average score (3.30) on ContextASR benchmarks, outperforming Qwen3-ASR and Kimi-Audio.
  • Dialect Recognition: Sets new SOTA for Sichuanese dialect TTS (84.42% accuracy) and podcast generation (CER 2.12).
• Streaming Dialogue:
  • Scores 75.92 on the proprietary StreamingMultiturnBench, outperforming Gemini 2.5 Pro (74.99) in accuracy and relevance.

5. Significance

Ming-Flash-Omni represents a significant leap toward Artificial General Intelligence (AGI) by demonstrating that a single, unified model can effectively handle complex, multi-turn interactions across vision, speech, and text.

• Practical Foundation: It proves that a unified architecture can serve as a practical foundation for general-purpose multimodal intelligence, moving beyond specialized silos.
• Efficiency: The sparse MoE design allows for scaling model capacity without linearly increasing inference costs, making advanced multimodal AI more accessible.
• Real-World Applicability: The model's robustness in dialect recognition, context-aware ASR, and high-fidelity image editing makes it highly suitable for real-world deployment in diverse, multilingual, and creative scenarios.
• Open Science: The authors plan to open-source the models and code, fostering further research and development in the multimodal AI community.