Technical Summary: Nemotron 3 Nano Omni

1. Problem Statement

The field of multimodal AI is rapidly evolving, yet existing models face significant challenges in balancing efficiency, native audio support, and long-context reasoning. Previous iterations often treated audio as an afterthought or relied on heavy, dense architectures that suffer from high inference latency and throughput limitations. Furthermore, handling long-form video and documents with audio requires massive token counts, leading to prohibitive computational costs. There is a critical need for an omni-modal model (text, image, video, audio) that:

• Natively processes audio alongside other modalities without performance degradation.
• Maintains high reasoning capabilities across all modalities.
• Achieves superior inference efficiency (low latency, high throughput) suitable for real-world agentic applications.
• Supports ultra-long contexts (up to 256K tokens) for complex document and video analysis.

2. Methodology

2.1 Model Architecture

Nemotron 3 Nano Omni is built on a Mixture-of-Experts (MoE) hybrid backbone, specifically the Nemotron 3 Nano 30B-A3B (30B total parameters, 3B active). The architecture follows an Encoder-Projector-Decoder design:

• LLM Backbone: The 30B-A3B MoE model provides the core reasoning and language capabilities.
• Vision Encoder: Utilizes C-RADIOv4-H1, a high-performance vision foundation model.
• Audio Encoder: Uses Parakeet-TDT-0.6B-v2, a FastConformer-based encoder for speech and audio understanding.
• Connectors: Modality-specific MLP projectors align vision and audio tokens with the LLM's embedding space.

Key Architectural Innovations:

• Dynamic Image Resolution: Replaces static tiling with a dynamic strategy that preserves native aspect ratios, decomposing images into variable $16 \times 16$ patches (1,024–13,312 tokens).
• Temporal Video Compression: Introduces Conv3D-based temporal compression, fusing every two consecutive frames into a single "tubelet" before the ViT block, reducing temporal tokens by 2×.
• Audio Tokenization: Audio is resampled to 16kHz, encoded via log-mel spectrograms, and downsampled to ~12.5 tokens/second.
• Context Length: Supports up to 256K tokens, enabling analysis of long documents and extended video/audio streams.

2.2 Training Strategy

The model employs a multi-stage curriculum learning approach to ensure stable cross-modal alignment and prevent catastrophic forgetting:

1. Stage 0 (Vision Warmup): Trains only the vision projector (16k context).
2. Stage 1 (Vision SFT): Joint vision-language fine-tuning with high-quality synthetic data (Qwen3-VL, Kimi-K2.5).
3. Stage 2 (Audio Warmup): Trains the audio projector using ASR data (Granary v1.1).
4. Stage 3 (Audio Encoder & Proj): Unfreezes the audio encoder for joint training on sound, music, and speech.
5. Stage 4 (Joint Omni SFT 16k): Full model training on a mix of vision, audio, video, and text data.
6. Stage 5 (Omni SFT 48k): Extends context to 48k, emphasizing long video and reasoning data.
7. Stage 6 (Omni SFT 256k): Extends context to 256k, focusing on ultra-long documents and text reasoning.
8. Reinforcement Learning (RL): Post-training via Mixed Preference Optimization (MPO) and Group Sequence Policy Optimization (GSPO) across text, image, video, and audio modalities to enhance reasoning and safety.

2.3 Efficiency Techniques

• Efficient Video Sampling (EVS): A runtime pruning mechanism that drops redundant video tokens based on cosine dissimilarity, further reducing token counts after the ViT.
• Quantization: Releases models in BF16, FP8, and NVFP4 (4.98 effective bits/weight) formats, achieving significant memory savings with minimal accuracy loss (<1% median drop).

3. Key Contributions

1. Native Omni-Modal Support: First in the Nemotron series to natively support audio inputs alongside text, images, and video, enabling true cross-modal reasoning.
2. State-of-the-Art Efficiency: Delivers 3× higher throughput than its predecessor (Nemotron Nano V2 VL) and 7.5× higher throughput than Qwen3-Omni on NVIDIA B200 GPUs, while maintaining low latency.
3. Advanced Token Reduction: Combines Conv3D temporal compression and EVS to reduce video input tokens by up to 70%, drastically lowering inference costs.
4. Ultra-Long Context: Successfully scales to 256K context length, outperforming predecessors in long-document understanding and long-video comprehension.
5. Open Release: Provides model checkpoints (BF16, FP8, FP4), training datasets (~6.9M samples), data pipelines, and code (Megatron-Bridge, NeMo-RL) to the community.

4. Results

4.1 Performance Benchmarks

Nemotron 3 Nano Omni achieves leading or competitive results across diverse modalities:

• Document & OCR: Top-tier performance on OCRBench-V2 (88.3), MMLongBench-DOC (46.1), and ChartQA (89.9).
• Visual Reasoning: Strong scores on MMMU (55.2), MathVista-Mini (71.9), and RefCOCO (80.6).
• Audio & Voice: Outperforms Qwen3-Omni on VoiceBench (89.4 avg) and achieves competitive OpenASR results (5.95 WER avg).
• Omni-Modal (Video+Audio): Leads on DailyOmni (74.5) and WorldSense (55.2), demonstrating superior temporal and cross-modal alignment.
• Text Reasoning: Retains the strong reasoning capabilities of the base 30B-A3B LLM (e.g., MMLU-Pro: 77.3, GPQA: 72.2).

4.2 Efficiency Metrics

• Throughput: On a single NVIDIA B200, it achieves 5,000 output tokens/s on multi-document workloads and >500 tokens/s in single-stream low-latency mode.
• Latency: Time-to-First-Token (TTFT) is reduced by 33% (from ~8s to ~5.3s) when combining Conv3D and EVS.
• Quantization: The NVFP4 version reduces model size from 61.5 GB (BF16) to 20.9 GB with negligible accuracy degradation.

5. Significance

Nemotron 3 Nano Omni represents a major leap in efficient, open, and agentic multimodal intelligence. By solving the trade-off between model capability and inference cost, it enables:

• Real-World Deployment: The high throughput and low latency make it viable for real-time agents, such as autonomous GUI controllers and voice assistants.
• Long-Form Analysis: The 256K context and token-reduction techniques allow for the analysis of entire books, financial reports, and hour-long videos without memory bottlenecks.
• Community Advancement: The open release of checkpoints, data, and training pipelines democratizes access to state-of-the-art omni-modal training recipes, fostering further research in efficient multimodal AI.

This work establishes a new standard for open-source omni-modal models, proving that high-performance reasoning across text, vision, and audio can be achieved with a compact, efficient architecture.