Technical Summary: Toward Native Multimodal Modeling (NMM) Roadmap

Problem Statement

Multimodal modeling has evolved from modality-agnostic text processors toward holistic world models, yet the field lacks a formalized definition and structured roadmap for Native Multimodal Modeling (NMM). Early approaches relied on late-fusion paradigms (e.g., LLaVA, DeepSeek-VL), which assemble pre-trained encoders with frozen language backbones via shallow projectors. These non-native compositions suffer from a fundamental blindness to raw sensory signals, limiting the depth of cross-modal interaction and preventing true synergy. While recent efforts have shifted toward native architectures that intrinsically integrate modalities, the design space remains fragmented. There is an insufficient definition of "architectural nativity," hindering the community's ability to evaluate the degree of integration in emergent models and select optimal architectures for specific tasks.

Methodology and Framework

The paper proposes a formalized taxonomy and a comprehensive investigation into the transition from modular assembly to native convergence. The methodology is structured around two primary dimensions: Architectural Nativity (depth of integration) and Input-Output Duality (functional flow).

1. Formalizing Architectural Nativity

The authors define NMM as a paradigm where multimodal synergy is an intrinsic architectural property, distinguishing it from late-fusion through two regimes:

• Mid-Fusion: A naturally interacted regime where features from distinct encoders are injected into a joint multimodal backbone (e.g., via cross-attention or adapters). While insightful across modalities, it maintains explicit modality-aware boundaries and structural asymmetry between upstream encoders and the central backbone.
• Early-Fusion: A "born-native" regime where all modalities are mapped by a unified operator into a single, shared embedding space from the outset, bypassing independent frozen encoders. This achieves deep synergy by treating all modalities as equivalent tokens within a unified transformer.

2. Taxonomy by Input-Output Duality

The existing NMM ecosystem is organized into three functional categories based on modality flow:

• Multi-to-Text (M2T) Unimodal Generation: Asymmetric comprehension where multimodal inputs are collapsed into a single linguistic space for reasoning (e.g., Qwen3-VL, MiniCPM-V).
• Multi-to-Target (M2G) Scenario-based Generation: Asymmetric generation where the model synthesizes specific non-textual targets (image, audio, video) directly from native hidden representations, bypassing post-hoc decoders (e.g., Wan2.2, Kling-Omni).
• Multi-to-Multi (M2M) Symmetric Modeling: The ultimate phase where understanding and generation coexist symmetrically within a single network, enabling any-to-any intelligence (e.g., Emu3.5, Moshi, Chameleon).

3. Industrial-Grade Investigation

The paper systematically unpacks the end-to-end pipeline for NMM, analyzing:

• Model Architecture: Technical challenges in unifying modalities, handling computational explosions (especially in video), and bridging discrete/continuous paradigms.
• Dataset Curation: The shift from simple image-text pairs to heterogeneous mixtures including interleaved sequences, interaction-oriented data (web/GUI/robotics), and preference/alignment signals.
• Training Strategies: The evolution of training signatures across fusion regimes, including differential learning rates in mid-fusion, unified Next-Token Prediction (NTP) in early-fusion, and the specific roles of Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), and On-Policy Distillation (OPD).
• Inference & Deployment: Strategies for mitigating sequence explosion, managing heterogeneity/scale via MoE and discrete tokenization, and enabling real-time full-duplex streaming.
• Evaluation: The need for benchmarks that jointly assess understanding and generation across modalities, moving beyond static metrics to temporally-aware and efficiency-aware protocols.

Key Contributions

1. Problem Formalization: The paper presents the first systemic definition of NMM, establishing a principled structural taxonomy based on integration depth (Mid/Early-Fusion) and input-output duality (M2T/M2G/M2M) to clarify the fragmented design space.
2. Technological Roadmap: It provides a comprehensive analysis of the full lifecycle of NMM, extracting core bottlenecks and technical solutions across architecture, data, training, inference, and evaluation. It details specific engineering recipes, such as the necessity of differential learning rates in mid-fusion and the critical role of modality-mixture scheduling in early-fusion.
3. Future Outlook: The authors offer empirical insights from state-of-the-art implementations to project future trajectories, identifying crucial strategic directions for the evolution toward advanced NMM, including the convergence toward symmetric M2M models and the transition to self-generating multimodal streams.

Results and Findings

The paper does not present new experimental results but synthesizes findings from a wide range of existing models (e.g., Qwen2.5-VL, Kimi K2.5, Emu3.5, Chameleon, Moshi) to illustrate the current state of the art. Key findings include:

• Architectural Shift: The field is rapidly moving from late-fusion to mid-fusion and increasingly toward early-fusion "born-native" transformers.
• Training Signatures: Different fusion regimes impose distinct training requirements. Early-fusion models require mandatory stabilization techniques (e.g., QK-Norm, Z-loss) and precise modality-mixture scheduling to prevent degenerate priors, whereas mid-fusion relies on differential learning rates and progressive unfreezing.
• Data Complexity: Effective NMM requires complex data curricula that balance understanding-oriented, generation-oriented, and interaction-oriented data, with a growing emphasis on interleaved and preference-aligned datasets.
• Evaluation Gaps: Current benchmarks often evaluate modalities in isolation. There is a critical need for symmetric M2M benchmarks that assess the consistency between understanding and generation, as well as temporally-aware metrics for streaming interactions.

Significance and Claims

The paper claims to provide a formalized roadmap for the community to transition from modular assembly to native convergence. Its significance lies in:

• Clarifying the Design Space: By formally defining "nativity," it enables the evaluation of the degree of integration in emerging models, addressing a lack of formalization that previously complicated architecture selection.
• Industrial Relevance: It delivers an "industrial-grade" investigation, offering practical insights into the full-stack pipeline from data curation to deployment, which is essential for building truly native world models.
• Strategic Direction: It synthesizes open problems and outlines research directions necessary for the evolution toward Native World Models—unified backbones that perceive raw sensory streams, maintain persistent state, and act in continuous time.

The authors position this work not as a final solution but as a structured starting point for the next phase of multimodal intelligence, emphasizing the journey toward a unified, symmetric, streaming, and embodied multimodal intelligence.