Technical Summary: MOSS-Video-Preview

Problem Statement

Current video understanding models predominantly operate under an offline paradigm, where the entire video clip is available before generation begins. While "streaming" systems have emerged to respond during playback, they typically treat perception and generation as serial, mutually exclusive phases: the model stops ingesting new frames while generating a reply. This creates a latency gap where the model cannot react to events occurring within the reply window, delaying corrections by at least one response length.

The paper identifies a gap for real-time interaction, defined by a single core constraint: perception must not be blocked by generation. In this paradigm, a model must continuously perceive new frames while generating text, enabling it to revise or interrupt its current answer immediately upon detecting new evidence, and to remain silent when no relevant changes occur. Existing architectures, particularly decoder-only designs, struggle to support this because they force visual features into the autoregressive sequence, causing perception and generation to compete for the same context window.

Methodology

1. Architecture: Cross-Attention Backbone

The authors propose a two-channel architecture based on a cross-attention backbone (specifically adapting Llama-3.2-11B-Vision) rather than the prevailing decoder-only design.

• Separation of Pathways: Visual features are injected via a side channel (Keys and Values) rather than joining the autoregressive token stream. The text backbone generates tokens autoregressively while retrieving visual context from the side channel.
• Structural Benefits:
  • Non-blocking: Perception (frame reading) and generation (token output) run on physically separate pathways, allowing them to proceed concurrently.
  • Efficiency: Visual features are encoded once during the prefill stage and retrieved only in specific cross-attention layers (8 out of 40), avoiding the repeated self-attention overhead of large visual token pools in decoder-only models.
  • Independent Compression: The separation allows for independent compression of visual tokens (via 2D pooling) without affecting the text sequence.
• Temporal Alignment: The model employs Rotary Positional Encoding (RoPE) on both text and visual sides, mapping frames to a unified temporal axis based on their arrival order, enabling the model to understand temporal sequences.

2. Data Synthesis Pipeline

Since static caption and QA datasets do not contain trajectories of evolving answers or silence decisions, the authors introduce a real-time data synthesis pipeline to convert dense hierarchical captions into real-time supervision.

• Semantic Construction: The pipeline identifies "change points" in video descriptions where the state of the world shifts. It generates state-dependent instructions (questions) where the correct answer changes over time.
• Decision-Generation: For each instruction, the model synthesizes a response trajectory that includes:
  • Revisions: Updating the answer when new evidence appears.
  • Silence: Emitting a special <|silence|> token when no new information is available or the question is not yet answerable.
• Temporal Layout: The synthesized responses are aligned to a per-second frame stream, interleaving <|video|> placeholders with text or <|silence|> tokens to simulate a continuous, event-driven interaction.

3. Training Strategy

The training follows a "general first, specialize later" approach:

1. Bridge Training: Aligning the vision encoder and projector with the frozen LLM backbone.
2. Offline SFT: Training on large-scale general instruction data to establish robust offline video and multimodal understanding.
3. Real-Time SFT: Specializing the offline model using the synthesized real-time data. This stage uses a mixed corpus of real-time and offline QA, distinguished by system prompts, to teach the model when to speak, when to revise, and when to stay silent.

• Inference Control: At inference, a threshold-based gating mechanism (e.g., $P(\text{silence}) \geq 0.6$) determines whether the model outputs text or <|silence|>, allowing a single set of weights to support both offline and real-time modes.

Key Contributions

1. Paradigm Formalization: The paper explicitly distinguishes between offline, streaming, and real-time paradigms, defining real-time interaction by the concurrency of perception and generation and the ability to perform "timely corrections."
2. Architectural Design: It demonstrates that a cross-attention backbone is structurally superior to decoder-only designs for real-time interaction, providing a non-blocking, two-channel pathway that reduces visual processing frequency.
3. Data Synthesis: It introduces a pipeline to generate real-time supervision data, including dynamic answer revisions and silence decisions, which are absent in standard datasets.
4. Real-Time Behavior: It successfully instills continuous perception, answer revision, and proactive silence behaviors into a large language model without sacrificing its offline capabilities.

Results

Offline and Multimodal Understanding

• Competitive Performance: The model achieves competitive results on general video and multimodal benchmarks. While it trails the strong Qwen2.5-VL-7B baseline on memory-intensive tasks (e.g., OCR, fine-grained perception) and general video QA, it outperforms Qwen2.5-VL-7B on benchmarks requiring spatial and fine-grained temporal reasoning (e.g., VisuLogic, VSI-Bench, Video-Holmes).
• Cost of Specialization: The transition from offline to real-time specialization incurs negligible degradation in offline understanding capabilities. In some video benchmarks, the real-time version even outperforms the offline-only version.

Inference Efficiency

• Speedup: On a single NVIDIA H200 with 256 frames per video, the 11B parameter MOSS-Video-Preview achieves:
  • ~5× speedup in Time to First Token (TTFT) compared to Qwen2.5-VL-7B (1.95s vs. 9.94s).
  • ~2.7× higher decoding throughput (38.41 vs. 14.26 tokens/s).
  • ~46% reduction in end-to-end total latency.
• Attribution: These gains are attributed directly to the cross-attention architecture (lifting vision out of the autoregressive sequence) rather than engineering optimizations in the serving stack.

Significance and Claims

The paper positions itself as a preview and a position paper, prioritizing the validation of a paradigm and architecture over achieving state-of-the-art (SOTA) performance on general benchmarks.

• Feasibility: The study validates that real-time video understanding is feasible using a cross-attention backbone and a specialized data synthesis pipeline.
• Architectural Necessity: It argues that the "perception must not be blocked by generation" constraint necessitates a two-channel architecture, making cross-attention a natural fit over decoder-only designs.
• Scalability: The authors attribute the remaining performance gap with SOTA models primarily to data scale and parameter size, not the architecture itself. They claim the architecture provides a viable path forward, with the potential to close the gap through further scaling.
• Limitations: The paper acknowledges the lack of a standardized decision-level latency benchmark for real-time evaluation, noting that current metrics often fail to capture the "timeliness" of corrections. It also admits that data engineering and ablation studies are preliminary.

In summary, MOSS-Video-Preview outlines a viable path toward real-time video understanding by decoupling perception from generation, demonstrating that this approach yields significant efficiency gains and robust reasoning capabilities, even if general benchmark scores currently lag behind larger, offline-optimized models.