Mobile-VideoGPT: Fast and Accurate Model for Mobile Video Understanding

1. Problem Statement

Video understanding using Large Multimodal Models (LMMs) has achieved remarkable results but faces significant barriers for deployment on edge devices (e.g., mobile phones, autonomous vehicles, IoT).

• Computational Bottlenecks: Existing state-of-the-art (SoTA) video LMMs often have billions of parameters (e.g., 7B–8B) and large memory footprints (>20GB), making them impractical for real-time inference on resource-constrained hardware.
• Inefficiency: Traditional approaches process all video frames or use high-resolution inputs, leading to redundant computations and slow inference speeds (low tokens per second).
• Trade-offs: Existing efficient models often sacrifice accuracy for speed, or require specialized hardware accelerators that are not universally available.

2. Methodology: Mobile-VideoGPT

The authors propose Mobile-VideoGPT, a lightweight multimodal framework designed specifically for real-time video understanding with fewer than 1 billion parameters. The architecture consists of four core components:

A. Dual-Encoder Architecture

Instead of relying on a single heavy encoder, the model uses a specialized dual-encoder approach to balance spatial and temporal features efficiently:

1. Efficient Image Encoder: A CLIP-based encoder (CLIP-B/16) extracts spatial features from all sampled frames to capture semantic details.
2. Efficient Video Encoder: A lightweight VideoMamba-M encoder extracts temporal features. Crucially, this encoder does not process every frame; it only processes a selected subset of key frames.

B. Attention-Based Frame Scoring (Key Innovation)

To reduce computational load, the model does not feed all $T$ sampled frames into the video encoder.

• Mechanism: The spatial features from the image encoder are flattened, and a spatial attention matrix is computed to score the importance of each frame.
• Selection: The top-$K$ most salient frames (where $K = T/2$, e.g., 8 out of 16) are selected.
• Benefit: This discards redundant frames while preserving crucial contextual information, significantly reducing the input to the temporal encoder.

C. Efficient Token Projector (ET-Proj)

To bridge the visual encoders and the Small Language Model (SLM), the authors introduce a novel projection module that compresses visual tokens without losing fidelity:

• Structure: It consists of a Feed-Forward Network (FFN), Token Reduction via Adaptive Average Pooling (reducing spatial dimensions $H \times W$), and a Positional Encoding module with skip connections.
• Function: It projects high-dimensional spatial and temporal features into a unified vision-language embedding space, effectively pruning redundant tokens while retaining essential spatiotemporal cues.

D. Small Language Model (SLM)

The system utilizes compact SLMs (specifically Qwen-2.5 0.5B and 1.5B) as the reasoning engine. This choice drastically reduces the parameter count compared to standard 7B+ LLMs used in other video models.

E. Training Strategy

The model is trained in three stages:

1. Image Pre-training: Training the image token projector with the image encoder.
2. Video Pre-training: Extending to the video projector with the video encoder.
3. Instruction Tuning: Full fine-tuning of the projectors and LoRA (Low-Rank Adaptation) on the SLM using a curated dataset of ~1M samples (including Kinetics, Something-Something, and NExT-QA).

3. Key Contributions

1. Lightweight Architecture: A dual-encoder design with an SLM that fits within 1GB (FP16) and requires only 3GB VRAM, enabling deployment on edge devices like the NVIDIA Jetson Orin Nano.
2. Novel Frame Selection: An attention-based mechanism that dynamically selects key frames, reducing computational redundancy by 50% without significant accuracy loss.
3. Efficient Token Projection: A specialized module that compresses visual tokens while preserving spatial and temporal dependencies, outperforming standard MLP projectors.
4. Superior Efficiency-Accuracy Trade-off: The model achieves higher throughput and better accuracy than larger models, proving that compact designs can rival massive architectures.

4. Experimental Results

The model was evaluated on six diverse benchmarks: ActivityNet-QA, EgoSchema, MLVU, MVBench, NExTQA, and PerceptionTest.

• Performance vs. SoTA:
  • Mobile-VideoGPT-0.5B outperforms LLaVA-OneVision-0.5B by an average of 6 points across benchmarks.
  • Mobile-VideoGPT-1.5B surpasses the 8B-parameter LLaVA-Mini on several metrics (e.g., +2.1% on ActivityNet-QA) while being significantly smaller.
• Efficiency & Throughput:
  • Edge Device (Jetson Orin Nano): Achieves 7.3 tokens/second.
  • Desktop (RTX A6000): Achieves 45.9 tokens/second.
  • Comparison: It is 2× faster than LLaVA-OneVision-0.5B and 9× faster than LLaVA-Mini-8B on desktop GPUs.
• Qualitative Results: The model demonstrates superior temporal reasoning and scene description capabilities, providing accurate, context-grounded answers with lower latency (e.g., 0.3s latency vs. 13.5s for LLaVA-Mini on complex queries).

5. Significance and Impact

• Democratizing Video AI: Mobile-VideoGPT bridges the gap between high-performance video understanding and the strict resource constraints of edge devices. It enables real-time, on-device video analysis without relying on cloud computing.
• Architectural Insight: The paper demonstrates that dual-encoder designs combined with intelligent frame selection are more effective for video tasks than simply scaling up a single encoder or using massive language models.
• Practical Deployment: With a model size of just 1GB and high throughput, this framework opens new possibilities for applications in robotics, autonomous driving, and mobile augmented reality where low latency and high efficiency are critical.

In summary, Mobile-VideoGPT proves that by optimizing the encoder selection strategy and token projection, it is possible to build video understanding models that are not only accurate but also fast enough for real-time, on-device deployment.