Technical Summary: LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

1. Problem Statement

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in the explosion of visual-token context length. Current Video LLMs typically follow a multi-stage architecture: an image encoder (e.g., Vision Transformer) processes frames, an alignment projector maps features, and an LLM reasons over interleaved visual and text tokens. As frame counts increase, computational costs rise due to both the vision encoder's per-frame processing and the LLM's quadratic self-attention complexity.

Existing strategies primarily focus on "post-hoc" token reduction, where a heavy, frozen vision encoder extracts dense features, which are then reduced (via spatial/temporal pruning or merging) before entering the LLM. While this alleviates the LLM's computational burden, the authors observe a critical scaling blindspot: as the LLM compute decreases, the latency bottleneck shifts to the expensive per-frame processing of the vision encoder. Consequently, aggressive post-hoc reduction fails to unlock efficient long-video understanding because the vision encoder's latency becomes the prohibitive floor for end-to-end inference.

2. Methodology

To address this, the authors introduce LiteFrame, a lightweight, efficient video encoder backbone designed to internalize spatio-temporal token compression, coupled with a novel training framework.

2.1. Architecture: Spatio-Temporal Token Compressive Encoding

LiteFrame utilizes a compact student encoder (87M parameters, ViT-Base backbone) compared to the heavy teacher (304M parameters, ViT-Large). The architecture is designed to progressively reduce spatio-temporal redundancies:

• Interleaved Layers: Standard spatial attention layers are interleaved with lightweight, depth-wise (DW) 1D temporal convolution layers.
• Strided Downsampling: Depth-wise strided convolution layers are integrated at strategic intervals (after the 4th and 8th blocks) to gradually downsample feature maps in both spatial and temporal dimensions.
• Efficiency: This design allows the model to capture temporal dynamics with significantly lower latency and FLOPs compared to full spatio-temporal attention or standard temporal attention blocks.

2.2. Compressed Token Distillation (CTD)

Training a compact student to match a large teacher while reducing token count is non-trivial. Standard distillation forces the student to learn redundant details. The authors propose Compressed Token Distillation (CTD):

• Supervision Target: Instead of mimicking the teacher's dense output, the student is trained to predict the compressed representation produced by the teacher.
• Weighted Average Pooling (WAP): The authors identify WAP as a superior compression primitive compared to complex token merging or pruning. WAP aggregates dense tokens into a compact latent space using attention-based weighting within spatio-temporal blocks.
• Objective: The student minimizes the Mean Squared Error (MSE) between its output and the teacher's WAP-compressed features. This effectively transfers the attention-based weighting mechanism into the student's static parameters, bypassing the need for runtime attention over redundant patches.

2.3. Language Model Adaptation (LMA)

To bridge the modality gap and optimize the compressed latent space for the LLM, a minimal Language Model Adaptation (LMA) stage is employed:

• The LLM and encoder are fine-tuned on video-text pairs using Low-Rank Adaptation (LoRA).
• This stage aligns the student's latent space with the LLM and enables the model to handle extended temporal contexts (up to 512 frames).

3. Key Contributions

1. Identification of a Scaling Blindspot: The paper highlights that post-hoc token reduction shifts the latency bottleneck from the LLM to the vision encoder, preventing efficient scaling to long videos.
2. LiteFrame Architecture: A lightweight video encoder that internalizes token compression via a spatio-temporal architecture using depth-wise temporal convolutions and strided downsampling.
3. Compressed Token Distillation (CTD): A novel training framework that teaches a student encoder to directly predict information-dense, spatio-temporally compressed representations (via WAP) from a teacher, rather than dense features.
4. Redefining the Pareto Frontier: The integration of LiteFrame and LMA achieves a new latency-accuracy trade-off, processing significantly more frames with lower end-to-end latency.

4. Experimental Results

The authors evaluate LiteFrame against the InternVL3-8B baseline and various state-of-the-art post-hoc methods (e.g., FastVID, PruMerge, ToMe) and efficient encoders (e.g., VideoPanda, AutoGaze).

• Latency and Accuracy: Compared to InternVL3-8B, LiteFrame achieves a 35% reduction in end-to-end latency while processing 8× more frames. For instance, at 64 frames, LiteFrame reduces total latency from ~208ms to ~150ms while improving average accuracy across benchmarks (Video-MME, MLVU, LongVideoBench).
• Comparison with Post-Hoc Methods: While post-hoc methods (like FastVID) reduce LLM tokens, they remain bottlenecked by the heavy original vision encoder. LiteFrame outperforms these methods in both efficiency and accuracy by avoiding the "computational floor" of the original encoder.
• Comparison with AutoGaze: LiteFrame significantly outperforms AutoGaze. While AutoGaze attempts to address both ViT and LLM bottlenecks, its pre-reduction auxiliary module introduces substantial overhead (accounting for nearly half of total inference time in some configurations). LiteFrame achieves lower latency and higher accuracy without such auxiliary overhead.
• Spatial Scaling: LiteFrame demonstrates strong zero-shot spatial scaling capabilities. On the HLVid benchmark, it achieves a state-of-the-art score of 54.1 at 2688px resolution with 48 frames, surpassing AutoGaze (52.6) which required higher resolutions and frame counts.
• Ablation Studies:
  • CTD vs. RTD: CTD (using WAP targets) significantly outperforms Reconstructive Token Distillation (RTD), suggesting that explicit alignment with the WAP primitive provides a more robust semantic foundation than generic auto-encoding.
  • Architecture: Depth-wise temporal convolutions outperform full spatio-temporal attention in terms of latency and accuracy.
  • Compression Strategy: Spatio-temporal compression (4× spatial, 4× temporal) outperforms spatial-only compression (16×), preserving critical spatial fidelity while aggregating temporal redundancy.

5. Significance and Claims

The paper claims that LiteFrame demonstrates a new potential path to unlocking longer-form video understanding under fixed compute budgets. By shifting the focus from "extract-and-reduce" (post-hoc) to "internalize-and-compress" (architectural), the authors show that it is possible to bypass the vision encoder latency bottleneck that has limited current Video LLMs.

The authors emphasize that their approach redefines the efficiency-accuracy Pareto frontier, allowing models to trade spatio-temporal redundancy for significantly richer temporal context. They conclude that the architectural internalization of token compression via distillation is a more scalable solution for long-form video understanding than purely token-reduction methods.

Limitations Acknowledged:

• The LMA stage was trained on a subset of existing video data; higher-quality, extreme long-form datasets could further improve performance.
• The model was evaluated primarily on video-centric benchmarks; performance on static image benchmarks remains unexplored.
• Scaling down to even smaller student models (<87M) was constrained by training instabilities (e.g., loss explosions), indicating a need for further stabilization of the CTD framework.