💻 computer science

Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learning

The paper proposes SCORE, a reinforcement learning-based framework that utilizes surprise-augmented state representations and a curriculum learning strategy to achieve adaptive token compression, significantly accelerating video understanding while maintaining near-perfect performance.

Original authors: Shida Wang, YongXiang Hua, Zhou Tao, Haoyu Cao, Linli Xu

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shida Wang, YongXiang Hua, Zhou Tao, Haoyu Cao, Linli Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a two-hour movie to a friend who has a very short attention span and can only listen to a few sentences before getting overwhelmed.

If you try to describe every single frame of the movie—every leaf on a tree, every cloud in the sky, every second of a character standing still—your friend's brain will crash. They will get confused, forget the beginning of the story, and miss the important plot twists. In the world of Artificial Intelligence, this is called "Context Rot." The AI is drowning in too much information, and the "noise" (redundant, boring frames) drowns out the "signal" (the exciting action).

This paper introduces a new system called SCORE to solve this problem. Think of SCORE as a super-smart, fast-forward button that doesn't just skip randomly, but knows exactly what to keep and what to throw away.

Here is how SCORE works, broken down into simple concepts:

1. The Problem: The "Information Flood"

Modern AI models (like the ones that watch videos) are like sponges. When you feed them a video, they turn every frame into thousands of tiny data points called "tokens."

The Issue: A 10-minute video might generate 50,000 tokens. But 80% of them are boring! They are just the sky, a wall, or a person standing still.
The Result: The AI gets "clogged up." It spends all its energy processing the boring stuff and forgets the important stuff. It's like trying to find a needle in a haystack, but the haystack is made of 99% other needles.

2. The Solution: The "Surprise Detective"

SCORE is a smart filter that sits between the video camera and the AI brain. Its job is to decide which tokens to keep and which to delete before the AI even sees them.

To do this, SCORE uses a trick called "Surprise-Augmented State."

The Analogy: Imagine you are watching a movie. If the screen shows a static image of a wall for 10 seconds, your brain gets bored and stops paying attention. But if a cat suddenly jumps onto the table, your brain goes, "Whoa! Surprise!" and focuses immediately.
How SCORE does it: It compares every frame to the one before it.
- Frame A: A bird sitting on a branch.
- Frame B: The bird is still sitting there. (Difference = 0. Boring. Delete.)
- Frame C: The bird flies away. (Difference = Huge. Surprise! Keep.)
By looking for these "surprises" (changes), SCORE knows exactly when something important is happening and keeps those tokens, while deleting the boring, repetitive ones.

3. The Training: Learning by Playing a Game

How does the AI learn to be this good at filtering? It uses Reinforcement Learning, which is like training a dog with treats.

The Game: The AI tries to compress the video.
The Reward: If it deletes too much and the AI brain can't answer the question correctly, it gets a "bad grade" (penalty). If it deletes the boring stuff but keeps the important stuff, and the AI brain gets a perfect score, it gets a "treat" (reward).
The Curriculum: At first, the AI trains on "fake videos" made of static pictures (easy to learn the basics of what changes). Once it masters that, it moves on to real, messy videos with real motion. This step-by-step learning ensures it doesn't get confused by the chaos of the real world.

4. The Result: Speed and Smarts

The paper shows that SCORE is a game-changer:

Speed: It makes the AI 16 times faster at processing videos. It's like going from a dial-up internet connection to fiber optics.
Smarts: Even when it throws away 90% of the video data (keeping only 10%), the AI still understands the video almost perfectly. In fact, by removing the "noise," the AI sometimes understands the video better than if it had seen the whole thing!

Summary

SCORE is like a personal editor for AI. Instead of forcing the AI to read the entire encyclopedia of a video, SCORE highlights the most exciting chapters, skips the boring filler, and hands the AI a concise, high-quality summary. This allows the AI to think faster, remember better, and understand long videos without getting overwhelmed.

1. Problem Statement

Multimodal Large Language Models (MLLMs) struggle with long-form video understanding due to two primary bottlenecks:

Computational Overhead: Standard vision encoders generate tens of thousands of visual tokens per video. Since self-attention in LLMs scales quadratically ( $O(N^2)$ ) with sequence length, processing these massive token streams results in prohibitive inference latency and memory costs.
Context Rot: As the token sequence grows, the model's ability to utilize information diminishes, particularly for tokens in the middle of the sequence. Massive visual redundancy (e.g., static backgrounds, repetitive actions) drowns out semantically critical information, leading to performance degradation.

Limitations of Existing Methods:

Transformation-based: Methods using fixed pooling or convolution lack content adaptivity, often discarding critical information along with redundancy.
Heuristic-based: Methods relying on similarity metrics or attention pruning are decoupled from the downstream task objective, creating an optimization gap that limits their effectiveness across diverse videos.

2. Methodology: SCORE Framework

The authors propose SCORE (Surprise-augmented token COmpression via REinforcement learning), a unified framework that learns an adaptive, end-to-end token compression policy.

A. Architecture

SCORE inserts a lightweight, trainable Token-Level Visual Compressor between a frozen vision encoder and a frozen LLM.

Input: Visual tokens $X^{[t]}$ from the vision encoder.
State Representation (Surprise-Augmented): To capture temporal dynamics, the compressor does not just look at the current token embedding. It concatenates the current embedding with an inter-frame residual (surprise signal):
$H^{[t]} = [X^{[t]}; \Delta X^{[t]}], \quad \text{where } \Delta X^{[t]} = X^{[t]} - X^{[t-1]}$
This residual acts as a temporal high-pass filter, emphasizing motion and semantic changes while suppressing static redundancy.
Policy Network: A lightweight MLP maps the state $H^{[t]}$ to a retention probability $p_{t,i}$ for each token via a sigmoid function.

B. Training Strategy

The policy is optimized using On-Policy Reinforcement Learning with a specific focus on stability and curriculum learning.

Group Rollouts: For each video, the policy samples $K$ independent binary masks (stochastic pruning). The LLM generates responses for each masked input to evaluate performance.
Reward Function: A composite reward balances Accuracy and Sparsity:
- Performance Reward ( $R_{perf}$ ): Based on the reduction in Cross-Entropy (CE) loss compared to the full-token baseline.
- Sparsity Reward ( $R_{comp}$ ): Measures the compression ratio (percentage of tokens kept).
Split-Advantage Estimator: To stabilize training, rollouts are categorized into two zones:
- Safe Zone ( $\Delta CE > 0$ ): Performance is maintained or improved. The advantage is normalized by sparsity to encourage higher compression among successful samples.
- Penalty Zone ( $\Delta CE \le 0$ ): Performance drops. The advantage is negative and weighted by sparsity to penalize aggressive pruning that hurts accuracy, forcing the policy to retain more tokens.
Two-Stage Curriculum Learning:
- Stage 1 (Pseudo-Videos): Pre-training on synthetic clips created by repeating static images. This creates high-contrast "binary" temporal signals (zero residual for repeats, large residual for transitions), allowing the policy to quickly learn the association between residuals and token importance.
- Stage 2 (Real Videos): Fine-tuning on real video-caption datasets. The policy adapts from the clean pseudo-signals to the subtle, noisy motion of real-world videos.

C. Inference

During inference, the stochastic sampling is replaced by a deterministic global Top-K selection. The model retains the top $\lfloor \rho TN \rfloor$ tokens based on their retention probabilities, preserving their original spatiotemporal positions.

3. Key Contributions

SCORE Framework: The first RL-based token-level visual compressor for video MLLMs that learns per-token retention policies by directly optimizing downstream generation quality.
Surprise-Augmented State: A novel state representation using inter-frame residuals to explicitly capture temporal dynamics, breaking the symmetry of static token embeddings.
Advanced Training Strategy: A synergistic combination of group-wise policy gradients, a split-advantage estimator (handling constraint satisfaction vs. violation), and a progressive pseudo-to-real curriculum.
State-of-the-Art Performance: Demonstrates that learned compression can not only reduce costs but also alleviate context rot, sometimes outperforming uncompressed models.

4. Experimental Results

The method was evaluated on Qwen2.5-VL and LLaVA-Video across three benchmarks: Video-MME, MLVU, and LVBench.

Accuracy vs. Efficiency:
- At 25% retention, SCORE achieves an average score of 58.9, surpassing the uncompressed Vanilla model (57.3) and all baselines. This confirms that removing redundant tokens mitigates context rot.
- At 10% retention, SCORE retains 99.5% of the original performance while achieving a 16× speedup in prefill latency.
Comparison: SCORE consistently outperforms strong baselines like DyCoke, HoliTom, VidCom2, and FastVID across all retention ratios (10%, 25%, 40%).
Generalization: The policy transfers effectively to different MLLM backbones (e.g., LLaVA-Video), indicating it learns a generalizable compression strategy rather than overfitting to a specific model.
Efficiency: Reducing tokens from ~42k to ~4k (10% retention) results in a 16.2× reduction in prefill time with negligible overhead from the compressor itself (<1% of total time).

5. Significance

SCORE addresses a critical barrier to the deployment of video MLLMs in latency-sensitive and real-time applications. By framing token compression as a reinforcement learning problem, it moves beyond static heuristics to create a dynamic, content-aware filter. The key insight is that redundancy is not just a computational burden but a source of "context rot"; intelligently pruning it can actually improve model reasoning capabilities. The proposed curriculum learning and surprise-augmented state encoding provide a robust blueprint for handling the complex temporal dynamics of long-form video understanding.