💻 computer science

MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding

The paper proposes MARC, a memory-augmented reinforcement learning framework that achieves near-baseline video understanding accuracy while reducing visual tokens by 95% and GPU memory by 72% through a novel retrieve-then-compress strategy combining a Visual Memory Retriever and a Compression Group Relative Policy Optimization distillation method.

Original authors: Peiran Wu, Zhuorui Yu, Yunze Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peiran Wu, Zhuorui Yu, Yunze Liu, Chi-Hao Wu, Enmin Zhou, Junxiao Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, high-definition movie file that you need to show to a very smart but busy assistant (an AI model). The problem is, the file is so huge that the assistant gets overwhelmed, runs out of memory, and takes forever to give you an answer. This is the current struggle with AI video understanding: videos are too big to process quickly.

The paper introduces a new system called MARC (Memory-Augmented RL Token Compression) to solve this. Think of MARC as a two-step "smart editor" that shrinks a movie down to the size of a single photo without losing the story.

Here is how it works, using simple analogies:

1. The Problem: The "Too Much Information" Bottleneck

Currently, to understand a video, AI models often try to look at every single frame, like watching a movie at 60 frames per second. If the video is long, this creates a mountain of data.

The Analogy: Imagine trying to read a 500-page book to answer one simple question. It's inefficient. You don't need to read every word on every page; you just need the specific chapters where the answer is hidden.

2. Step One: The "Visual Memory Retriever" (The Smart Librarian)

Before the AI tries to compress the video, it first needs to find the right parts. The paper calls this the Visual Memory Retriever (VMR).

How it works: Instead of looking at the video as a continuous stream, this tool acts like a human librarian who understands stories. It breaks the video down into "events" (like scenes in a movie).
The Metaphor: If you ask, "What happened when the car crashed?", the librarian doesn't show you the whole movie. They immediately pull out the specific 3 clips where the crash happens and the moments right before and after. They ignore the boring parts where nothing happens.
The Result: The AI now only has to deal with a few short, relevant clips instead of the whole movie.

3. Step Two: C-GRPO (The "Master and Apprentice" Training)

Now that the AI has the right clips, it still has too many details (tokens) to process efficiently. The paper introduces a new training method called C-GRPO.

The Analogy: Imagine a Master Chef (the Teacher) who cooks a complex 64-course meal using a full kitchen. Then, you have an Apprentice (the Student) who is only allowed to use a tiny camping stove and one ingredient.
The Challenge: Usually, if you force the Apprentice to use so little, the food tastes terrible.
The Solution: The paper uses a special "Reinforcement Learning" reward system. The Apprentice tries to cook, and the system checks: "Did the Apprentice's dish taste as good as the Master's, even though they used so little?"
- If the Apprentice succeeds, they get a reward.
- If they fail, they get a penalty.
The Outcome: Through this trial-and-error process, the Apprentice learns to extract the essence of the flavor (the reasoning) using only a tiny fraction of the ingredients.

4. The Results: Shrinking the Elephant

The paper claims that by combining the "Smart Librarian" (finding the right clips) and the "Master/Apprentice" training (compressing the data):

Size Reduction: They can take a video that normally requires 64 frames of data to understand and compress it down to the equivalent of just 1 frame. That is a 95% reduction in data.
Performance: Despite using 95% less data, the AI's ability to answer questions remains almost exactly the same as if it had seen the full 64 frames.
Speed & Memory:
- Memory: It uses 72% less computer memory (RAM).
- Speed: It generates answers 23.9% faster.

Why This Matters (According to the Paper)

The authors state that this makes it possible to run these powerful video-AI models on devices with limited resources. They specifically mention applications like:

Real-time video question answering (asking questions about a video as it happens).
Surveillance systems (monitoring cameras without needing supercomputers).
Autonomous driving (cars needing to understand video quickly with limited onboard power).

In short, MARC teaches an AI to be a "skimmer" rather than a "reader," finding the most important moments and learning to understand them deeply without needing to process the entire file.

Technical Summary: MARC – Memory-Augmented RL Token Compression for Efficient Video Understanding

Problem Statement

Visual Language Models (VLMs) face significant computational bottlenecks when scaling from image to video understanding. High frame rates and long video durations necessitate processing thousands of tokens, leading to sharp increases in inference costs, latency, and GPU memory usage. This hinders deployment in resource-constrained and latency-sensitive environments such as autonomous driving and surveillance.

While token compression is a promising solution, existing methods rely primarily on training-free strategies (e.g., spatial or temporal token merging). The authors argue that these training-free approaches inevitably cause information loss during compression, resulting in a significant drop in model performance. The core challenge is to achieve aggressive token reduction (e.g., reducing a video to the token count of a single image) without sacrificing the reasoning capabilities of the model.

Methodology

The paper proposes MARC (Memory-Augmented Reinforcement Learning-based Token Compression), a two-stage framework designed to retrieve relevant information before compressing it.

1. Visual Memory Retriever (VMR)

Inspired by cognitive science and event segmentation theory, the VMR replaces fixed-length temporal windows with a semantic, event-based approach.

Event-Based Segmentation: A deep event detection network partitions continuous video streams into semantically coherent, short clips (visual memory fragments) based on scene changes, topic shifts, or action commencements.
Memory Retrieval: Query representations and visual fragments are mapped into a shared latent space using contrastive learning. A nearest-neighbor search retrieves the top- $k$ fragments most relevant to the user query.
Function: This step acts as a filter, reducing the search space and ensuring that subsequent compression operates only on "episodic memories" containing the most causally useful evidence, rather than redundant frames.

2. Memory-Aware Temporal Compression

Before the RL distillation, a specific compression layer is applied to the retrieved segments:

Local Coherence: Within each retrieved event segment, consecutive frames are merged based on cosine similarity to retain local dynamics while reducing redundancy.
Global Consolidation: If the token budget is still exceeded, a lightweight cross-segment merge is applied. This ensures the event structure defined by the VMR is preserved.

3. Compression Group Relative Policy Optimization (C-GRPO)

To maintain performance under extreme compression (reducing input to ~1 frame's worth of tokens), the authors introduce C-GRPO, a post-training reinforcement learning strategy.

Teacher-Student Distillation: A "Teacher" network processes 64 frames, while a "Student" network processes the compressed single-frame input.
Retention Alignment Reward: Unlike standard GRPO which focuses on answer correctness, C-GRPO introduces a compression reward ( $r_c$ $r_{c}$ ). This reward is calculated based on the retention ratio ( $\eta$ $η$ ), which measures how much of the Teacher's performance the Student retains.
- $r_c = \alpha \cdot \max(0, \eta - \tau)$ , where $\tau$ is a minimum acceptable retention threshold (set to 0.6).
Gating Mechanism: The retention reward is gated by correctness ( $1[\text{correct}]$ ), ensuring that only semantically valid generations receive the bonus. This prevents "reward hacking" where the model might guess correctly by chance without actually retaining reasoning ability.
Objective: The policy is optimized to maximize the retention of the Teacher's reasoning ability while adhering to the strict token budget.

Key Contributions

MARC Framework: A novel architecture integrating structured visual retrieval (VMR) with RL-based token compression, enabling efficient video understanding that mimics human episodic memory.
C-GRPO Algorithm: The first post-training RL strategy specifically designed for video token compression. It successfully distills complex reasoning abilities from a high-token Teacher to a low-token Student by explicitly optimizing for performance retention.
Empirical Validation: Extensive experiments across six benchmarks demonstrating that the method achieves near-baseline performance with drastically reduced computational costs.

Experimental Results

The authors evaluated MARC on six benchmarks: VSI-Bench, VideoMMMU, MMVU, MVBench, TempCompass, and VideoMME, using Qwen2.5-VL-3B as the backbone.

Performance: MARC achieved a mean score of 42.20, nearly identical to the 64-frame baseline of 42.21.
- It outperformed the baseline on MMVU, MVBench, and TempCompass.
- It surpassed larger models (e.g., InternVL3.5-4B, Gemma-3-4B) despite using a 3B parameter model.
Token Reduction: The method reduced visual tokens by 95% (from ~2590 tokens to ~123 tokens), effectively using only the token equivalent of a single frame (4.71% of original tokens).
Efficiency Gains:
- GPU Memory: Reduced by 72.4% (from 41.63 GB to 11.48 GB on an A6000).
- Latency: Generation latency reduced by 23.9%, with a 11.1% reduction in end-to-end latency for processing samples.
Ablation Studies:
- VMR: Using VMR alone boosted baseline accuracy (mean 45.56 vs 42.21), proving the value of event-based retrieval.
- C-GRPO vs. SFT: MARC (C-GRPO) outperformed a Supervised Fine-Tuning (SFT) baseline by 9.6% in mean score, highlighting the superiority of RL distillation for compression tasks.
- Threshold ( $\tau$ ): A threshold of 0.6 provided the optimal balance between stability and performance retention.

Significance and Claims

The paper claims that MARC offers a robust solution for the post-training compression of large-scale models, bridging the gap between efficiency and accuracy. By combining cognitive-inspired retrieval with RL-based distillation, the method enables the deployment of VLMs in scenarios with strict resource constraints, such as real-time video question answering, surveillance, and autonomous driving.

The authors emphasize that their approach avoids the performance degradation typical of training-free compression methods. Instead, by treating compression as an alignment problem where the student must match the teacher's reasoning, MARC maintains high fidelity even when reducing input to a single frame's worth of tokens. The work is presented as a practical pathway to making video understanding scalable and deployable in real-world, latency-sensitive applications.