MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding

This paper introduces MSJoE, a novel framework that jointly evolves a multimodal large language model and a lightweight key-frame sampler via reinforcement learning to efficiently select informative frames for long-form video understanding, achieving state-of-the-art performance on multiple benchmarks.

The Gist

Imagine you are trying to solve a mystery, but instead of a few clues, you are handed a 10-hour security camera tape of a busy city. You need to find the one moment where a specific person dropped a key.

If you watch the whole tape at normal speed, it takes forever. If you just watch every 10th second (uniform sampling), you might miss the key drop because it happened in the split second between your glances.

This is the problem MSJoE solves for Artificial Intelligence.

Here is the simple breakdown of how this new system works, using a few creative analogies.

The Problem: The "Too Much Information" Bottleneck

Current AI models (called MLLMs) are like brilliant detectives who can read and understand text and images. But when you give them a long video, they get overwhelmed.

• The Old Way: They try to look at every frame, or they just pick frames at random intervals (like checking your watch every 5 minutes). This is slow and often misses the important stuff.
• The Flaw: Most of the video is boring (people walking, clouds moving). The AI wastes its brainpower on the boring parts and misses the "key frames" (the detective clues).

The Solution: The "Smart Detective & The Scout" Team

The authors of this paper created a team of two AI agents that learn to work together perfectly. They call this MSJoE (Jointly Evolving MLLM and Sampler).

Think of it as a Detective and a Scout.

1. The Scout (The Sampler)

The Scout's job is to watch the 10-hour video and pick out the 32 most important frames to show the Detective.

• The Old Scouts: They used simple rules (e.g., "Pick a frame every 5 minutes"). Sometimes they picked boring frames.
• The MSJoE Scout: This Scout is "trainable." It learns to look for specific things. But it can't guess what to look for on its own yet.

2. The Detective (The MLLM)

The Detective is the big brain. It looks at the frames the Scout brings and answers the question.

• The Problem: If the Detective just says, "Find me the key," the Scout might not know what a "key" looks like in a video context.
• The MSJoE Detective: Instead of just saying "Find the key," the Detective first thinks and breaks the question down into specific visual clues.
  • Bad Question: "Who dropped the key?"
  • MSJoE Detective's Clues: "Show me a hand holding a metal object," "Show me a person looking down at the ground," "Show me a shiny object on the sidewalk."

How They Learn Together (The "Joint Evolution")

This is the magic part. In previous methods, the Detective and the Scout were trained separately. The Detective didn't know how the Scout picked frames, and the Scout didn't know what the Detective needed.

In MSJoE, they train together using a method called Reinforcement Learning (think of it as a video game where they get points for winning).

1. The Loop:

   • The Detective looks at a tiny preview of the video and writes a list of "Visual Clues" (Queries).
   • The Scout uses those clues to scan the whole video and picks the best frames.
   • The Detective looks at those frames and answers the question.
   • If they get the answer right: Both get a "High Five" (Reward).
   • If they get it wrong: They both learn what went wrong.

2. The Result:

   • The Detective learns to write better, more specific clues so the Scout can find the right frames.
   • The Scout learns to ignore the boring parts and focus exactly on what the Detective is asking for.
   • They evolve together, becoming a perfect team.

The "Training Camp" (The New Dataset)

To train this team, the researchers realized existing videos weren't hard enough. So, they built a new "Gym" (a dataset called LongVideoQA).

• They took 2,800 long videos (movies, sports, docs).
• They used AI to generate thousands of tricky questions that require connecting events across time (e.g., "Why did the character change their diet?" requires seeing the dentist visit and the family dinner).
• They filtered out the easy questions so the AI team only practiced on the hard stuff.

Why Does This Matter?

• Speed: It's much faster. Instead of processing 1,000 frames, it processes 32.
• Smarter: It doesn't just guess; it reasons about what to look for before looking.
• Better Results: In tests, this team beat the previous best methods by a significant margin (about 8% better accuracy).

The Bottom Line

Imagine trying to find a needle in a haystack.

• Old AI: Grabs a handful of hay from random spots.
• MSJoE: First, it asks, "What does the needle look like?" Then it sends a magnet (the Scout) to pull out only the metal parts, and finally, it examines the metal to find the needle.

By teaching the AI to think about what to look for and learn how to look for it at the same time, they made long-video understanding efficient, accurate, and much smarter.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) struggle with long-form video understanding due to two primary bottlenecks:

• Computational Efficiency: Processing long videos with dense uniform sampling causes the visual context length to scale linearly with duration, while attention computation grows quadratically. This makes traditional approaches computationally prohibitive.
• Information Redundancy & Missed Events: Most frames in a long video are visually similar or irrelevant to a specific question. Fixed uniform sampling strategies often miss critical, sparse events or waste compute on uninformative regions.

Existing solutions face three specific challenges:

1. Insufficiency (Q1): The question text alone often lacks explicit visual cues, making direct CLIP-based retrieval unreliable.
2. Sampling Strategy (Q2): Converting similarity scores into sampling weights is non-trivial; naive top-k selection often picks redundant frames. Heuristic algorithms lack generalizability.
3. Lack of Collaboration (Q3): Current methods typically freeze the MLLM while training a sampler (or vice versa). This prevents the MLLM from learning to generate better queries to guide the sampler, and prevents the sampler from adapting to the specific reasoning needs of the MLLM.

2. Methodology: MSJoE Framework

The authors propose MSJoE (MLLM-Sampler Joint Evolution), a unified framework that jointly optimizes an MLLM and a lightweight key-frame sampler using Reinforcement Learning (RL).

Core Components

1. Query-Guided Retrieval:
   • Instead of using the raw question for retrieval, the MLLM first processes a sparse video preview (low-resolution, few frames) and the question to reason out multiple visual queries (e.g., "An image of a tooth," "An image of a doctor").
   • These queries describe diverse visual perspectives relevant to the answer, addressing the "Insufficiency" problem.
2. Similarity Matrix Construction:
   • The generated queries are paired with densely sampled frames (1 FPS) and passed through a frozen CLIP model to create a query-frame similarity matrix.
3. Learnable Sampler (U-Sampler):
   • A lightweight 1D U-Net (approx. 2M parameters) takes the similarity matrix as input.
   • It learns to predict sampling weights for each frame, selecting a compact set of diverse, informative key-frames. This addresses the "Sampling" problem by learning to transform similarity scores into optimal selection probabilities rather than using fixed heuristics.
4. Joint Evolution via Reinforcement Learning:
   • Training Strategy: Both the MLLM and the Sampler are trained end-to-end using GRPO (Group Relative Policy Optimization).
   • Reward Design: The system is optimized using a composite reward:
     • Accuracy Reward: Based on correct answers.
     • Format Reward: Ensures structured output.
     • Informativeness Reward: Encourages the MLLM to generate queries that produce peaked similarity distributions (high attention on specific frames) rather than generic matches.
   • Pre-training: To stabilize training, the sampler undergoes a difficulty-aware pre-training phase before joint RL, where rewards are adjusted based on the inherent difficulty of the question (estimated by a base MLLM's performance).

3. Key Contributions

1. MSJoE Framework: A novel architecture enabling co-adaptation between the MLLM (query generation) and the Sampler (frame selection). This is the first approach to jointly evolve both components via RL for long-video understanding.
2. New Dataset (LongVideoQA): The authors collected a new dataset containing 2,800 long videos and 7,100 question-answer pairs.
   • It features multi-hop reasoning across events.
   • It includes difficulty labels to support curriculum learning and reinforcement learning.
3. State-of-the-Art Performance: The method demonstrates that a small subset of "key-frames" is sufficient for high-accuracy understanding, outperforming both dense-sampling models and heuristic-based samplers.

4. Experimental Results

The method was evaluated on four major benchmarks: VideoMME, LongVideoBench, LVBench, and MLVU.

• Accuracy Gains:
  • MSJoE achieves an 8.0% accuracy gain over the base MLLM (Qwen2.5-VL-7B).
  • It outperforms the strongest baseline (TSPO) by 1.1% on average.
  • With a 64-frame budget, it achieves 75.1% on MLVU, significantly higher than the base model (65.3%).
• Efficiency:
  • MSJoE achieves comparable or better accuracy than dense-sampling methods (like LongVU) while using far fewer frames (e.g., 32 frames vs. 1024+).
  • It outperforms heuristic methods (Q-Frame, BOLT) by 0.3–3.0 points, proving that learned sampling is superior to fixed rules.
• Ablation Studies:
  • Joint Evolution is Critical: Freezing the MLLM or the Sampler leads to significant performance drops, confirming that bidirectional adaptation is necessary.
  • Query Reasoning: Using MLLM-generated queries significantly outperforms using the raw question as the sole retrieval cue.
  • Sampler Necessity: A trained U-Net sampler significantly outperforms naive top-k selection on similarity matrices.

5. Significance and Impact

• Paradigm Shift: MSJoE moves away from static, heuristic-based frame selection toward dynamic, reasoning-guided selection. It proves that the MLLM can "teach" the sampler what to look for, and the sampler can "teach" the MLLM to focus on the most relevant visual evidence.
• Scalability: By reducing the number of frames required for high accuracy, MSJoE makes long-form video understanding feasible on standard hardware, reducing memory and compute costs.
• Generalizability: The framework is model-agnostic (demonstrated on Qwen2.5-VL) and offers a scalable direction for future multi-modal systems dealing with long-context inputs.

In conclusion, MSJoE solves the efficiency-accuracy trade-off in long-video understanding by treating frame selection not as a pre-processing step, but as an integral part of the reasoning process, optimized jointly with the language model.