💻 computer science

LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs

LDDR is a training-free, plug-and-play framework that enhances video understanding in multimodal large language models by combining query-aware Linear Determinantal Point Process selection with dynamic resolution allocation to efficiently identify and prioritize informative frames under token budgets.

Original authors: Jingfeng Chen, Jiawen Qian, Wendi Deng, Yinuo Guo, Jiaqi Yu, Sicong Leng, Raghuveer Thirukovalluru, Bhuwan Dhingra

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingfeng Chen, Jiawen Qian, Wendi Deng, Yinuo Guo, Jiaqi Yu, Sicong Leng, Raghuveer Thirukovalluru, Bhuwan Dhingra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a 2-hour movie to a friend, but you only have a tiny amount of time and a very limited number of "memory slots" to describe it. If you just pick frames evenly (like taking a snapshot every 10 seconds), you might miss the exciting chase scene or the crucial clue, and you might waste time describing the same boring hallway five times.

This is the problem LDDR solves for AI video understanding. Here is how it works, broken down into simple concepts:

1. The Problem: Too Much Video, Too Little Brainpower

Multimodal Large Language Models (MLLMs) are like super-smart detectives that can read text and look at pictures. But when they watch a long video, they get overwhelmed. Videos have thousands of frames, but the AI can only "look" at a few hundred "visual tokens" (tiny pieces of image data) before it gets tired or runs out of memory.

Current methods are like a lazy tourist:

Uniform Sampling: Taking a photo every 5 seconds. This misses the action and repeats the boring parts.
Simple Relevance: Picking only the frames that look like the question. This often picks 10 photos of the same person talking, missing the context.

2. The Solution: LDDR (The Smart Curator)

The authors propose LDDR, a "training-free" tool. Think of it as a Smart Curator who doesn't need to be taught how to curate; they just use a set of clever rules to pick the best frames and decide how much detail to show.

It does two main things:

A. The "Anti-Redundancy" Filter (Linear DPP)

Imagine you are packing a suitcase for a trip, but you can only bring 10 items.

Old Way: You pick the 10 items that look most like your destination. If you are going to the beach, you might pack 10 different shades of blue swimsuits. You have no variety.
LDDR's Way: It uses a mathematical trick called Linear DPP. Think of this as a rule that says: "If you pick a blue swimsuit, you cannot pick another blue swimsuit. You must pick a hat, a towel, and sunglasses instead."

It looks at the whole video at once (not just chunks) and picks a set of frames that are both relevant to your question and different from each other.

The Magic: Usually, doing this math is slow and heavy (like trying to calculate the perfect suitcase packing for a whole fleet of ships). LDDR invented a shortcut (Linear DPP) that makes this calculation 3 times faster, allowing it to handle long videos without slowing down.

B. The "Dynamic Resolution" Budget (Group DPP)

Once the Curator picks the best 10 frames, they have to decide how much "memory" to spend on each one.

Old Way: Give every frame the same amount of detail (e.g., 100 pixels for everyone). This is wasteful. Why spend 100 pixels on a blurry background when you could spend 100 pixels on a tiny, important text on a sign?
LDDR's Way: It acts like a smart photographer.
- If a frame has a crucial clue (like a license plate or a face), it zooms in and uses high resolution (lots of memory tokens).
- If a frame is just a boring background or very similar to the previous one, it zooms out or blurs it (using fewer tokens).

This is guided by a metric called Group DPP Importance. It asks: "How much new information does this specific frame add to the group we already picked?" If the answer is "a lot," it gets a high budget. If the answer is "nothing new," it gets a low budget.

3. The Results: Faster and Smarter

The paper tested this on four different video benchmarks (ranging from short clips to hour-long videos) and various AI models.

Speed: Because of the "Linear" shortcut, LDDR is much faster than previous methods that tried to do the same math.
Accuracy: It consistently scored higher than other methods.
- In tight situations (where the AI has very little memory), it improved scores by 2.5 points.
- In generous situations, it still improved scores by 1.6 points.
Versatility: It works like a "plug-and-play" adapter. You don't need to retrain the AI model. You just feed the video through LDDR first, and then the AI watches the result. It even works on "black box" models (like GPT-5-mini) where you can't see inside the code.

Summary Analogy

Imagine you are hiring a tour guide for a 10-hour city tour, but you only have a 1-hour budget for the guide's notes.

Old Guides would write a note every 10 minutes, even if nothing happened, and they would write the same note 5 times about the same statue.
LDDR is a guide who:
1. Skips the boring parts entirely.
2. Picks only the most unique and interesting moments (no duplicate statues).
3. Writes a huge, detailed paragraph about the one moment where the magic happened, and just a tiny scribble about the boring street corners.

The result? You get a much better understanding of the whole day in the same amount of time.

Technical Summary: LDDR (Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs)

1. Problem Statement

Multimodal Large Language Models (MLLMs) face a critical bottleneck in video understanding: compressing thousands of video frames into a limited set of visual tokens under strict computational budgets. Existing approaches suffer from two primary limitations:

Inefficiency and Redundancy: Standard uniform sampling ignores query relevance, leading to the inclusion of redundant frames and the potential omission of critical evidence.
Computational Overhead: Advanced query-driven methods, such as agentic exploration or learned selectors, introduce significant training and inference costs. Furthermore, existing training-free methods based on Determinantal Point Processes (DPPs), like MDP3, rely on kernel-space inference which scales quadratically ( $O(T^2)$ ) with video length, making them prohibitive for long videos. Additionally, these methods typically allocate a uniform token budget to all selected frames, failing to account for the varying informativeness and visual complexity of individual frames.

2. Methodology: LDDR

The authors propose LDDR (Linear DPP-Based Dynamic Resolution), a training-free, plug-and-play framework that jointly optimizes frame selection and per-frame token allocation.

A. Linear Feature-Space DPP Selection

To address the quadratic complexity of standard DPPs, LDDR performs greedy Maximum A Posteriori (MAP) inference directly in the feature space rather than the kernel space.

Mechanism: Instead of constructing a dense $T \times T$ kernel matrix $L$ , LDDR utilizes a query-weighted feature matrix $\Phi = \text{diag}(\tilde{r})\hat{F}$ , where $\hat{F}$ contains normalized frame embeddings and $\tilde{r}$ encodes query relevance. The kernel is implicitly defined as $L = \Phi\Phi^\top$ .
Algorithm: The method employs a greedy selection process that maintains an orthonormal basis for the selected frames and updates residual gains for remaining frames. This reduces the computational complexity from $O(T^2d + TK^2)$ to $O(KTd)$ time and $O(Td)$ memory, where $T$ is the number of frames, $K$ is the budget, and $d$ is the feature dimension.
Outcome: This enables global frame selection across entire long videos without the need for temporal chunking, effectively capturing long-range dependencies while minimizing redundancy.

B. Group DPP (GD) Importance and Dynamic Resolution

Once a diverse candidate set is selected, LDDR introduces a Group DPP (GD) importance metric to guide adaptive token allocation.

GD Importance Score: For a selected frame $t$ , the importance $I_t$ is defined as its marginal contribution to the determinant of the group Gram matrix: $I_t = \exp(\log \det(G_S) - \log \det(G_{S \setminus \{t\}}))$ . Geometrically, this measures the squared residual norm of the frame after projecting it onto the span of other selected frames, quantifying its "residual uniqueness."
Density-Aware Scoring: To prevent over-emphasizing geometric outliers with weak query alignment, the score is modulated by a density prior based on the frame's feature magnitude: $\tilde{I}_t = I_t \cdot \rho_t^\tau$ .
Dynamic Allocation: Frames are sorted by their density-aware GD scores. A binary search determines the optimal subset size $k^*$ and token allocation $w_t$ for each frame within a total budget $C_{total}$ . Highly informative, non-redundant frames receive higher resolutions (more tokens), while less useful frames are down-scaled or pruned, all while respecting global budget constraints ( $w_{min} \le w_t \le w_{max}$ ).

3. Key Contributions

Linear DPP Selection: The authors demonstrate that performing DPP inference in feature space allows for global, query-aware frame selection with linear complexity relative to video length, eliminating the need for expensive kernel construction and temporal chunking.
GD-Guided Dynamic Resolution: The paper introduces a Group DPP importance score to quantify the marginal contribution of frames, using it to drive a training-free, adaptive token allocation strategy that maximizes information density under fixed budgets.
Comprehensive Empirical Validation: LDDR is evaluated across four benchmarks (Video-MME, LongVideoBench, LVBench, MLVU) and multiple MLLM backbones (including Qwen2.5-VL, Qwen3-VL, LLaVA-OneVision, InternVL, and GPT-5-mini).
Efficiency and Scalability: The method achieves significant runtime speedups (3 $\times$ over standard DPP baselines) while maintaining or improving accuracy, making it suitable for both open- and closed-source models.

4. Experimental Results

Performance Gains: Under budget-constrained settings, LDDR achieves an average improvement of +2.5 points over the next-best baselines. In high-budget scenarios, it still yields gains of +1.6 points.
Long-Video Superiority: The improvements are most pronounced on longer videos (e.g., 3600s subsets), where global selection effectively captures dependencies that chunked methods miss. For instance, on LongVideoBench, LDDR improved the 3600s subset score by significant margins compared to uniform sampling and chunked DPPs.
Model Agnosticism: The framework successfully improves performance across diverse backbones, including closed-source models like GPT-5-mini, without requiring access to internal model parameters or additional training.
Ablation Studies:
- Global vs. Chunked: Global selection outperforms chunked selection, particularly on long videos, confirming that local budgets can waste resources on low-information segments.
- Dynamic Resolution: Dynamic resolution allocation consistently adds ~1.2 points of improvement over fixed-resolution variants, with the largest gains observed when the frame budget is tight.
- Components: Removing either the relevance component or the diversity component of the DPP kernel significantly degrades performance, validating the necessity of balancing both.

5. Significance and Claims

The paper positions LDDR as a practical, training-free input-construction strategy that addresses the architectural necessity of efficient frame sampling in video MLLMs. The authors claim that by shifting from uniform or point-wise selection to a global, diversity-preserving, and budget-aware framework, LDDR enables MLLMs to reason more effectively over long videos without incurring the overhead of agentic loops or retraining.

The significance lies in its ability to:

Decouple selection from training: It functions as a plug-and-play preprocessing module compatible with any MLLM that supports dynamic resolution inputs.
Optimize the "Token Budget": It treats visual token allocation as a joint optimization problem, ensuring that computational resources are concentrated on the most informative and unique visual evidence.
Scale to Long Videos: By reducing DPP complexity to linear time, it makes global diversity-aware selection feasible for videos with thousands of frames, a task previously limited by quadratic computational costs.

The authors conclude that LDDR represents a robust step toward efficient long-video understanding, offering consistent gains across varying model architectures and video lengths while maintaining high inference efficiency.