DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding

Based on the provided paper, here is a detailed technical summary of DynFrame.

Problem Statement

Recent Multimodal Large Language Models (MLLMs) have advanced video understanding by coupling step-by-step reasoning with on-demand visual evidence retrieval. However, existing "thinking-with-video" systems suffer from two structural gaps:

1. Fixed Sampling Density: While models can decide where to look (temporal window), the frame rate (sampling density) within that window is typically a fixed system hyperparameter. Consequently, recovering fine-grained evidence often requires repeated retrieval calls, which inflates inference context length and complicates training.
2. Credit Assignment Imbalance: Current reinforcement learning (RL) approaches often optimize the entire generation trajectory with a single advantage signal. This entangles the credit for the "where to look" decision (retrieval tokens) with the "how to answer" decision (reasoning tokens), leading to suboptimal learning when one component is correct and the other is not.

Methodology

The authors propose DynFrame, an end-to-end trainable framework that unifies adaptive frame retrieval with dynamically interleaved vision-language reasoning. The core components are:

1. Tokenized Retrieval Interface
DynFrame treats frame-rate adaptation as a learnable decision within the autoregressive generation trajectory. Instead of external tool calls, the model emits native tokens to specify the retrieval command:

• <span>ts–te</span>: Defines the temporal window.
• <fps>f</fps>: Defines the sampling frame rate.
  This allows the model to acquire multi-granularity evidence (dense frames for motion, sparse for semantics) in a single retrieval step.

2. Dynamic Frame Injection
Upon generating the closing </fps> token, the system triggers an on-the-fly injection mechanism. It extracts $N$ frames from the specified window at the predicted frame rate, encodes them via a frozen vision encoder, and appends timestamped visual tokens to the generation buffer. The decoder then re-prefills KV states to allow the post-injection reasoning to attend bidirectionally to the newly retrieved frames.

3. Segment-Decoupled GRPO (SD-GRPO)
To address the credit assignment imbalance, the authors introduce SD-GRPO, a reinforcement learning algorithm that splits the rollout at the retrieval boundary ($T_{fps}$):

• Sampling Segment ($t \le T_{fps}$): Receives a specific reward ($R_{samp}$) based on the overlap of the predicted temporal span with the ground truth and the accuracy of the predicted FPS.
• Reasoning Segment ($t > T_{fps}$): Receives an answer-specific reward ($R_{ans}$) based on task correctness (e.g., exact match or IoU).
  Token-level advantages are computed separately for these segments, ensuring that the model is explicitly rewarded for making good retrieval decisions independent of the final answer quality, and vice versa.

4. Data Curation
The framework is trained on two curated datasets:

• DM-CoT-74k: A supervised fine-tuning (SFT) dataset with interleaved retrieval commands and grounded reasoning.
• DM-RL-45k: A reinforcement learning dataset with ground-truth temporal spans and FPS targets, used to compute the decoupled rewards.

Key Contributions

1. Native Tokenized Adaptation: DynFrame transforms adaptive temporal evidence acquisition from an externally scheduled tool operation into a model-native reasoning capability by emitting temporal span and sampling density as native tokens.
2. Segment-Decoupled Optimization: The introduction of SD-GRPO allows for targeted credit assignment, separating the optimization of the sampling decision from the answer reasoning, which stabilizes the co-adaptation of retrieval and reasoning.
3. Curated Datasets: The creation of DM-CoT-74k and DM-RL-45k to support the training of robust, native tokenized adaptive retrieval.

Experimental Results

DynFrame was evaluated on six benchmarks spanning temporal grounding, grounded VideoQA, and long-form video understanding (NExT-GQA, Charades-STA, ActivityNet-MR, Video-MME, MLVU, LVBench).

• Performance: DynFrame-8B achieved state-of-the-art results on most metrics, outperforming strong 7B–8B baselines (including Video-o3, VITAL, and VideoChat-R1.5). DynFrame-4B was competitive with these larger baselines despite having half the parameters.
• Efficiency: Unlike multi-turn retrieval methods that significantly expand context length, DynFrame achieves high accuracy with a single retrieval round, maintaining a more manageable inference context.
• Ablation Studies:
  • Removing the segment decoupling (using vanilla GRPO) resulted in lower performance, confirming the necessity of separate credit assignment.
  • Replacing dynamic FPS prediction with a fixed rate degraded performance, validating the need for adaptive density.
  • The model demonstrated robustness to sparse initial sampling ($f_1$), successfully recovering missing evidence through dynamic retrieval.

Significance

The paper claims that DynFrame represents a shift from rigid, system-defined visual passes to flexible, model-driven evidence acquisition. By learning to decide both where to look and how densely to sample, the model can acquire task-adaptive, multi-granularity evidence efficiently. The proposed SD-GRPO framework provides a mechanism to train these complex retrieval-reasoning behaviors without entangling the learning signals for different stages of the generation process. This approach enables smaller models (4B) to compete with larger baselines and sets new performance standards for long-form video understanding.