ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning

1. Problem Statement

Multimodal Large Language Models (MLLMs) achieve superior visual understanding by scaling input fidelity (resolution and frame count). However, this creates a fundamental bottleneck: high spatial resolution and long temporal context are mutually exclusive under fixed computational budgets.

• The Trade-off: Reducing resolution risks losing fine-grained visual cues (e.g., small text, subtle objects) necessary for the answer. Shortening the video clip removes the temporal context required for long-horizon reasoning.
• Limitations of Existing Solutions:
  • Model-side Token Compression: Methods that prune or merge tokens after encoding (e.g., ToMe, FlashVid) discard fine-grained evidence irreversibly and often disrupt optimized inference kernels due to irregular token layouts.
  • Output-side Agentic Reasoning: Methods that iteratively retrieve or zoom in add significant multi-turn latency and still rely on an initial coarse view that may miss decisive cues.
• Core Insight: The bottleneck lies not in how representations are compressed after encoding, but in the volume of pixels the encoder receives in the first place.

2. Methodology: ResAdapt

ResAdapt proposes an Input-side Adaptation framework that learns to allocate visual budgets before encoding. It consists of three main components:

A. The Allocator (Input-side Policy)

• Architecture: A lightweight neural network (based on SmolVLM) that takes coarse visual features and the text query as input.
• Function: It predicts a per-frame allocation vector $s = (s_1, \dots, s_T)$, where each $s_t$ represents a scaling factor (resolution) or selection probability for frame $t$.
• Mechanism: The Allocator uses a Beta distribution to parameterize continuous actions, allowing for both downscaling (compression) and selective upscaling.
• Operator: The predicted allocation is realized via a pre-encoding operator (e.g., bilinear resizing or frame selection) before the video enters the main MLLM backbone. This preserves the backbone's native token interface and compatibility with optimized kernels (e.g., FlashAttention, vLLM).

B. Cost-Aware Policy Optimization (CAPO)

Training the Allocator is challenging because the operator is non-differentiable, and naive cost penalties cause the policy to collapse to the minimum budget. ResAdapt introduces CAPO to stabilize learning:

• Contextual Bandit Formulation: The problem is framed as a one-step contextual MDP where the goal is to maximize answer quality $Q(x, y)$ subject to a visual cost constraint $C(s)$.
• Asymmetric Shaping: Instead of a raw linear penalty, CAPO uses a dynamic cost pivot $\tau_{dyn}$ to shape the reward.
  • Correct & Low Cost: Receives a moderate bonus.
  • Incorrect & High Cost: Receives a strong penalty.
  • This asymmetric signal prevents the policy from collapsing to a "cheap but wrong" solution.
• Dynamic Pivot: The cost threshold adapts based on the current batch's performance, balancing exploration and exploitation.

C. Temporal Similarity Regularization

To prevent the Allocator from assigning high budgets to redundant, near-duplicate frames (a common failure mode in long videos), a temporal-similarity regularizer ($L_{sim}$) is introduced.

• It penalizes high-budget allocations on adjacent frames that are visually similar (high cosine similarity).
• This forces the policy to differentiate content, concentrating resources on information-dense moments (e.g., scene changes, text overlays) rather than uniform compression.

3. Key Contributions

1. Input-side Adaptation Framework: ResAdapt shifts the efficiency burden from post-encoding token compression to pre-encoding visual budget control, preserving the MLLM's native architecture and hardware optimizations.
2. CAPO Algorithm: A novel reinforcement learning objective that converts sparse rollout feedback into a stable, asymmetric learning signal, enabling joint optimization of accuracy and cost without hand-crafted heuristics.
3. Temporal Regularization: A mechanism that suppresses redundant high-budget allocation on similar frames, ensuring content-aware differentiation in a single forward pass.
4. Open-Loop Active Perception: The learned policy exhibits "open-loop" behavior, concentrating visual budget on information-dense content without requiring iterative feedback or modifying the backbone.

4. Experimental Results

ResAdapt was evaluated on multiple benchmarks (VideoMME, LongVideoBench, MMVU, VideoMMMU, etc.) using Qwen2.5-VL-7B and Qwen3-VL-8B backbones.

• Efficiency-Accuracy Frontier: ResAdapt consistently lies on or near the Pareto frontier. Under aggressive compression (e.g., ~10-12% visual token retention), it significantly outperforms model-side compression (ToMe, VisionZip) and fixed-resolution baselines.
  • Example: On VideoMMMU with 32 frames at ~11% retention, ResAdapt achieves 45.7% accuracy, outperforming ToMe (39.2%) and VisionZip (39.1%).
• Temporal Coverage: By saving spatial compute, ResAdapt can process 16x more frames at the same visual budget.
  • Example: Processing 128 frames with ResAdapt at ~11% retention yields higher accuracy than processing 32 frames at 100% resolution.
• Reasoning vs. Perception: The gains are most pronounced on reasoning-intensive benchmarks (e.g., VideoMMMU) where fine-grained evidence is sparse but critical.
• Latency: While the Allocator adds a small front-end overhead, the reduction in backbone tokens leads to significant end-to-end latency savings, especially for long videos (e.g., ~50-60% reduction in generation time for 128-frame inputs at 28% retention).
• Generalization: The policy trained on resizing generalizes zero-shot to frame selection tasks, confirming it learns a robust measure of visual importance.

5. Significance and Impact

• Paradigm Shift: ResAdapt demonstrates that pre-encoding adaptation is a more effective strategy for long-context video reasoning than post-encoding token pruning. It solves the "irreversible information loss" problem inherent in current methods.
• Hardware Compatibility: Unlike token merging/pruning methods that break optimized attention kernels, ResAdapt outputs a standard sequence of tokens (just fewer of them), making it compatible with existing high-performance inference stacks (vLLM, SGLang).
• Scalability: It provides a practical route to scaling video understanding to hour-long contexts by trading spatial resolution for temporal coverage, a trade-off that is dynamically managed by the learned policy.
• Active Perception: The system learns to act as an "active perceiver," focusing computational resources only where the answer is likely to be decided, mimicking human attention mechanisms.

In conclusion, ResAdapt offers a robust, trainable, and hardware-efficient solution for the long-context video reasoning bottleneck, achieving state-of-the-art efficiency-accuracy trade-offs without modifying the underlying MLLM architecture.