Technical Summary: Starve to Perceive

Problem: The Phenomenon of Lazy Perception

Vision-Language Models (VLMs) deployed as situated agents in high-resolution environments require active perception—the ability to dynamically decide where to look via operations like zooming, cropping, and panning. However, current training paradigms (Supervised Fine-Tuning and Reinforcement Learning) often produce models that exhibit lazy perception. These models mimic the syntactic form of perception actions (e.g., generating correct zoom commands) without functionally depending on their outputs.

The root cause is a fundamental learning asymmetry: when a coarse global view combined with strong language priors suffices for moderate accuracy, the model has no gradient-based incentive to invest in the computationally harder strategy of multi-step visual search. If a model can succeed without actively looking, it will never learn to look. Diagnostic evidence shows that replacing intermediate visual observations with blank inputs or corrupting tool calls often leaves model accuracy unchanged, indicating reliance on language priors rather than visual evidence.

Methodology: Starve to Perceive

To address this, the authors propose Starve to Perceive, a training paradigm that enforces constrained visual bandwidth. The core mechanism restricts the visual observation at every reasoning step to a tight token budget ($B$), ensuring that no single view (including the initial global glimpse) contains sufficient information to complete the task reliably. This forces active perception to become the only viable strategy for success.

The framework operates in two stages within standard post-training pipelines (SFT-then-RL):

1. Budget-Aware Visual Instruction Tuning (SFT):

   • Goal: Bootstrap the policy with a behavioral prior for active perception under token constraints.
   • Data Construction: Trajectories are generated via Teacher Model Distillation (using a strong teacher in a constrained environment with rejection sampling) and Trajectory Rewriting (converting existing high-quality tool-use datasets into budget-aware formats by downsampling views and realigning spatial coordinates).
   • Constraint: The visual observation $v_t$ is generated by a deterministic downsampling operator $D(\cdot, B)$ applied to the cropped region of the original image, ensuring the token count never exceeds $B$.

2. Reinforcement Learning via Perceptual Starvation:

   • Goal: Develop genuine functional dependence on visual observations through autonomous exploration.
   • Resolution-Conditioned Budget: The token budget $B$ scales dynamically with the native resolution of the input image ($B(X) = \text{clip}(\lfloor |X|/\gamma \rfloor, B_{\min}, B_{\max})$) to maintain consistent perceptual starvation pressure across diverse inputs.
   • Reward Signal: The framework utilizes a minimalist, sparse reward ($R=1$ for correct answer, $0$ otherwise). Because the visual budget structurally forbids resolving fine details from the global view alone, the model must perform intermediate focus actions to achieve the reward. No auxiliary losses or reward shaping are required.
   • Optimization: The policy is optimized using GRPO (Group Relative Policy Optimization) on trajectories collected within the constrained environment.

Key Contributions

• Identification of Lazy Perception: The paper identifies and traces the failure of current VLMs to genuinely rely on active visual operations to a learning asymmetry where passive shortcuts dominate.
• Starve to Perceive Paradigm: A novel training mechanism that integrates into standard SFT-then-RL pipelines as a plug-in component. It requires no architectural changes, auxiliary losses, or complex reward shaping.
• Transferability of Skills: The authors provide the first evidence that active perception skills learned under constrained training robustly transfer to unconstrained test-time settings.
• Efficiency Gains: The approach yields significant efficiency dividends, reducing training time by ~50% and inference latency by 2.7–5.1× due to the reduced number of visual tokens processed per step.

Results

The authors evaluated their model (based on Qwen2.5-VL-7B) against strong baselines (e.g., DeepEyes, Chain-of-Focus, PixelReasoner) across diverse benchmarks including HR-Bench, V* Bench, and perception-intensive reasoning tasks (MME-RealWorld-Lite, RealWorldQA).

• Unconstrained Performance: Despite being trained exclusively under visual bandwidth constraints, the model achieved a 64.59 overall score on unconstrained benchmarks, outperforming the strongest 7B-scale tool-augmented baseline (DeepEyes at 61.77) by a clear margin.
• Constrained Robustness: Under strict token budgets ($B=256$), the proposed model degraded by only 0.18 points relative to its unconstrained performance. In contrast, baselines trained without visual constraints suffered performance drops of 6.96 to 7.29 points, demonstrating that their "active perception" was superficial and collapsed when visual richness was removed.
• Ablation Studies:
  • SFT vs. RL: Budget-aware SFT alone improved performance over standard SFT, but the combination with constrained RL yielded the best results.
  • Necessity of SFT: Starting RL from a base model without SFT caused the policy to collapse into "lazy" behavior (relying on language priors), confirming that SFT provides a necessary behavioral prior.
  • Constraint Necessity: RL trained without visual constraints (even with SFT initialization) resulted in lower active perception ratios and lower final accuracy compared to the fully constrained pipeline.

Significance

The paper argues that superior visual reasoning does not emerge from input density but from the optimization pressure to extract information efficiently. By making passive observation structurally unavailable during training, the model is forced to internalize the functional dependency between local visual cues and task success. This paradigm shifts active perception from an optional enhancement to a foundational capability, enabling the deployment of robust, efficient visual agents in compute-constrained settings (e.g., on-device applications) without sacrificing performance in high-resolution environments. The work suggests that visual bandwidth should be treated as a deliberate design variable for building perceptually grounded vision-language agents.