Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

Technical Summary: Pause-and-Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

Problem Statement
Recent Vision-Language Models (VLMs) demonstrate strong perceptual and conversational capabilities but struggle with video-grounded actionable reasoning. While frontier models can identify objects, they often fail to generate concise, context-aware, and temporally consistent instructions for multi-step tasks (e.g., "What should I do next?"). Existing benchmarks primarily rely on multiple-choice questions that test recognition rather than the ability to produce free-form, grounded guidance. Furthermore, large-scale models frequently suffer from contextual drift, verbosity, and hallucinations, making them inefficient for real-time, edge-deployable assistive agents.

Methodology
The authors introduce a reasoning-centric framework comprising a new dataset, a benchmark, and a fine-tuning strategy designed to enforce "pause-and-think" behavior in compact models.

1. Dataset Construction (Pause-and-think-T):

   • Source: Derived from three large-scale video datasets (Epic-Kitchens, Assembly101, Ego4D) covering both egocentric and exocentric perspectives.
   • Pipeline: Raw videos and annotations undergo a three-stage refinement:
     • Annotation Refinement: A multimodal model (gpt-oss-120b) corrects temporal inconsistencies and organizes actions into goal-oriented phases without access to the video frames (text-only refinement).
     • Video Segmentation: Long videos are clipped into 5–20 second segments representing specific task transitions.
     • Reasoning QA Generation: Using Qwen3-VL-235B, the system generates structured conversational pairs. Crucially, responses are decomposed into two parts: a <thinking> segment (intermediate reasoning grounded in visual evidence) and an <answer> segment (concise, actionable instruction).
   • Scale: The final training set contains ~10,051 high-quality, reasoning-structured QA instances, balanced between scene understanding and goal planning.

2. Benchmark (Pause-and-think-B):

   • A curated 300-sample benchmark designed to evaluate contextual understanding and goal planning in video-driven environments. It includes a challenging subset of failure cases from frontier models to test robustness.

3. Training Protocol:

   • The authors fine-tune a compact 4B-parameter model (Qwen3-VL-4B) on the Pause-and-think-T dataset.
   • The training objective encourages the model to internally structure inference around the reasoning stage before generating the final answer, without exposing the reasoning chain during inference.
   • Experiments compare models trained with the <thinking> component (T) versus those without (NT), and against frontier models (e.g., GPT-5.2, Qwen3-VL-235B).

Key Contributions

• Pause-and-think-T & B: Introduction of a reasoning-centric training dataset (~10k samples) and a benchmark specifically targeting assistive action suggestion, moving beyond recognition-based multiple-choice to free-form, grounded guidance.
• Pareto-Efficient Performance: Demonstration that targeted reasoning supervision on a small, high-quality dataset enables a 4B model to achieve performance competitive with frontier-scale systems (e.g., Qwen3-VL-235B, GPT-5.2) while using 59× fewer parameters.
• Reduced Verbosity and Drift: Evidence that structured reasoning supervision significantly reduces contextual drift and verbosity, producing concise, user-friendly, and context-faithful instructions.

Experimental Results

• Benchmark Performance: The fine-tuned 4B model achieves 58.0% accuracy on the Pause-and-think-B benchmark, matching GPT-5.2 (58.9%) and surpassing GPT-4o (50.3%) and Qwen3-VL-235B (58.9% with 59× more parameters).
• Reasoning Impact: Models trained with the <thinking> supervision (T) consistently outperform their non-thinking (NT) counterparts (58.0% vs. 54.67% for the 4B model), particularly in goal-oriented planning tasks.
• Generalization: The model shows strong out-of-distribution (OOD) performance on external benchmarks (TempCompass, EgoThink) without specific training. Notably, it surpasses GPT-5.2 in event-order reasoning (+4.7%) and situated reasoning (+11.0%) on EgoThink.
• Qualitative Analysis: The reasoning-tuned model exhibits superior spatial grounding (left/right orientation, object-relative positioning) and robustness to occlusion compared to frontier models, which often rely on linguistic priors or hallucinate steps.
• Helpfulness Metrics: The fine-tuned model achieves the highest aggregate "helpfulness" score (0.760), driven by significantly higher conciseness (0.806) while maintaining competitive politeness.

Significance and Claims
The paper argues that data quality and reasoning supervision, rather than sheer parameter scaling, are the primary drivers for effective video-grounded assistance. By enforcing a "pause-and-think" paradigm through structured supervision, compact models can deliver actionable, visually grounded guidance suitable for real-time, edge-based deployment. The work challenges the necessity of frontier-scale compute for assistive agents, showing that a 4B model can rival systems hundreds of times larger when trained on high-signal, reasoning-structured data. The authors conclude that this approach enables models to generalize beyond training distributions and perform complex temporal reasoning without requiring Reinforcement Learning (RL) infrastructure or massive parameter counts.