Technical Summary: FineBench and FineAgent

Problem Statement

While Vision-Language Models (VLMs) have demonstrated proficiency in general video understanding tasks such as scene recognition and high-level activity summarization, they struggle with fine-grained, human-centric comprehension. Real-world applications—from assistive technologies to autonomous systems—require the ability to perceive subtle visual details, precise temporal dynamics, complex spatial relationships, and nuanced human interactions (e.g., distinguishing a deliberate sit from a fall, or identifying specific social cues).

Current benchmarks fail to adequately address this gap. Existing datasets often rely on sparsely annotated clips, focus on object-centric or broad activities, or lack the scale and density required to probe deep, temporally-grounded comprehension. There is a critical need for a standardized evaluation framework that combines long-form videos, dense question-answering (QA) coverage, and frame-level spatial/temporal grounding to measure and drive improvements in VLMs' ability to interpret subtle human behavior.

Methodology

1. FineBench: A Dense Human-Centric Benchmark

To address the evaluation gap, the authors introduce FineBench, a multiple-choice VQA dataset specifically designed for fine-grained human activity understanding.

• Scale and Density: The dataset comprises 199,420 QA pairs derived from 64 long-form videos (averaging 15 minutes each, sourced from AVA v2.2). Unlike previous datasets with sparse annotations, FineBench links questions to an average of 785 unique keyframes per video, resulting in approximately 3,100 QA pairs per video.
• Annotation Granularity: Questions are densely annotated across three core domains:
  1. Person Movement: Body posture and specific actions (e.g., standing, sitting).
  2. Person Interaction: Social cues and relationships between individuals (e.g., watching, talking).
  3. Object Manipulation: Interactions between humans and objects (e.g., carrying, holding).
  • Over 20% of questions require compositional reasoning, involving combined actions.
• Construction Process: The dataset leverages human-annotated action classes and bounding boxes from AVA v2.2.
  • Template-Based Generation: 69 structured question templates (23 for movement, 21 for manipulation, 25 for interaction) are used to prevent LLM hallucinations.
  • Spatial Referencing: A dynamic system uses bounding box coordinates to generate unambiguous spatial descriptors (e.g., "the leftmost person," "the second person from the left") to target specific individuals in multi-person scenes.
  • Distractor Strategy: Incorrect options are generated using a two-tiered approach: selecting semantically similar actions (e.g., "hand wave" vs. "hand clap") to increase difficulty, or selecting from the same broad category if no close semantic match exists.

2. FineAgent: A Modular Enhancement Framework

Motivated by the identification of specific VLM weaknesses, the authors propose FineAgent, a modular framework designed to enhance existing VLMs at inference time without retraining. It addresses two primary bottlenecks: spatial reasoning in multi-person scenes and the interpretation of nuanced human actions.

• Localizer: Utilizing the foundation model EVFSam, this module takes the video frame and question to output explicit bounding box coordinates for the subject of interest. This aids in subject disambiguation and spatial grounding.
• Descriptor: Utilizing Qwen2.5-VL (7B), this module generates textual captions summarizing the relevant frame. This provides rich semantic context to help interpret ambiguous human movements and interactions.
• Workflow: The framework first uses the Localizer to identify who and where the question refers to, then the Descriptor explains what is happening. This auxiliary information is combined with the original input to guide the base VLM's inference.

Key Results

Benchmarking Performance (FineBench)

The authors evaluated 15 VLMs (both proprietary and open-source) on FineBench:

• Proprietary vs. Open-Source: Proprietary models (e.g., GPT-5-mini, Gemini-2.0-Flash) achieved peak accuracies around 77%, significantly outperforming open-source models.
• Open-Source Performance: The best open-source model, Qwen2.5-VL (7B), achieved 68.8% average accuracy. Most other open models clustered between 55–60%.
• Action-Type Disparity: A significant performance divide was observed across categories:
  • Object Manipulation: Models performed well, with scores in the high 70s to low 80s.
  • Person Movement & Interaction: Performance dropped significantly, with many models scoring in the 50s and 60s.
• Impact of Scene Complexity: Accuracy consistently degraded as the number of people in a scene increased. For example, Qwen2.5-VL (7B) dropped from 71.7% accuracy in 2-person scenes to 53.4% in scenes with 10+ people. Smaller models (e.g., InternVL-2.5 1B) saw even steeper declines.

FineAgent Effectiveness

Integrating FineAgent yielded consistent improvements across various open-source architectures:

• Overall Gains: The framework improved average accuracy across all tested models.
• Targeted Improvements: The most significant gains were observed in the challenging Person Movement and Person Interaction categories.
  • InternVL-2.5 (1B): Improved by +8.3% overall, with a massive +14.1% boost in Person Movement accuracy.
  • mPlugOwl-3 (7B): Improved by +7.1% overall.
• Ablation Study: Both the Localizer and Descriptor contributed positively. The Localizer aided spatial disambiguation, while the Descriptor provided semantic context. Their combination yielded synergistic gains exceeding the sum of their individual contributions.

Significance and Claims

The paper claims that FineBench serves as the first densely annotated, human-centric benchmark capable of rigorously testing fine-grained video understanding, revealing that current open-source VLMs have significant room for improvement (over 20% gap) in spatial reasoning and nuanced action interpretation.

The authors assert that FineAgent demonstrates a practical approach to enhancing these capabilities. By explicitly providing spatial grounding and contextual descriptions at inference time, the framework compensates for architectural weaknesses in existing models without the need for retraining. This suggests that targeted auxiliary information can effectively bridge the gap between current VLM capabilities and the demands of fine-grained human-centric video analysis.

The work positions fine-grained understanding not merely as an academic exercise but as a fundamental capability required for VLMs to function effectively in complex real-world scenarios where subtle distinctions in human behavior carry significant meaning.