Technical Summary: FLAT-PACK BENCH

Problem Statement

Current Large Vision-Language Models (LVLMs) have advanced video understanding capabilities, yet existing benchmarks predominantly focus on coarse-grained tasks such as action segmentation, classification, captioning, and retrieval. These benchmarks often rely on easily identifiable entities (e.g., household objects, animals) and uncluttered scenes, failing to stress-test models on complex, real-world scenarios requiring fine-grained spatio-temporal reasoning.

Many practical applications, such as furniture assembly, cooking, or equipment repair, demand the ability to:

1. Track specific objects across long, multi-step demonstrations.
2. Localize specific temporal events and states.
3. Understand physical interactions (e.g., part mating, contact, and connection).
4. Resolve spatial references in cluttered environments where parts may look similar.

The paper argues that current LVLMs lack the necessary skills to handle these fine-grained spatio-temporal challenges, a gap not adequately addressed by existing evaluation frameworks.

Methodology: FLAT-PACK BENCH

To address this gap, the authors introduce FLAT-PACK BENCH, a novel benchmark centered on furniture assembly tasks. The benchmark is constructed using the IKEA-Manuals-at-Work (IMaW) dataset, augmented with significant new annotations to support fine-grained evaluation.

Data Construction

• Source: 50 furniture assembly videos sourced from the IMaW dataset, covering 24 unique furniture items with an average of 7 parts each.
• Annotations: The authors manually curated segmentations for 343 frames to create visual prompts, as original IMaW annotations were incomplete (limited to sub-assemblies or parts currently being connected). They also added fine-grained part-to-part connection annotations specifying which parts connect, to what, and when.
• Video Types: The benchmark utilizes two video formats:
  • Trimmed Videos: Realistic, continuous videos with text-only instruction cards removed.
  • Key-frame Videos: Concise sequences of salient frames (1 FPS) offering a simplified view of the assembly process.

Task Categories

The benchmark evaluates LVLMs on four specific fine-grained video understanding tasks, presented as multiple-choice questions (MCQs) paired with visual prompts (segmented and labeled frames):

1. Temporal Ordering (TORD): Determining the correct sequence of connection events between parts.
2. Temporal Localization (TLOC): Identifying events immediately before or after a specific assembly state shown in a visual prompt.
3. Tracking (TRACK): Recovering correct correspondences between parts in two shuffled frames (Image A and Image B) using the video context.
4. Mating (MATE): Determining whether two specific parts are physically connected in the final assembly.

Evaluation Setup

• Models: The benchmark evaluates a wide range of proprietary (e.g., GPT-5, Gemini 2.5/3.1) and open-source LVLMs (e.g., InternVL3, Qwen-VL, LLaVA variants), as well as specialized models trained for regional understanding or temporal sensitivity.
• Prompting Strategies: The study tests three visual prompt configurations:
  • Mixed-media: Video and visual prompt images provided separately.
  • Collage: Visual prompts embedded as a grid within every video frame.
  • Concat: Visual prompts concatenated as the initial frames of the video.
• Baselines: Performance is compared against random chance, frequency-based baselines, and human performance (collected from CS students and crowd-sourced workers).
• Agentic Approach: The authors also evaluate a "Temporal Video Agent" (TVA) that decomposes tasks into tracking (using SAM2) and contact-reasoning (using VLMs) sub-problems to see if tool-augmented pipelines can overcome model limitations.

Key Results

The experiments reveal that state-of-the-art LVLMs struggle significantly with fine-grained spatio-temporal reasoning:

• Performance Gap: Human performance on the benchmark is exceptionally high (94.18% accuracy), whereas the best-performing proprietary model (GPT-5) achieves only ~38% accuracy. Open-source models perform similarly, with the best (InternVL3-78B) reaching ~41%.
• Task-Specific Struggles:
  • Tracking (TRACK): Models perform particularly poorly on tracking tasks (e.g., GPT-5: 25.68%, InternVL3-78B: 42.02%), indicating a failure to maintain object identity through occlusions, camera cuts, and rotations.
  • Temporal Ordering (TORD): Models show modest improvement over chance but fail to reliably infer complex connection sequences.
  • Mating (MATE): Performance is slightly better but still far below human levels, suggesting difficulties in understanding physical contact states.
• Ineffective Use of Temporal Context: When evaluated on an "image-only" version of the benchmark (removing the video), model performance dropped significantly (by ~8.8% overall), but notably, performance on Mating and Temporal Localization tasks actually improved or stayed stable. This suggests models are not effectively utilizing the video's temporal information, instead relying on static image cues and commonsense reasoning.
• Error Analysis: Qualitative analysis of model self-explanations (via Gemini 2.5 Pro) identified primary error sources as Object Grounding (37.28%) and Spatio-Temporal Reasoning (32.45%), confirming failures in tracking objects and understanding their interactions over time.
• Agentic Limitations: The TVA agent, which attempts to solve tasks by calling external tools (SAM2 for tracking, VLM for contact), also performed poorly (11.79% accuracy). Analysis showed that the underlying tools themselves (SAM2 for tracking in the wild, VLMs for contact reasoning) are bottlenecks, failing to provide the reliable primitives needed for the agent to succeed.

Significance and Claims

The paper claims that FLAT-PACK BENCH exposes a critical limitation in current LVLMs: despite rapid progress in general video understanding, these models lack the robust, fine-grained spatio-temporal reasoning required for practical, step-by-step physical tasks.

• Benchmark Contribution: FLAT-PACK BENCH provides a rigorous testbed for evaluating the ability of models to track objects, understand physical interactions, and reason about temporal sequences in cluttered, real-world scenarios.
• Model Limitations: The results indicate that current models (both proprietary and open) are not yet capable of effectively leveraging temporal information from videos or understanding spatial interactions like physical contact.
• Tool Limitations: The failure of the agentic baseline suggests that simply decomposing tasks and using state-of-the-art vision tools (like SAM2) is insufficient; the fundamental capabilities of these tools in "in-the-wild" scenarios are also limited.
• Future Directions: The authors suggest that future work should focus on task-specific fine-tuning with synthetic simulated data, improved visual prompting techniques for regional understanding, and more sophisticated agentic pipelines that can leverage low-level signals like 3D geometry and depth.

The paper concludes that achieving the level of spatio-temporal understanding demonstrated by humans in this domain remains a significant challenge for the field of computer vision and multimodal AI.