Perception Test 2025: Challenge Summary and a Unified VQA Extension

1. Problem Statement

The paper addresses the rapid evolution of Large Multimodal Models (LMMs) and the need to benchmark their capabilities beyond specialized, single-task models. While previous iterations of the Perception Test evaluated models on distinct tasks (e.g., separate tracks for object tracking, point tracking, action localization), the field has shifted toward Unified Perception.

The core problem is determining whether current State-of-the-Art (SOTA) models can handle diverse, complex spatiotemporal perception tasks through a single, unified interface (specifically, a language-based interface) rather than relying on engineered pipelines with task-specific models. The 2025 challenge aimed to test if models could generalize across tasks like tracking, localization, and reasoning without task-specific fine-tuning or architectural changes.

2. Methodology and Challenge Setup

The challenge was organized as a workshop alongside ICCV 2025, utilizing the eval.ai platform. It featured five consolidated tracks, moving away from disjointed tasks to unified approaches:

A. Unified Task Tracks (No Language Interface)

These tracks required a single model to solve two distinct tasks simultaneously:

1. Unified Object & Point Tracking: Models must track both bounding boxes (objects) and specific 2D points (points) across video frames, handling occlusions.
   • Metric: Average Jaccard (AJ), averaging Point AJ (TAP-Vid) and Object AJ (IoU).
2. Unified Action & Sound Temporal Localization: Models must identify and localize both visual actions and auditory events within a video.
   • Metric: Mean Average Precision (mAP) across action and sound classes.

B. Unified VQA Tracks (Language Interface)

These tracks used natural language queries to test perception, including a novel extension:
3. Unified Multiple-Choice Video QA (mc-VQA): The core innovation of this year.
* Novelty: Traditional non-language tasks (point tracking, temporal action localization) were reformulated as multiple-choice questions.
* Technique: The authors used visual in-painting (e.g., colored bars, flashing indicators) and temporal markers to ground spatial/temporal queries directly in the video stream. This allows VLMs to "see" the task requirements without complex decoders.
* New Question Types:
* Point Tracking: Identify a specific surface point among distractors after a "Green Flash."
* Action Ordering: Identify the correct sequence of actions.
* Object-Action Interaction: Identify objects involved in a specific temporal segment.
* Pretend Action: Identify segments where an action is "fake" or incorrect.
* Dataset: Expanded from 11,528 original 3-choice questions to include 1,842 new 5-choice questions to mitigate shortcut learning.
4. Grounded Video QA: Models answer a question by tracking the specific object(s) representing the answer throughout the video.
* Metric: HOTA (Higher Order Tracking Accuracy).
5. Hour-long Video QA: Models answer questions about very long videos (1+ hours) to test long-term memory and reasoning.
* Metric: Top-1 Accuracy.

Evaluation Protocol:

• Phases: Validation (unlimited submissions) and Test (limited to 30 submissions).
• Constraints: For joint tracks, models had to solve both tasks simultaneously. Sequential fine-tuning (e.g., training on object tracking then point tracking) was disallowed.
• Modes: Participants specified if they used fine-tuning, few-shot, or zero-shot evaluation.

3. Key Contributions

• Unified Benchmarking: Successfully merged previously separate tasks (tracking, localization) into unified tracks, forcing the community to develop generalist models rather than ensemble pipelines.
• Unified mc-VQA Extension: Introduced a novel framework that recasts complex spatiotemporal tasks (tracking, localization) as multiple-choice VQA questions using visual in-painting. This bridges the gap between visual perception and language reasoning.
• Comprehensive Evaluation: Evaluated 450 submissions from over 100 teams, providing a robust comparison of SOTA models (including Gemini, GPT-4V, Qwen, Llama 3-V) against human baselines.
• Long-Context Analysis: Included a dedicated track for hour-long videos, pushing the limits of current context windows and temporal reasoning.

4. Results

The results highlight a divergence in performance based on the presence of a language interface:

• Tracks with Language Interface (Significant Progress):

  • Grounded Video QA: Performance nearly doubled compared to 2024. The top model (SGVR@KAIST) achieved 0.499 HOTA, compared to a baseline of 0.057. This indicates models are now effectively grounding language in visual streams.
  • Hour-long Video QA: Top models achieved 0.780 accuracy (vs. 0.203 random baseline), showing strong capabilities in processing long-form content.
  • Unified mc-VQA: The top zero-shot model achieved 81.2% accuracy on the original set and 65.4% on the new, harder 5-choice set.
  • Observation: Models are rapidly approaching human baselines in language-grounded tasks.

• Tracks without Language Interface (Stagnation/Modest Gains):

  • Unified Object & Point Tracking: The top unified model achieved 0.62 AJ, which did not exceed the average of the best specialized models from 2024 (0.47 for point, 0.81 for object).
  • Unified Action & Sound Localization: The top model achieved 0.52 mAP, similar to previous specialized winners.
  • Observation: While multi-task models are capable, they struggle to outperform specialized, single-task pipelines when a language interface is absent.

• Model Performance:

  • Zero-Shot vs. Fine-Tuned: Top results were achieved by both fine-tuned ensembles (e.g., Qwen-2.5VL + GLM-4.5V) and sophisticated zero-shot approaches (e.g., Seed 1.6 Vision + Qwen-VL-Max).
  • Human Baseline: Humans scored 91.4% on the mc-VQA task, while the best models reached ~82%, indicating a remaining gap in reasoning.

5. Significance

The Perception Test 2025 serves as a critical milestone in the development of general perception models:

1. Validation of Unified Interfaces: The challenge proves that language interfaces significantly boost performance on complex perception tasks, suggesting that VLMs are becoming the preferred architecture for general perception.
2. Identification of Bottlenecks: The stagnation in non-language unified tracks (tracking/localization) reveals that current models still struggle to integrate diverse visual tasks without the scaffolding of natural language.
3. Future Direction: The results suggest that the next frontier for multimodal models is not just scaling, but improving spatiotemporal reasoning and long-context understanding to match human-level performance in tasks that do not rely on explicit language prompts.
4. Benchmark Evolution: By introducing the "Unified mc-VQA" subset, the authors provide a scalable method to evaluate tracking and localization capabilities using standard VQA metrics, simplifying the evaluation pipeline for future research.

In conclusion, while LMMs have made impressive strides in language-grounded perception (doubled performance in grounded QA), the challenge highlights that true "general perception" capable of handling diverse visual tasks in a unified, non-linguistic manner remains an open and difficult problem.