Technical Summary: SYNCR – A Cross-Video Reasoning Benchmark with Synthetic Grounding

Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant progress in single-video understanding, including captioning dynamic scenes and answering visual queries. However, their ability to reason across multiple independent video streams remains poorly understood. Real-world applications, such as multi-angle security feeds or synchronized autonomous vehicle cameras, require systems to synthesize fragmented spatiotemporal information from disjointed sources.

Existing benchmarks for multi-video reasoning (e.g., MVU-Eval, CVBench) rely heavily on human-annotated real-world footage. This reliance creates a critical bottleneck: human annotators cannot reliably provide exact 3D relative distances, sub-second temporal offsets, camera-aligned object trajectories, or absolute kinematic variables across disjointed views. Consequently, model errors are difficult to diagnose, as failures may stem from genuine reasoning limitations, ambiguity in visual evidence, or noise in ground-truth labels. Furthermore, this annotation bottleneck limits the scalability of training data for tasks requiring dense spatial, temporal, or physical supervision.

Methodology

To address these limitations, the authors introduce SYNCR, a controlled synthetic benchmark designed to isolate core reasoning skills required for cross-video understanding. SYNCR utilizes programmatically verified grounding derived from three complementary simulation engines: Habitat, Kubric, and CLEVRER.

Framework and Diagnostic Pillars

SYNCR decomposes cross-video understanding into four diagnostic pillars, instantiated by eight specific tasks:

1. Temporal Alignment: Evaluates the ability to synchronize and sequence disjointed events.
   • Multi-Angle Synchronization: Inferring temporal offsets between unaligned videos of the same event.
   • Sequential Ordering: Reconstructing the chronological order of shuffled clips based on motion continuity.
2. Spatial Tracking: Evaluates object permanence and cross-view geometry.
   • Object Re-identification: Tracking a target entity from a reference video to a second stream.
   • Spatial Measurement: Estimating relative 3D proximity in dynamic multi-camera scenes.
3. Comparative Reasoning: Evaluates the analysis of parallel events to identify structural or kinematic differences.
   • Kinematic Comparison: Identifying the object with the highest peak velocity across videos.
   • Numerical Comparison: Aggregating discrete events (e.g., collisions) and computing numerical differences.
4. Holistic Synthesis: Evaluates the aggregation of fragmented observations into a global scene representation.
   • Object Counting: Counting distinct entities across overlapping trajectories without double-counting.
   • Route Planning: Constructing a topological map to find the shortest path between disconnected regions.

Dataset Generation

The dataset comprises 8,163 question-answer pairs grounded in 9,650 unique videos. The generation pipeline involves:

1. State Extraction: Gathering precise, frame-aware variables (e.g., 3D coordinates, velocities, collision events) directly from the simulator state.
2. Rendering: Synchronized video rendering and sequence cropping.
3. QA Formulation: Programmatic generation of multiple-choice questions and distractors based on the ground truth.

This approach ensures that the benchmark evaluates a model's capacity for visual alignment and reasoning in isolation from human annotation errors.

Key Results

The authors conducted zero-shot evaluations on a diverse suite of leading open-weight (e.g., Qwen3-VL, InternVL, LLaVA-OneVision) and closed-weight (Gemini-3, GPT-5.4) MLLMs.

• Performance Gap: There is a substantial gap between current models and human performance. The best-performing model (Gemini-3-Flash) achieved only 52.5% average accuracy, compared to an 89.5% human baseline.
• Task-Specific Challenges:
  • Models perform relatively well on Temporal Alignment, particularly Sequential Ordering (up to 100% for Gemini-3-Flash), suggesting they can recover chronological order from event continuity.
  • Models struggle significantly with Comparative Reasoning and Spatial Tracking. The Kinematic Comparison task proved most difficult, with the best open-weight model reaching only 26.0% and proprietary models at 24.0% (near random chance for a 4-way choice).
  • Holistic Synthesis shows a split: models perform reasonably on Object Counting but struggle with Route Planning, indicating difficulty in constructing coherent global spatial maps from fragmented views.
• Scaling and Training Effects:
  • Parameter scaling improves average performance (e.g., Qwen3-VL improves from 30.3% at 2B to 47.3% at 32B), but gains are family-dependent and do not resolve bottlenecks in fine-grained physical reasoning.
  • Reasoning-specialized post-training (e.g., "Thinking" checkpoints) improves Temporal Alignment but fails to reliably address fine-grained physical tracking or global spatial synthesis.
• Sim-to-Real Correlation: Exploratory analysis suggests that several SYNCR tasks (e.g., Object Re-identification, Route Planning) track model-level trends observed in real-world benchmarks, while others (e.g., Kinematic Comparison) expose reasoning gaps underrepresented by existing evaluations.

Significance and Claims

The paper positions SYNCR as a foundational "unit test" for cross-video reasoning. The authors argue that if a model cannot perform spatial-temporal reasoning on clear, simulated objects with verified ground truth, it will inevitably fail on unconstrained, noisy real-world footage.

Key contributions include:

1. A Controlled Benchmark: SYNCR provides the first large-scale, programmatically verified framework for evaluating cross-video reasoning, eliminating the ambiguity inherent in human-annotated datasets.
2. Diagnostic Precision: By isolating four specific reasoning pillars, SYNCR reveals that current MLLMs possess strong general video understanding but lack robust capabilities in precise physical tracking, cross-view synchronization, and global spatial synthesis.
3. Limitations of Scaling: The results suggest that simply increasing parameter count or applying reasoning-specialized post-training is insufficient to solve the complex physical and topological reasoning required for true cross-video intelligence.

The authors conclude that SYNCR serves as a necessary testbed for diagnosing specific failure modes in multimodal systems, guiding future research toward building more physically grounded and robust models capable of synthesizing information from multiple independent sources.