Technical Summary: CrossView – Can Vision-Language Models Reason Across Cameras?

Problem Statement

Current Vision-Language Models (VLMs) and Video Understanding benchmarks have largely operated under a "single-camera assumption," where models are evaluated on their ability to reason over a single visual stream. While state-of-the-art models achieve near-human performance on benchmarks like VQAv2 and Video-MME, real-world systems (autonomous vehicles, security networks, robotics) inherently rely on multi-camera networks.

The authors argue that multi-camera reasoning is not merely a scaling of the single-camera problem but a fundamentally different challenge characterized by two specific hurdles:

1. Context Scaling: Processing $N$ simultaneous video streams expands the required context window, often exceeding the effective limits of long-context models.
2. Cross-View Spatial Reasoning: Models must perform spatio-temporal stitching, synthesizing discrete viewpoints into a coherent scene. This requires resolving occlusions visible only from specific cameras, selecting the most informative views, and integrating evidence across overlapping or divergent perspectives.

Existing datasets either focus on narrow perception tasks (e.g., merging views into a Bird's-Eye-View representation) or are restricted to specific domains like egocentric activity understanding. There is a distinct lack of benchmarks that force models to reason spatio-temporally across raw, simultaneous, and heterogeneous camera feeds.

Methodology

Dataset Construction: CrossView

The authors introduce CrossView, a multi-camera video question-answering (VQA) benchmark comprising 6,000 questions across four real-world domains: Autonomous Driving (nuScenes), Surveillance (MEVA), Egocentric-Exocentric interaction (Ego-Exo4D), and Robotics (AgiBot).

The dataset is constructed via a two-stage pipeline to ensure semantic correctness and avoid model hallucination:

1. Spatio-temporal Scene Graph (STSG) Construction:
   • Metadata from source datasets is consolidated. For nuScenes, LiDAR point clouds refine 3D localization; for others, 3D or bounding-box annotations are used.
   • Object-centric activities and descriptions are generated using VLMs (e.g., InternVL-3.5) where native labels are missing.
   • Pairwise spatial relationships (e.g., behind, left, right) and orientation deltas are computed between entities across camera views.
   • Event intervals $[t_{start}, t_{end}]$ are formed by grouping consecutive timestamps where an object maintains a consistent activity.
   • These elements form a chronological sequence of graph snapshots $G_t$, creating the final STSG.
2. Programmatic Question Generation:
   • A "Grounding-Target" architecture queries the STSG. The system identifies grounding events/objects, selects target events, and samples negative candidates (distractors).
   • This structured metadata is passed to GPT-5.2 to synthesize natural-language questions, ensuring grounded descriptions based on object appearance and activity rather than generic labels.

Question Categories

The benchmark includes five distinct reasoning categories, plus a specific "Camera-ID" task:

• Temporal: Reasoning about Before, After, During, or In-between relationships.
• Event Ordering: Reconstructing the chronological sequence of 3–5 distinct events.
• Spatial: Determining 3D relationships (e.g., relative position of a traffic pole to a parked car) across viewpoints.
• Counting: Aggregating unique object instances (UIDs) across the full temporal span and multiple views to handle occlusion and re-appearance.
• Summarization: Generating a holistic narrative of the scene's dynamics across all camera perspectives.
• Best Camera (Camera-ID): Identifying which specific camera view provides the most informative or persistent evidence for a given event.

Evaluation Strategy

The authors evaluated 11 VLMs, including proprietary models (GPT-5.2) and open-source families (Qwen, InternVL, Gemma).

• Sampling Strategies: Two strategies were compared: Uniform (sampling $N$ frames independently per camera) and Stitched (tiling all camera frames into a single composite image per timestep).
• Metrics: Accuracy for multiple-choice tasks (Counting, Temporal, Event Ordering, Spatial, Camera-ID) and ROUGE scores for open-ended Summarization.

Key Results

1. The Multi-Camera Reasoning Gap

Evaluation reveals a consistent and significant performance gap in multi-camera settings, even for state-of-the-art models.

• Low Accuracy: GPT-5.2, which approaches saturation on single-camera benchmarks, scores below 50% on nuScenes temporal reasoning and below 35% on camera identification in Ego-Exo4D.
• Scale Independence: The deficit persists across model families and parameter counts (from 3B to 14B+), suggesting the bottleneck is not model capacity but a structural absence of multi-camera understanding in pretraining data.
• Task Difficulty: Tasks requiring joint synthesis across cameras (Counting and Best Camera selection) yield the lowest scores (often 20–35%), significantly lower than temporal or event-ordering tasks (40–55%) derived from the same videos.

2. Scene Complexity and Camera Density

Performance degrades monotonically as scene complexity increases.

• Controlled Environments: AgiBot (robotics, fixed cameras, low clutter) yields the highest accuracies (up to 69.6% on temporal reasoning).
• Complex Environments: MEVA (wide-area surveillance, many overlapping cameras) produces the lowest scores across all categories, particularly in counting and summarization.

3. Input Strategies: Uniform vs. Stitched

• Stitched Sampling: Tiling frames into a single image improves performance on tasks requiring simultaneous spatial comparison (e.g., +10.5% on nuScenes counting, +16.6% on MEVA spatial reasoning).
• Uniform Sampling: In simpler, low-clutter environments (AgiBot), stitching can introduce visual noise and slightly degrade performance, suggesting that frame sampling strategies should be task-dependent.

4. The Necessity of Multi-View Inputs

Restricting inputs to a single "best" camera generally reduces performance compared to full multi-camera inputs.

• On nuScenes, using only the front camera dropped counting accuracy by 8.8% and spatial reasoning by 4.4% compared to the multi-camera baseline.
• This confirms that many questions in the benchmark genuinely require evidence from multiple viewpoints that cannot be inferred from a single stream.

Significance and Claims

The paper claims that CrossView is the first benchmark specifically curated to evaluate joint reasoning over 2 to 8 simultaneous camera streams across diverse real-world domains. Its primary significance lies in:

1. Exposing a Fundamental Limitation: It demonstrates that current VLMs, despite their success in single-view tasks, lack the architectural mechanisms or training data to effectively integrate evidence across multiple viewpoints.
2. Defining a New Challenge: It establishes that multi-camera reasoning involves distinct challenges (context scaling, spatio-temporal stitching, view selection) that are not solved by simply increasing model scale or context window size.
3. Benchmarking Rigor: By verifying empirically that single-view inputs fail to answer many questions (Table 5), the paper validates the necessity of multi-view integration for real-world deployment in autonomous systems and robotics.

The authors conclude that CrossView serves as a rigorous stress-test for modern VLMs, highlighting the need for new architectures, training objectives, and data pipelines that explicitly support cross-view evidence integration.