💻 computer science

Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding

This paper introduces SportMV-Bench, a new benchmark for multi-view sports video understanding, and proposes SportMV-Agent, an agentic framework that significantly improves performance by iteratively selecting relevant camera views and grounding reasoning in multi-view evidence to overcome the limitations of current single-view models.

Original authors: Kerui Chen, Jinglu Wang, Xiaoyi Zhang, Yan Lu

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kerui Chen, Jinglu Wang, Xiaoyi Zhang, Yan Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a high-stakes basketball game or a soccer match. The action is fast, players are bumping into each other, and sometimes a crucial moment gets hidden behind a crowd of bodies. If you only had one camera angle, you might miss the whole story. Maybe you see a player fall, but you can't tell if they were tripped or just slipped. That's the problem the authors of this paper are tackling: trying to teach AI to understand sports the way a human referee does—by looking at multiple cameras at once.

The Big Problem: One Eye vs. Many

The researchers found that even the smartest AI models today are like a referee who is forced to wear a blindfold on one eye. They are great at watching a single video, but when you give them a whole bundle of different camera angles for the same play, they get confused.

In their tests, they built a new playground called SportMV-Bench. It's like a giant video library containing 787 different game moments, each recorded from 2 to 4 different angles, along with 2,592 tricky questions about what happened. They split these questions into three levels of difficulty:

Perception (PAR): "Did the player actually touch the ball?" (Low-level seeing).
Rules (REI): "Was that a foul based on the rules?" (Mid-level understanding).
Judgment (ADR): "Should the player get a yellow card or a penalty?" (High-level decision making).

What the AI Got Wrong (The "Aha!" Moment)

The authors ran a bunch of experiments and discovered something surprising. They thought the AI might be failing because it didn't know the rules of basketball or soccer well enough. They were wrong.

When they gave the AI a cheat sheet with all the sports rules, it didn't get much better. They also tried making the AI "think step-by-step" (a method called Chain-of-Thought), but that actually made it perform worse. It seems that when the AI tries to reason without a clear picture, it starts making up things that didn't happen (hallucinations).

The real bottleneck? The AI can't see clearly enough.
When the researchers gave the AI perfect, human-verified descriptions of exactly what the players were doing (like "Player A kicked Player B's shin"), the AI's score jumped up by 16.47 points. This suggests the problem isn't logic; it's that the AI is struggling to spot tiny details in a blurry, fast-moving video.

Also, just throwing more camera angles at the AI didn't help. In fact, giving the AI all the views at once only improved its score by a tiny 2.1 points compared to giving it just one random view. The AI got overwhelmed by the noise. However, if they told the AI exactly which camera to look at (the "best" view), its score jumped by 10.6%. This proves the AI knows the answer is there, but it doesn't know where to look.

The Solution: The "Agentic" Detective

To fix this, the authors built a new system called SportMV-Agent. Instead of forcing the AI to stare at all the screens at once, they gave it a "manager" (an orchestrator) that acts like a curious detective.

Here's how it works:

Ask: The detective reads the question.
Choose: Instead of looking at everything, it actively picks the one camera angle that looks most promising.
Zoom In: If that angle is still blurry, it switches to a different camera.
Use Tools: It can call in special "tools" (like a magnifying glass) to check specific things, like "Did contact happen?" or "What part of the body was hit?"
Decide: It keeps gathering clues until it feels confident enough to make a call.

This approach worked like a charm. By letting the AI actively choose what to look at and when to use its tools, SportMV-Agent improved the performance by 14.46% compared to the best standard AI model.

The Takeaway

The paper suggests that to make AI good at sports, we don't need to teach it more rules or make it "think" harder. We need to teach it how to look. Just like a human referee who runs around the field to get the best angle, the AI needs to be able to switch cameras and zoom in on the details. The authors' new system, SportMV-Agent, shows that when you give the AI the power to choose its own view, it can finally solve the puzzle.

They are confident in these results because they tested them on a massive, carefully checked dataset built from real game footage and referee reports. While they haven't solved every problem in the world yet, they've proven that for sports, seeing is believing, and knowing where to look is the most important skill of all.

Technical Summary: Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding

1. Problem Definition

Sports video understanding presents unique challenges for Multimodal Large Language Models (MLLMs) that are not adequately addressed by existing benchmarks. While current MLLMs perform well on single-view video tasks, sports scenarios involve dense occlusions, rapid motion, and complex player interactions that often render a single camera angle insufficient or misleading. In real-world officiating, referees rely on multiple asynchronous camera angles to resolve ambiguities (e.g., determining if a foul occurred or if a player was out of bounds). However, existing benchmarks (e.g., SoccerNet, Sports-QA) typically provide only single-view inputs, failing to evaluate an MLLM's ability to reason across multiple perspectives.

The core problem addressed is Multi-View Sports Video Understanding: given a set of $N$ videos ( $V = \{V_1, ..., V_N\}$ ) capturing the same event from different, potentially unsynchronized viewpoints, and a natural language question $q$ , the model must aggregate evidence across views to infer the correct answer $a$ .

2. Methodology

2.1. SportMV-Bench: A New Benchmark

To address the lack of multi-view evaluation, the authors introduce SportMV-Bench, constructed from official match recordings of basketball and football.

Data Construction Pipeline: The dataset is built through a four-stage process to ensure quality and verifiability:
1. Data Collection: Videos and official referee reports are sourced from public platforms (NBA replay archive, SoccerNet). Multi-view clips are segmented using scene detection to capture the same action from different angles.
2. LLM-Based QA Proposal: An LLM generates multiple-choice questions strictly grounded in the referee reports, ensuring the "gold answer" is factual and distractors are semantically similar but incorrect.
3. Multimodal Verification: A strong MLLM (GPT-5) acts as an auditor to verify consistency between the QA pair, the referee report, and the visual evidence, filtering out hallucinations or commonsense-only questions.
4. Human Filtering: Expert annotators refine the dataset to remove ambiguous or ill-posed questions.
Statistics: The final benchmark contains 787 multi-view video bundles (2–4 views each) and 2,592 QA pairs.
Task Categories: Questions are categorized into three reasoning levels:
- Perception-Aware Recognition (PAR): Low-level action identification (e.g., tackle, block) under occlusion.
- Rule Event Aware Interpretation (REI): Mid-level reasoning requiring localization of events based on rules.
- Adjudicative Decision Reasoning (ADR): High-level normative inference to determine sanctions (e.g., penalty vs. free kick) by synthesizing cross-view evidence.

2.2. Baseline Analysis and Findings

The authors evaluated state-of-the-art MLLMs (including GPT-4.1, Qwen3-VL, and others) using a standard baseline where all views are concatenated into a single input.

Key Findings:
- View Quantity $\neq$ Quality: Providing all views jointly yielded only marginal gains over a random single view and performed significantly worse than providing the single "best" view. Redundant or occluded views introduce noise.
- Perception is the Bottleneck: Providing ground-truth fine-grained perception labels (action/contact) improved accuracy by ~16.5%, whereas Chain-of-Thought (CoT) reasoning and additional domain knowledge provided negligible or negative gains.
- Conclusion: The primary failure mode of current MLLMs is not logical reasoning or lack of domain knowledge, but the inability to autonomously select informative views and perform fine-grained visual perception.

2.3. SportMV-Agent: An Agentic Framework

Motivated by the analysis, the authors propose SportMV-Agent, an agentic framework that replaces passive processing with an iterative, active reasoning loop.

Architecture: The system consists of an Orchestrator (a strong MLLM) and a set of Perception Tools.
Iterative Loop:
1. State Maintenance: The orchestrator maintains a state containing the question, currently active views, and accumulated evidence.
2. Decision Making: At each step, the orchestrator generates a thought and an operation.
3. Active View Selection: If the current view is occluded or lacks detail, the agent triggers a VIEWSELECT operation to switch to a more informative angle.
4. Tool Execution: The agent can trigger TOOLEXEC to invoke specialized perception tools (Action Recognition, Contact Detection, Contact Part Detection) on specific views to extract fine-grained evidence.
5. Termination: The loop continues until the orchestrator reaches sufficient confidence to emit a final answer.

3. Key Results

Performance: SportMV-Agent achieved an overall accuracy of 68.31% on SportMV-Bench, representing a 14.46% relative improvement over the strongest baseline (GPT-4.1 at 59.68%).
Ablation Studies:
- Orchestrator Strength: Using a weaker orchestrator (GPT-4o) dropped performance by ~8.6%, highlighting the need for strong reasoning capabilities.
- Active View Selection: Disabling the view-switching module reduced performance by ~7.3%, confirming that dynamic view selection is critical.
- Tool Robustness: The framework is robust to the specific perception model used; replacing the tool with a stronger model yielded only marginal gains, suggesting the orchestration strategy is the primary driver of success.
- Noise Handling: Providing top-3 tool predictions rather than top-1 improved robustness, indicating the orchestrator can filter retrieval noise.

4. Significance and Claims

The paper claims that SportMV-Bench is the first benchmark to evaluate MLLMs on multi-view sports video understanding, filling a critical gap between synthetic single-view tasks and real-world officiating scenarios.

The authors assert that their work demonstrates:

Limitations of Current MLLMs: Standard models fail to exploit multi-view evidence effectively because they lack mechanisms for active view selection and are sensitive to perceptual errors.
The Role of Perception: Fine-grained visual perception is the decisive bottleneck in sports reasoning, more so than logical reasoning or domain knowledge.
Agentic Solution: An agentic framework that orchestrates active view selection and tool-based evidence gathering significantly outperforms passive, single-pass approaches.

The paper concludes that future advancements in sports video understanding require moving beyond static, single-view inputs toward dynamic, evidence-grounded reasoning systems capable of handling the complexity of real-world multi-camera sports data.