DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning

DeepSport is a novel, end-to-end trained Multimodal Large Language Model that leverages a progressive two-stage training strategy involving supervised fine-tuning and agentic reinforcement learning to achieve state-of-the-art, efficient, and generalizable reasoning across diverse sports video tasks by dynamically extracting frames for active iterative thinking.

The Gist

Imagine you are watching a high-speed soccer match on TV. A player makes a quick, subtle move, and the referee blows the whistle for a foul.

If you ask a standard AI to explain what happened, it might be like asking a photographer to describe a movie. The photographer takes a few snapshots (frames) of the game, looks at them, and tries to guess the story. Because they only saw a few still images, they might miss the fast kick or the shove that happened between the snapshots. They might say, "Everything looks fine," when actually, a foul just occurred.

DeepSport is different. It's not just a photographer; it's a super-intelligent sports analyst with a remote control.

Here is how the paper explains this new technology in simple terms:

1. The Problem: The "Snapshot" Trap

Current AI models for sports are often "passive." They are fed a video, but they usually just look at a few static frames (like looking at 16 photos from a 2-hour game). They try to answer questions based on those limited photos.

• The Flaw: Sports happen fast. A foul, a goal, or a specific gymnastics move often happens in a split second between the photos the AI is looking at. If the AI misses that split second, it gets the answer wrong.

2. The Solution: "Thinking with Videos"

DeepSport changes the game. Instead of just staring at a few photos, it actively interrogates the video.

• The Analogy: Imagine you are watching a mystery movie and you miss a clue. A normal viewer keeps watching and hopes to catch it later. DeepSport is like a detective who says, "Wait, I missed something between minute 10 and minute 12. Let me rewind and look at that specific part in slow motion."
• How it works: DeepSport has a special "tool." If it's not sure about an answer, it can say, "I need to see frames 30 to 60 more closely," and it instantly grabs those specific frames to re-examine the action. It can do this multiple times, like a conversation, until it is confident.

3. How They Taught the AI (The Training)

You can't just give a smart AI a video and expect it to know the rules of 12 different sports (from Soccer to Fencing to Diving). The authors had to teach it very carefully using a two-step training camp:

• Step 1: The "Sports Curriculum" (SFT)
  Think of this like a sports school. They didn't throw the AI into the deep end immediately.

  • First, they taught it the basics: "What is a player? What is a ball? What does a goal look like?" (Fine-Grained Recognition).
  • Then, they taught it the rules: "What is a foul? What is a penalty?" (Rule & Logic).
  • Finally, they taught it the expert stuff: "Was that dive perfect? How would a coach critique this?" (Assessment & Coaching).
  • By starting simple and getting harder, the AI built a strong foundation before learning complex logic.

• Step 2: The "Coach with a Whistle" (Reinforcement Learning)
  Once the AI knew the basics, they used a technique called Agentic Reinforcement Learning.

  • Imagine a coach watching the AI play. If the AI tries to use its "rewind tool" when it didn't need to (wasting time), the coach gives it a penalty.
  • If the AI uses the tool only when it's stuck and then gets the right answer, the coach gives it a gold star.
  • This teaches the AI when to think harder and when to just give the answer, making it efficient and smart.

4. The Results: A New Champion

The authors tested DeepSport against other powerful AI models (including some from big tech companies) on a massive test of 6,700 sports questions.

• The Score: DeepSport won, achieving the highest score ever recorded for this type of task.
• The Efficiency: While other models looked at 16 frames to guess, DeepSport often figured it out by looking at fewer than 10 frames because it knew exactly which frames to zoom in on.
• The Magic: Even when they tested it on sports it had never seen before (like a new type of martial arts), it still did well. This proves it didn't just memorize the rules; it learned the physics and logic of how humans move in sports.

Summary

DeepSport is the first AI that doesn't just "watch" sports videos; it analyzes them. It acts like a human expert who can pause, rewind, and zoom in on the exact moment a play happens, allowing it to understand complex, fast-paced sports better than any previous computer model. It moves from being a passive observer to an active, thinking agent.

Technical Summary

1. Problem Statement

Sports video understanding presents unique challenges that current Multimodal Large Language Models (MLLMs) struggle to address effectively:

• High-Speed Dynamics & Temporal Complexity: Sports involve rapid movements and long temporal contexts where critical events (e.g., fouls, goals) occur in fleeting moments.
• Domain Fragmentation: Existing MLLMs are typically limited to a single sport (e.g., soccer-only), specific tasks (e.g., commentary generation only), or rely on training-free paradigms that lack deep reasoning capabilities.
• Passive Processing: Current models often process videos passively (e.g., uniformly sampling a fixed number of frames), leading to missed details and hallucinations when dealing with complex, rule-based scenarios.
• Lack of Unified Models: There is no existing end-to-end trained MLLM capable of performing diverse tasks (fine-grained recognition, rule application, coaching, commentary) across multiple sports.

2. Methodology

DeepSport introduces a paradigm shift from passive frame processing to active, iterative reasoning ("Thinking with Videos"). The framework consists of three core components:

A. The DeepSport Framework & Reasoning Paradigm

The model operates as an agentic system that can dynamically decide to extract specific video frames to answer a query.

• Action Space: The model chooses between two actions at each step:
  1. Frame Extraction Tool: choose_frames(start, end) to retrieve a specific temporal window for closer inspection.
  2. Output Answer: <answer>...</answer> to terminate the reasoning loop.
• Multi-Turn Loop: The model generates a Chain-of-Thought (CoT) trajectory, inspecting initial frames, deciding if more visual evidence is needed, retrieving new frames, and refining its reasoning until a final answer is reached.

B. Data Distillation Pipeline

To train the model, the authors curated a unified dataset from 10 existing sources covering 12 distinct sports (Soccer, Basketball, Fencing, Diving, etc.).

• Task Unification: Diverse data formats (e.g., action quality scores, commentary generation) were converted into a unified Question-Answer (QA) format across four dimensions:
  1. Fine-Grained Recognition (e.g., player ID, action sequencing).
  2. Rule & Procedural Logic (e.g., foul detection, referee decisions).
  3. Assessment & Coaching (e.g., tactical analysis, scoring prediction).
  4. Live Commentary & Reporting.
• Three-Step Filtering: A rigorous pipeline using advanced LLMs (DeepSeek, Kimi) filters the data to ensure:
  1. Accuracy: The predicted answer matches the ground truth.
  2. Logic Consistency: Reasoning steps are self-consistent and justified.
  3. Retrieval Usefulness: Tool calls actually provide new, necessary visual details (preventing redundant tool use).
• Result: A high-quality DeepSport-CoT-14K dataset for SFT and a DeepSport-RL-63K dataset for Reinforcement Learning.

C. Two-Stage Training Strategy

1. Stage 1: Sports Curriculum Supervised Fine-Tuning (SFT):
   • A "cold-start" phase to teach the model the CoT format and tool syntax.
   • Curriculum Learning: The training data is progressively structured. It starts with a high proportion of Fine-Grained Recognition tasks (building visual grounding) and gradually shifts toward complex Rule & Procedural Logic and Assessment tasks. This ensures the model masters basic perception before tackling expert-level reasoning.
2. Stage 2: Agentic Reinforcement Learning (RL):
   • Utilizes Group Relative Policy Optimization (GRPO).
   • Gated Tool-Use Reward Function: A novel reward mechanism designed to prevent "reward hacking" (blindly calling tools).
     • Class-A Samples: (Hard cases where the base model fails initially). The model is rewarded for using the tool to solve the problem.
     • Class-B Samples: (Easy cases solvable without tools). The model is rewarded for not using the tool.
     • This forces the model to learn when to retrieve frames, optimizing efficiency.

3. Key Contributions

1. First Multi-Task, Multi-Sport Trained MLLM: DeepSport is the first end-to-end model unifying diverse sports and four core reasoning capabilities, moving beyond single-sport or single-task limitations.
2. Agentic "Thinking with Videos" Paradigm: It shifts from static frame processing to dynamic, iterative interrogation, allowing the model to "re-watch" specific segments to resolve ambiguity.
3. Rigorous Data & Benchmark: Creation of a unified 78k-sample dataset (14k for SFT, 63k for RL) and a comprehensive 6.7k-sample test benchmark covering 12 sports and 4 task dimensions.
4. Novel Training Strategy: The combination of Sports Curriculum SFT (progressive difficulty) and Gated Tool-Use RL (incentivizing selective tool invocation) effectively trains compact models to perform complex reasoning.

4. Experimental Results

• State-of-the-Art (SOTA) Performance: DeepSport achieved an overall average score of 37.67 on the 6.7k test benchmark.
  • It outperformed powerful proprietary models (e.g., GPT-5) and massive open-source models (e.g., Qwen3-VL-235B).
  • It significantly surpassed its own backbone (Qwen2.5-VL-7B) by +20.69 points.
• Efficiency: DeepSport achieved these results using an average of only 9.81 frames per query, compared to the fixed 16 frames used by baselines, demonstrating superior information extraction efficiency.
• Generalization:
  • Zero-Shot Transfer: The model showed strong performance on unseen sports and general motion recognition tasks (e.g., DREAM-1K, MotionBench), indicating it learned underlying athletic mechanics rather than memorizing specific rules.
  • General Video Understanding: It maintained competitive performance on general long-video benchmarks (LVBench, LongVideoBench) while using fewer frames.
• Ablation Studies: The "Sports Curriculum SFT" was proven critical, boosting performance from 29.91 (standard SFT) to 32.48 before RL, highlighting the importance of progressive learning.

5. Significance and Future Work

• Paradigm Shift: DeepSport establishes a new standard for sports AI, proving that active, tool-augmented reasoning is superior to passive processing for complex temporal tasks.
• Scalability: The framework demonstrates that compact models (7B parameters) can rival or exceed massive models in domain-specific tasks if trained with the right data distillation and RL strategies.
• Limitations: The primary bottleneck identified is Tool Grounding Failure (42.9% of errors), where the model retrieves the wrong temporal window. Future work will focus on improving temporal precision and expanding data coverage to niche sports.

In summary, DeepSport represents a major advancement in sports video understanding by integrating agentic reinforcement learning with a curriculum-based training strategy, enabling models to actively "think" through video content to solve complex, multi-sport reasoning problems with high efficiency and accuracy.