Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports

This paper introduces CourtSI, a large-scale dataset and benchmark for evaluating spatial intelligence in vision-language models within sports scenarios, revealing significant performance gaps in existing models while demonstrating that fine-tuning on this data substantially improves accuracy and generalization.

The Gist

Imagine you are watching a high-speed tennis match on TV. You can easily tell who is winning, who is serving, and that the ball is flying fast. But could you tell a computer exactly how many meters the ball is from the player's foot, or precisely which side of the court the player is standing on relative to the net?

For a human, this is easy because we have "spatial intelligence"—an innate sense of depth, distance, and 3D space. For Artificial Intelligence (AI), specifically Vision-Language Models (VLMs) that "see" and "talk," this is incredibly hard. They are great at recognizing objects ("That's a tennis racket!") but terrible at measuring the world ("That racket is 2.4 meters away").

This paper introduces CourtSI, a new project designed to teach AI how to understand the 3D world of sports, specifically net sports like tennis, badminton, and table tennis.

Here is the breakdown of their work using simple analogies:

1. The Problem: The AI is "Flat-Earth"

Current AI models are like people who have only ever looked at paintings. They know what a ball looks like, but they don't truly understand that the ball is floating in the air or how far away it is. They struggle with:

• Distance: "Is the ball 1 meter or 10 meters away?"
• Perspective: "If I were the player, is the ball to my left or right?"
• Counting: "How many players are on the court?" (AI often gets confused by overlapping players).

2. The Solution: Building a "Digital Twin" of the Court

To fix this, the researchers didn't just feed the AI more pictures. They built a semi-automatic data engine. Think of this engine as a super-smart construction crew that builds a perfect 3D "digital twin" of every sports scene they analyze.

• The Anchor: They use the court lines as a ruler. Since a tennis court is always the exact same size, the AI can use the lines to figure out the camera's position and the scale of the world.
• The Reconstruction: The engine takes a flat 2D video frame and reconstructs it into 3D. It places the players and the ball into a virtual 3D space with real-world measurements (centimeters and meters).
• The Result: Instead of just "seeing" an image, the AI now has a blueprint of the scene with exact coordinates for everything.

3. The Dataset: The "Sports School" (CourtSI)

Using this engine, they created CourtSI, a massive textbook for AI.

• Size: It contains over 1 million question-and-answer pairs.
• Content: The questions are like a gym workout for the AI's brain, covering:
  • Counting: "How many players?"
  • Measuring: "How far is the ball from the net?"
  • Locating: "Where is the player's left foot?"
  • Reasoning: "Who is closer to the ball, Player A or Player B?"

They also created CourtSI-Bench, a strict "final exam" with 3,686 carefully checked questions to test the AI's skills.

4. The Results: The AI is Still a Rookie

The researchers tested 25 of the world's smartest AI models on this exam.

• The Gap: Even the best AI models scored significantly lower than humans. They are like a student who can memorize the dictionary but fails the math test.
• The Struggle: The models were particularly bad at measuring distances. They often guessed wildly because they couldn't translate the 2D image into 3D reality.
• The Breakthrough: When they took one specific model (Qwen3-VL) and fine-tuned it (trained it specifically) using the CourtSI dataset, its performance jumped by 23.5%. It went from a confused beginner to a competent player.

5. Why This Matters: Beyond the Scoreboard

This isn't just about winning at sports trivia.

• Real-World Application: If an AI can understand 3D space in sports, it can eventually help robots navigate a cluttered room, help self-driving cars judge distances, or assist surgeons in understanding 3D anatomy.
• Better Commentary: The researchers showed that the trained AI could write sports commentary that actually included accurate spatial details (e.g., "The player is 3 meters from the net!") rather than just vague descriptions.

The Big Picture

Think of CourtSI as a specialized gym where AI goes to learn "depth perception." Before this, AI was like a person with their eyes closed, guessing where things are. Now, thanks to this project, AI is learning to open its eyes, look at the "ruler" of the sports court, and finally understand the true 3D world around it. It's a crucial step toward building AI that can truly interact with our physical reality.

Technical Summary

Here is a detailed technical summary of the paper "Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports".

1. Problem Statement

While Vision-Language Models (VLMs) have achieved significant success in semantic understanding and 2D visual reasoning, their ability to perceive and reason about the 3D physical world (spatial intelligence) remains limited. Existing spatial intelligence benchmarks primarily focus on static scenes and rigid objects (e.g., indoor furniture), failing to address the complexities of dynamic human motion and non-rigid deformations found in real-world environments.

Sports scenarios, particularly net sports like badminton, tennis, and table tennis, present a unique and challenging testbed for spatial intelligence due to:

• High-intensity human motion: Rapid movement and complex body articulations.
• Dynamic object interactions: Fast-moving balls and changing spatial relationships.
• Metric reasoning: The need for precise distance measurement and localization in real-world units, not just relative positions.

The paper identifies a gap in current datasets and benchmarks that fail to capture these fine-grained, human-centric spatial reasoning capabilities.

2. Methodology

The authors propose a comprehensive framework consisting of a semi-automatic data engine, a large-scale dataset, and a rigorous evaluation benchmark.

A. Semi-Automatic Data Engine

To generate large-scale, metric-accurate 3D data from monocular broadcast footage, the authors developed a pipeline that leverages the fixed geometric structures of sports courts:

1. Court Annotation & Camera Calibration:
   • Annotators mark 2D keypoints on the court (corners, net height).
   • Since real-world court dimensions are fixed, these 2D-3D correspondences allow the use of a Perspective-n-Point (PnP) solver to recover metric-accurate camera intrinsics and extrinsics.
   • This establishes a unified world coordinate system anchored to the court geometry.
2. Ball Annotation:
   • Standard monocular depth estimation fails for small, fast-moving balls.
   • The authors use a ground projection estimation tool. Annotators click the ball's 2D position and its projection on the court plane. Using known camera parameters, the system analytically solves for the depth parameter ($\lambda$) to recover the 3D ball location.
3. Player Mesh Recovery:
   • PromptHMR is used to recover human meshes (SMPL-X representation) from bounding boxes.
   • To correct depth errors (e.g., feet floating or penetrating the ground), annotators manually label the height of the lowest mesh vertex relative to the court.
   • A similarity transformation is applied to the entire mesh based on this correction, ensuring the player is grounded correctly in the 3D world.

B. Dataset Construction: CourtSI

• Source: Derived from RacketVision, a dataset of professional broadcast videos for badminton, tennis, and table tennis.
• Scale: Contains 1,008,941 QA pairs generated from 52,481 images across 1,057 unique scenes.
• Taxonomy: Questions are organized into four categories:
  1. Spatial Counting: Counting players or balls.
  2. Distance Measurement: Calculating Euclidean distances (e.g., camera-to-player, player-to-ball) and heights.
  3. Localization: Predicting 3D coordinates of body parts (e.g., pelvis, foot).
  4. Relational Reasoning: Determining relative positions (e.g., "Is Player A to the left of Player B from the camera's view?").
• Features: The dataset is metric-aware (answers in meters) and human-centric (focusing on body parts and biomechanics).

C. Evaluation Benchmark: CourtSI-Bench

• A high-quality subset of 3,686 QA pairs sampled from 1,988 images (382 scenes).
• Quality Control: Undergoes rigorous human verification. Annotators review reconstructed scenes to flag and remove QA pairs with reconstruction failures or ambiguities.
• Metrics:
  • Accuracy: For counting and relational reasoning.
  • Threshold Mean Relative Accuracy (T-MRA): For numerical tasks, allowing a tolerance threshold (15cm) to account for minor estimation errors.

3. Key Contributions

1. CourtSI & CourtSI-Bench: The first large-scale spatial intelligence dataset and benchmark specifically tailored for sports scenarios, shifting focus from static objects to dynamic, human-centric spatial reasoning.
2. Semi-Automatic Data Engine: A novel pipeline that exploits court geometry to achieve centimeter-level accuracy in 3D scene reconstruction from monocular broadcast videos, enabling scalable data curation without expensive multi-view setups.
3. Comprehensive Evaluation & Analysis:
   • Evaluated 25 state-of-the-art VLMs (both proprietary and open-source).
   • Demonstrated that existing spatial benchmarks do not generalize well to sports.
   • Showed that fine-tuning on CourtSI significantly improves performance.
   • Validated transferability via CourtSI-Ext (an unseen sport: pickleball) and spatial-aware commentary generation.

4. Results

A. Baseline Performance (Human-AI Gap)

• Human Performance: Humans achieve high accuracy but still struggle with precise metric distance estimation, highlighting the difficulty of the task.
• VLM Performance:
  • Even the strongest proprietary models (e.g., GPT-5.2, Gemini-3-Pro) lag significantly behind humans, particularly in distance measurement and localization.
  • Open-source models generally perform poorly (often <40% overall accuracy).
  • Models fine-tuned on existing spatial benchmarks (e.g., SpaceR, VST) show limited generalization to CourtSI-Bench, indicating that sports scenarios introduce unique challenges not captured by current datasets.

B. Impact of Fine-Tuning

• Fine-tuning Qwen3-VL-8B on CourtSI resulted in a 23.5 percentage point increase in overall accuracy on CourtSI-Bench.
• The most significant gains were observed in Distance Measurement (+25.4 pp) and Relational Reasoning.
• The fine-tuned model demonstrated strong cross-sport generalization on CourtSI-Ext (pickleball), improving accuracy by 13.2 pp over its base model, proving the transferability of the learned spatial reasoning.

C. Downstream Application

• Spatial-Aware Commentary: When prompted to generate sports commentary incorporating spatial relationships, the fine-tuned model produced significantly more accurate spatial descriptions while maintaining high linguistic quality, as validated by user studies.

D. Error Analysis

• Perspective Ambiguity: Models struggle significantly when 3D distances are large but 2D projected distances are small (high perspective distortion).
• Localization Errors: Failures often stem from incorrect estimation of 3D positions relative to court geometry or misinterpreting ego-centric vs. allo-centric perspectives.

5. Significance

• Advancing Spatial Intelligence: CourtSI provides a scalable pathway to bridge the gap between current VLM capabilities and the requirements for interacting with the physical world, specifically in dynamic, human-centric environments.
• New Benchmark Standard: It establishes a rigorous standard for evaluating metric spatial reasoning, moving beyond simple 2D detection or semantic captioning.
• Practical Applications: The work has immediate implications for automated sports broadcasting (generating spatially accurate commentary), tactical analysis, and robotics (where understanding human motion in 3D space is critical).
• Methodological Insight: The paper demonstrates that leveraging domain-specific geometric priors (court lines) is a highly effective strategy for recovering metric 3D information from 2D data, a technique applicable to other structured environments.

In conclusion, the paper argues that sports are a critical frontier for spatial intelligence research. By providing the data, tools, and benchmarks, the authors enable the community to develop VLMs that can truly "see" and reason about the 3D world in a way that approaches human-level understanding.