AI Driven Soccer Analysis Using Computer Vision

This paper proposes an AI-driven soccer analysis system that combines object detection, SAM2 segmentation, and homography-based coordinate transformation to track player positioning and generate actionable tactical insights such as speed, distance covered, and heatmaps from game footage.

The Gist

Imagine you are watching a soccer game on TV. You see the players running, passing, and tackling, but you don't know exactly how far they ran, how fast they were going, or exactly where they were standing relative to the goal. Usually, only rich professional teams can afford expensive sensors and cameras to get this data.

This paper is about a group of students who built a "magic eye" using Artificial Intelligence (AI) that can turn any regular video of a soccer game into a detailed, data-rich map, without needing any special sensors.

Here is how they did it, broken down into simple steps with some fun analogies:

1. The Problem: The "Flat" Video vs. The "Real" Field

Think of a video camera like a pair of eyes. When you look at a soccer field from the stands, it looks flat and distorted because of the angle. The goal looks huge when the camera is close and tiny when it's far away.

• The Challenge: The computer needs to understand that a player running "up" the screen is actually running 50 meters down the field.
• The Solution: They needed a way to flatten the video and stretch it out to look like a perfect, top-down map (like a video game view).

2. Step One: Finding the Players (The "Spotter")

First, the computer needs to know where the players are.

• The Tool: They tested different AI "spotters" (models like YOLO and Faster R-CNN). Think of these like a security guard scanning a crowd.
• The Winner: They found that YOLOv5 was the best guard. It was fast and good at spotting players even when they were crowded together.
• The Catch: The security guard can spot a player, but if the player runs behind another person (occlusion) or the camera shakes, the guard might lose track.

3. Step Two: Tracking the Players (The "Memory Keeper")

This is where the magic happens. Once the "Spotter" finds a player, they hand them over to a special AI called SAM2 (Segment Anything Model 2).

• The Analogy: Imagine the Spotter points at a player and says, "That's Player #10!" SAM2 is like a super-attentive friend who grabs Player #10's hand and never lets go, even if they run behind a tree, get covered in mud, or the camera zooms in and out.
• Why it's cool: SAM2 doesn't just draw a box around the player; it traces their exact shape (pixel-perfect). It remembers who is who, so even if two players swap places, the computer knows who is who.

4. Step Three: Mapping the Field (The "Perspective Trick")

Now that they know where the players are, they need to translate "screen coordinates" (pixels) to "real-world coordinates" (meters).

• The Tool: They trained a custom AI to find specific landmarks on the field, like the center circle, the penalty box corners, and the halfway line.
• The Analogy: Imagine looking at a map of a city through a funhouse mirror. The lines are curved and weird. The AI finds the "corners" of the mirror (the field lines) and mathematically "un-warps" the image.
• The Result: This process is called Homography. It's like taking a crumpled piece of paper (the video) and ironing it flat so you can measure the distance between two points accurately.

5. Step Four: Sorting the Teams (The "Color Coder")

How does the computer know which team is which?

• The Trick: They didn't teach the AI to recognize faces or names. Instead, they looked at the colors of the jerseys.
• The Analogy: Imagine a bag of red and blue marbles. The computer just sorts them into two piles based on color. It's a simple "clustering" trick. If the jersey is red, they go in the "Team A" pile; if blue, "Team B."
• The Glitch: Sometimes, if the sun is glaring or shadows are weird, a red jersey might look dark, and the computer might accidentally put a player on the wrong team.

6. The Final Result: The "Coach's Dashboard"

Once all these pieces are put together, the system takes a raw video and outputs:

• A 2D top-down map of the game.
• How fast each player ran.
• How many meters they covered.
• Heatmaps showing where the team spends the most time.

Why This Matters

Previously, only teams with million-dollar budgets could get this kind of data. This system proves that you can get professional-grade insights just by using a standard camera and some clever AI. It's like giving every high school or college coach a supercomputer in their pocket, helping them make smarter decisions without spending a fortune.

In short: They built a system that watches a soccer game, remembers every player's moves, flattens the video into a map, and tells you exactly how the team played, all without needing expensive sensors.

Technical Summary

1. Problem Statement

The paper addresses the challenge of extracting actionable tactical and performance data from soccer game footage without relying on expensive sensor-based tracking systems (e.g., GPS, accelerometers) or pre-labeled datasets.

• The Gap: While professional teams use high-cost tracking tech, smaller teams often only have standard video cameras. Converting 2D camera footage into 3D real-world metrics (speed, distance, positioning) is difficult due to varying camera angles, zoom levels, lighting conditions, and the lack of ground truth data for training.
• The Goal: To create an automated, scalable computer vision pipeline that transforms raw, unlabeled video footage into a 2D top-down field representation, enabling the calculation of player speed, distance covered, and team heatmaps.

2. Methodology

The proposed system follows a multi-stage pipeline integrating object detection, segmentation, keypoint regression, and geometric transformation.

A. Player Detection and Tracking (Object Detection + SAM2)

• Initial Detection: The system evaluates several object detection models (Faster R-CNN, YOLOv5x, YOLOv8x, YOLOv11x) to identify players in the first frame.
• Segmentation & Tracking: Instead of using traditional tracking algorithms like DeepSORT, the authors use the detected bounding box centers as prompts for Meta's SAM2 (Segment Anything Model 2).
  • Why SAM2? SAM2 utilizes a streaming memory mechanism to maintain object identity across frames, handling occlusions, lighting changes, and players exiting/entering the frame more robustly than single-frame detectors.
  • Workflow: The detector initializes the process; SAM2 handles the continuous segmentation and tracking, assigning unique IDs to players.

B. Field Key Point Detection

• Custom CNN: A Convolutional Neural Network (CNN) is trained to predict the locations of 12 specific structural keypoints on the soccer field (e.g., center circle, penalty arc, midfield line).
• Preprocessing: Images are preprocessed to isolate white field lines via thresholding and morphological dilation.
• Multi-Task Output: The model predicts both the visibility (binary classification) and normalized coordinates (regression) for each keypoint.
• Loss Function: A custom loss function combines Mean Absolute Error (MAE) for visible keypoints and Binary Cross-Entropy for visibility classification, weighted to prioritize coordinate accuracy.

C. Homography and Coordinate Transformation

• Goal: To map camera coordinates to real-world field dimensions.
• Process:
  1. The CNN predicts field keypoints in the camera view.
  2. These are matched against a standardized 2D field template with known NCAA dimensions.
  3. A Homography Matrix is estimated using the Direct Linear Transformation (DLT) algorithm.
  4. This matrix transforms the segmented player masks from the camera perspective to a top-down, real-world coordinate system, regardless of camera angle or zoom.

D. Team Classification

• Unsupervised Clustering: To distinguish between the two teams without manual labeling, the system extracts the average RGB color from a small patch (5x5 pixels) within each player's bounding box.
• Algorithm: K-Means clustering ($k=2$) groups players based on jersey color similarity, effectively separating the two teams.

3. Key Contributions

1. End-to-End Unlabeled Pipeline: Demonstrates a fully automated system that derives tactical insights from raw, unlabeled video footage, removing the dependency on expensive sensors or pre-existing labeled datasets.
2. SAM2 Integration for Sports: Successfully adapts SAM2 for long-term player tracking in sports, leveraging its memory attention mechanism to handle occlusions and frame exits better than traditional trackers.
3. Custom Keypoint Regression: Develops a lightweight CNN specifically trained to identify field geometry under varying weather conditions (glare, overcast) to enable accurate homography estimation.
4. Cost-Effective Solution: Provides a scalable analytics solution accessible to teams with limited resources (only a camera required).

4. Results

The system was evaluated using footage from 10 MSOE Men's Soccer games (2024 season) under diverse conditions.

• Object Detection Performance:

  • Best Model: YOLOv5x was selected as the optimal prompt generator for SAM2.
  • Metrics: Achieved an F1-score of 0.8451, Recall of 0.7995, and Precision of 0.8963.
  • Comparison: All YOLO variants outperformed Faster R-CNN. YOLOv11x had high precision but lower recall (risk of missed detections), while YOLOv5x offered the best balance.
  • Tracking: Successfully identified and tracked 17/22 players without fine-tuning.

• Keypoint Prediction Performance:

  • Accuracy: Achieved 97.18% visibility accuracy on the test set.
  • Localization Error: Mean Absolute Error (MAE) was 0.0138 in normalized coordinates, translating to approximately 7.65 pixels on the image.

• System-Level Accuracy:

  • Homography Error: The average projection error between predicted and ground truth keypoints was 0.499 meters.
  • Distance Error: The MAE for actual player distances was 0.26 meters for predicted points.

• Limitations Observed:

  • False positives occur (e.g., mistaking ball boys or referees for players).
  • Clustering fails under extreme glare or shadows, occasionally misassigning team colors.
  • Keypoint inaccuracies can lead to slight distortions in the top-down view.

5. Significance and Future Work

• Significance: This work proves that high-level sports analytics (speed, heatmaps, tactical positioning) can be democratized using open-source AI models and standard video cameras. It bridges the gap between professional-grade data and amateur/college-level resources.
• Future Directions:
  • Re-identification: Improving the system to track players who leave and re-enter the frame by storing attributes (team, exit location) and re-prompting SAM2.
  • Generalization: Augmenting the dataset with footage from different fields and camera angles to prevent overfitting to the MSOE home field.
  • Ball Tracking: Extending the pipeline to detect the ball for analyzing game flow and possession statistics.

In conclusion, the paper presents a robust, modular framework that combines foundation models (SAM2, YOLO) with custom geometric learning to transform raw video into actionable tactical data, offering a viable alternative to expensive proprietary tracking systems.