A Computer Vision Framework for Multi-Class Detection and Tracking in Soccer Broadcast Footage

This paper presents a cost-effective, single-camera computer vision framework using YOLO and ByteTrack to extract player and official tracking data from standard soccer broadcast footage, thereby democratizing advanced analytics for lower-budget teams despite ongoing challenges in ball detection.

The Gist

Imagine you are watching a soccer match on TV. For years, only the super-rich, professional clubs could afford the "superpowers" to see exactly where every player is running, how fast they are moving, and what they are doing. They used expensive GPS vests and armies of cameras. The little guys—college teams, amateur clubs, and local academies—had to guess, relying on a scout's tired eyes and a notebook.

This paper is about giving those "little guys" a superpower too, using a trick called Artificial Intelligence (AI).

Here is the simple breakdown of what the author, Daniel Tshiani, did, explained with some everyday analogies.

The Big Idea: The "Magic Eye"

The main question was: Can we just use the regular TV broadcast feed (one camera) to automatically track every player, the referee, the goalie, and the ball?

Think of it like this: Instead of hiring a team of 50 people to watch the game and write down notes, we built a robotic eye that watches the video and does the note-taking for us.

How the Robot Eye Works (The Pipeline)

The author built a three-step machine to do this:

1. The Detective (YOLO):
   First, the system looks at the video frame by frame. It uses a model called YOLO (which stands for "You Only Look Once"). Imagine a super-fast detective who scans a crowd and instantly points a finger at everyone, saying, "That's a player," "That's a goalie," "That's a ref," or "That's the ball."

   • The Twist: The author made the detective look at the video in High Definition (like zooming in with a magnifying glass) so it wouldn't miss the tiny, fast-moving ball.

2. The Memory Keeper (ByteTrack):
   Once the detective spots someone, the system needs to know that "Player #10 in the red shirt" is the same person in the next second. This is where ByteTrack comes in. It's like a bouncer at a club who gives everyone a wristband with a unique ID number. Even if the player runs behind another person (occlusion) or the camera shakes, the bouncer keeps track of who is who.

3. The Team Sorter (The "Color" Trick):
   To figure out which team is which, the system looks at the colors of the jerseys. It uses a smart tool called CLIP to understand that "this blue shirt" and "that blue shirt" belong to the same group, while "this red shirt" is the enemy. It's like sorting a pile of mixed Lego bricks by color without even looking at the shapes.

The Results: The Good, The Bad, and The Bumpy

The robot eye worked incredibly well, but it had one major blind spot.

• The Players, Referees, and Goalies: The system was a champion. It caught almost every player (99% accuracy!) and could tell the difference between a goalie and a regular player. It was like a hawk spotting a mouse in a field.
• The Ball: This was the struggle. The system was good at saying, "If I see a ball, it's probably a ball," but it missed the ball half the time.
  • Why? The ball is tiny, moves faster than a cheetah, and gets hidden by players' feet constantly. It's like trying to track a specific grain of sand in a sandstorm while someone is kicking it around.

Why This Matters (The "So What?")

Before this, if you were a small college team, you couldn't afford the fancy data that big teams like Brighton & Hove Albion use to buy cheap players and sell them for millions.

This paper proves that you don't need expensive hardware anymore. You just need a computer and a video file.

• For the little guys: It levels the playing field. A high school coach can now see exactly how their team is moving without buying $50,000 worth of equipment.
• For the world: It shows that AI can turn a simple TV broadcast into a treasure trove of data.

The Hiccups (What Needs Work)

The system isn't perfect yet.

• The "Lost ID" Problem: If a player runs off the screen and comes back, the system sometimes thinks they are a new person and gives them a new ID number. It's like meeting a friend at a party, losing sight of them, and then thinking they are a stranger when they walk back in.
• The "Sunlight" Problem: If the sun hits a player's face or changes the color of their jersey, the system sometimes gets confused about which team they are on.
• The Ball: We still need to teach the robot to be better at chasing that tiny, fast ball.

The Bottom Line

This paper is a victory lap for affordable sports science. It says, "We can now extract the same kind of smart data that only the rich clubs had, using just a single camera and some clever code." It's not perfect yet, but it's a giant leap toward making professional-level analysis available to everyone, from the Premier League down to the local park.

Technical Summary

1. Problem Statement

The paper addresses the disparity in data accessibility within the soccer industry. Professional clubs with expensive multi-camera setups or GPS tracking systems possess a competitive advantage through granular data, while lower-budget teams (colleges, academies, amateur clubs) lack these resources. The core research question is: Can Artificial Intelligence reliably detect and track soccer players, goalkeepers, referees, and the ball using only a single standard broadcast camera? The study aims to replace specialized hardware with a scalable, software-based computer vision pipeline to democratize access to performance analytics.

2. Methodology

The proposed solution is an end-to-end computer vision pipeline designed to process broadcast footage frame-by-frame.

• Data Pipeline:

  • Input: Broadcast video from the German Bundesliga, NCAA Division I, and UPSL matches.
  • Preprocessing: Videos are decomposed into frames sampled at a stride of 30 (approx. 1 frame per second) to balance temporal continuity with computational cost. Frames are resized to 1280×1280 pixels (higher than the standard 640×640) to preserve fine-grained details for small objects like the ball.
  • Augmentation: During training, augmentations included horizontal flipping, rotation, brightness changes, and random cropping.

• Detection Architecture (YOLOv8s):

  • The study utilizes YOLOv8s (You Only Look Once version 8, small variant) as the primary object detector.
  • Training Strategy: The model was fine-tuned on pretrained weights using the AdamW optimizer for 25 epochs.
  • Loss Functions: The training monitored three specific loss components:
    1. Box Loss: For precise localization.
    2. Classification Loss: To distinguish between players, referees, goalkeepers, and the ball.
    3. Distribution Focal Loss (DFL): To refine edge predictions for small, fast-moving objects.
  • Hardware Constraints: The larger YOLOv8x model was discarded due to GPU memory limitations; YOLOv8s offered the optimal efficiency/performance trade-off.

• Tracking and Analysis:

  • Tracking Algorithm: ByteTrack is used to associate detections across frames, assigning unique IDs to entities and handling occlusions.
  • Appearance Embeddings: Detected objects are cropped and encoded using the CLIP image encoder to generate 512-dimensional appearance embeddings.
  • Clustering: High-dimensional CLIP embeddings are projected into 3D space using UMAP and clustered via K-Means to segment players into teams based on jersey colors.

3. Key Contributions

• Class-Specific Benchmarking: Unlike prior studies that often treat all objects uniformly, this paper provides a granular, empirical evaluation of detection performance across four distinct classes: Players, Goalkeepers, Referees, and the Ball.
• Modern Architecture Application: It is the first known study to apply YOLOv8 specifically to soccer broadcast footage with detailed class-level metrics, establishing a new baseline for sports analytics.
• Scalable Framework: The work demonstrates a viable, low-cost pipeline that eliminates the need for multi-camera arrays or wearable sensors, making data-driven analysis accessible to non-professional entities.

4. Results and Performance

The model was evaluated using Precision, Recall, mAP@0.5, and mAP@0.5–0.95 across 49 images and 1,174 instances.

Class	Precision	Recall	mAP@0.5	mAP@0.5–0.95
Player	0.957	0.978	0.993	0.767
Referee	0.864	0.925	0.946	0.610
Goalkeeper	0.801	0.897	0.931	0.659
Ball	0.914	0.511	0.616	0.296
Overall	0.884	0.828	0.871	0.583

• High Performance: The system achieved near-saturation performance for Players (mAP@0.5: 0.993) and strong results for Referees and Goalkeepers.
• Primary Limitation: Ball detection remains the significant bottleneck. While precision is high (0.914), recall is low (0.511), indicating the model frequently misses the ball. This is attributed to the ball's small size, rapid motion, and frequent occlusion by players' feet or bodies.
• Localization: The lower mAP@0.5–0.95 scores generally reflect the difficulty of achieving tight bounding box localization in crowded, dynamic scenes.

5. Limitations and Future Work

• Ball Tracking: The current pipeline struggles with the ball. Future work suggests multi-frame reasoning, temporal smoothing, or specialized ball-focused modules.
• Identity Persistence: ByteTrack assigns random IDs that reset when players leave and re-enter the frame. Future iterations need Re-Identification (ReID) networks or sequence modeling to maintain consistent player identities for long-term statistics.
• Camera Angle Sensitivity: The model is trained exclusively on broadcast footage and performs poorly on tactical views, drone shots, or sideline handheld recordings.
• Team Classification: Unsupervised clustering via CLIP/UMAP can fail under varying lighting (e.g., direct sunlight) or when kits are similar, leading to inconsistent team labeling across frames.

6. Significance

This paper validates that single-camera computer vision is a viable alternative to expensive hardware for soccer analytics.

• Democratization: It enables colleges, academies, and amateur clubs to adopt data-driven scouting and performance analysis previously reserved for elite professional teams.
• Cost Reduction: By leveraging standard broadcast footage and open-source AI models (YOLO, ByteTrack, CLIP), the barrier to entry for advanced sports analytics is significantly lowered.
• Research Baseline: The study provides a critical, class-specific performance baseline for future research in sports computer vision, highlighting exactly where current state-of-the-art models succeed (players/officials) and where they require further innovation (ball tracking).