💻 computer science

Pixels or Positions? Benchmarking Modalities in Group Activity Recognition

This paper introduces SoccerNet-GAR, a synchronized multimodal dataset of World Cup 2022 matches, to benchmark group activity recognition and demonstrate that tracking-based models using a novel role-aware graph architecture significantly outperform video-based approaches in both accuracy and computational efficiency.

Original authors: Drishya Karki, Merey Ramazanova, Anthony Cioppa, Silvio Giancola, Bernard Ghanem

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Drishya Karki, Merey Ramazanova, Anthony Cioppa, Silvio Giancola, Bernard Ghanem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a complex soccer game. You have two ways to watch it:

The "Cinema" View (Pixels): You watch the full broadcast on TV. You see the green grass, the colorful jerseys, the sweat on the players' faces, and the crowd cheering. It's rich in detail, but it's also heavy, messy, and can get confusing if the camera zooms in too much or if players block each other.
The "Tactical Board" View (Positions): You look at a simplified digital map. You don't see the players' faces or the grass; you just see 22 dots moving around a field, connected by lines showing who is passing to whom. It's stripped of all the "fluff," leaving only the pure logic of the game.

This paper asks a simple but profound question: To understand what a team is actually doing as a group, which view is better? The rich movie or the simple map?

The Problem: We've Been Guessing

For years, researchers have tried to teach computers to recognize "Group Activities" (like a "Counter-Attack" or a "Set Piece"). They mostly used the Cinema View (video). It works okay, but it's expensive to compute and gets confused easily (e.g., if a player's jersey is the same color as the grass).

Meanwhile, the Tactical Board View (tracking data) has been largely ignored for this specific task, even though it's compact and perfectly captures how players move in relation to each other. The problem? No one had a fair way to compare them. It was like trying to compare a Ferrari to a bicycle without a track that both could drive on.

The Solution: SoccerNet-GAR

The authors built a brand new "race track" called SoccerNet-GAR.

They took 64 matches from the 2022 World Cup.
They took the raw broadcast video (the movie) and the raw tracking data (the dots on the map).
They synchronized them perfectly so that for every single action (like a "Goal" or a "Pass"), the computer sees both the video and the map at the exact same time.
They created a dataset of nearly 88,000 events to test the computers.

The Experiment: The Heavyweight vs. The Lightweight

They set up a race between two types of AI:

The Video AI (The Heavyweight): This model is like a giant, hungry monster. It eats massive amounts of video data. It has 86 million parameters (think of these as "brain cells"). It needs a lot of electricity (GPU hours) to train.
The Tracking AI (The Lightweight): This model is a nimble, efficient athlete. It only looks at the dots and lines. It has only 180,000 parameters. It's tiny compared to the video model.

The Results: The Underdog Wins Big

The results were shocking. The Lightweight Tracking AI didn't just win; it crushed the Heavyweight Video AI.

Accuracy: The Tracking AI got 77.8% accuracy. The Video AI only got 60.9%.
Efficiency: The Tracking AI used 479 times fewer "brain cells" than the Video AI.
Speed: It took 7 times less time and energy to train the Tracking AI.

The Analogy: Imagine trying to identify a specific dance move.

The Video AI is like a photographer taking 10,000 high-resolution photos of the dancer, analyzing the lighting, the fabric of the shirt, and the background. It's slow and gets confused if the dancer wears a different shirt.
The Tracking AI is like a choreographer watching the dancers' feet and arm positions on a grid. It ignores the clothes and the lighting. It instantly sees the pattern of the movement.

Why Did the "Map" Win?

The paper found that for team sports, structure matters more than appearance.

The "Role" Secret: The Tracking AI was smart. It knew that a "Goalkeeper" is always near the goal, and a "Forward" is near the opponent's goal. It connected the dots based on these tactical roles (like a coach drawing lines on a whiteboard). This gave it a huge advantage.
The Video Struggle: The Video AI got distracted. It tried to guess the action based on what the players looked like. If the lighting changed or a player was blocked by another, it got confused.

The Only Exception

There was one moment where the Video AI won: Headers.
When a player jumps to hit the ball with their head, the visual of the jump is very distinct. The simple dots on the map couldn't quite capture the "jumping" nuance as well as the video could. But for almost everything else (Passes, Tackles, Goals), the map was superior.

The Big Takeaway

This paper changes the game. It proves that for understanding team sports, you don't need a 4K camera; you need a good map.

By focusing on where players are and how they move relative to each other (positions), rather than what they look like (pixels), we can build AI that is:

Smarter (more accurate).
Faster (trains in hours, not days).
Cheaper (runs on small computers, not supercomputers).

The authors are now releasing this data and code to the world, inviting everyone to build better, lighter, and smarter sports AI.

1. Problem Statement

Group Activity Recognition (GAR) aims to identify collective behaviors arising from multiple interacting agents. While current state-of-the-art methods predominantly rely on RGB video (pixels), these approaches suffer from high computational costs, sensitivity to occlusions/editorial cuts, and an over-reliance on appearance cues rather than explicit interaction modeling.

Conversely, agent position and trajectory data (tracking) offer a compact, agent-centric representation that explicitly encodes spatial interactions and is robust to visual shifts. However, the field lacks a standardized benchmark that aligns broadcast video and tracking data for the same group activities. This absence prevents "apples-to-apples" comparisons, leaving it unclear whether pixels or positions yield superior performance for GAR in sports.

2. Key Contributions

The authors address these gaps through three primary contributions:

SoccerNet-GAR Dataset: A novel multimodal benchmark curated from 64 matches of the 2022 FIFA World Cup. It synchronizes raw broadcast video, player tracking data, and event annotations into a unified dataset containing 87,939 group activity instances across 10 football action classes (e.g., Pass, Tackle, Goal). Crucially, every event is paired with both video and tracking data.
Unified Evaluation Protocol: A standardized framework to benchmark unimodal approaches fairly. The authors compare competitive video-based classifiers against novel tracking-based classifiers leveraging Graph Neural Networks (GNNs).
Role-Aware Graph Architecture: A novel tracking-based classifier that encodes tactical structure by defining graph connectivity based on player roles (Goalkeeper, Defender, Midfielder, Forward) rather than just geometric proximity.

3. Methodology

A. Dataset Construction (SoccerNet-GAR)

Source: Data from PFF FC (Gradients Sports) covering 64 World Cup matches.
Alignment: The authors synchronized video and tracking streams using UTC timestamps. They applied strict filters to remove temporal misalignments (>10ms tolerance), missing data, and duplicate labels, resulting in a clean dataset of 87,939 events.
Structure: Each event is a 4.5-second window sampled at ~3.3 fps (16 frames).
- Video Modality: 720p broadcast footage.
- Tracking Modality: 2D player positions ( $x, y$ ) and 3D ball coordinates ( $x, y, z$ ), plus velocity and role metadata.

B. Recognition Framework

The study employs a Backbone-Neck-Head architecture for both modalities:

Video Backbone: Uses pre-trained Vision Transformers (specifically VideoMAEv2-Base) to extract spatiotemporal features.
Tracking Backbone: A Graph Neural Network approach.
- Nodes: Represent players and the ball.
- Features: 8-dimensional vectors including coordinates, velocity, ball height, and entity type.
- Graph Connectivity: The authors propose Positional Edges, where connections are defined by tactical roles (e.g., Goalkeeper $\to$ Defenders $\to$ Midfielders $\to$ Forwards) rather than simple distance. This encodes football-specific tactical structures.
- Temporal Aggregation: Frame-level embeddings are aggregated to produce a sequence-level prediction.

C. Experimental Setup

Baselines:
- Video: VideoMAEv2-Base (Fine-tuned).
- Tracking: DeepGCN (20 layers, GIN aggregation) with Positional Edges.
Metrics: Balanced Accuracy and Macro F1 score to handle class imbalance (Pass events dominate at 65.4%).

4. Key Results

The study reveals that tracking data (positions) significantly outperforms video data (pixels) in terms of both accuracy and efficiency.

Metric	Tracking (GIN + Pos. Edges)	Video (VideoMAEv2-B)	Advantage
Balanced Accuracy	77.8%	60.9%	+16.9 pp
Macro F1	57.0%	50.1%	+6.9 pp
Parameters	180K	86.3M	479× fewer
Training Time	4 GPU hours	28 GPU hours	7× faster

Detailed Findings:

Class Performance: Tracking outperforms video on 9 out of 10 classes. The gap is most significant for structurally distinct events like Goal (+56.7 pp) and High Pass (+41.7 pp). Video only slightly outperforms tracking on Header (+1.1 pp), likely due to visual cues in the ball's trajectory.
Data Scarcity: Tracking models maintain high performance even on rare classes (e.g., Goal with only 30 test samples), whereas video models collapse (16.7% accuracy).
Ablation Studies:
- Connectivity: Positional Edges (role-based) significantly outperform geometric edges (distance/KNN) and fully connected graphs. This confirms that encoding tactical structure is more critical than raw proximity.
- Temporal Aggregation: Simple MaxPool achieved the highest balanced accuracy (77.8%), outperforming learnable methods like Attention and TCN, though the latter offered slightly better Macro F1.
- Graph Layers: GIN (Graph Isomorphism Network) provided the best balance of accuracy and low variance compared to GATv2 or EdgeConv.

5. Significance and Conclusion

Paradigm Shift: The paper challenges the dominance of video-centric GAR, demonstrating that explicit spatial relationships (positions) provide a stronger signal for recognizing coordinated group activities than appearance-based cues (pixels).
Efficiency: Tracking-based models achieve superior performance with a fraction of the computational cost (479× fewer parameters, 7× faster training), making them highly suitable for real-time applications and resource-constrained environments.
Future Directions: The complementary failure modes (e.g., video better for headers, tracking better for goals) suggest that multimodal fusion is the next logical step. Additionally, the success of role-aware graphs highlights the importance of domain knowledge in architectural design.

In summary, SoccerNet-GAR establishes a new standard for evaluating GAR, proving that for sports analytics, "positions" are currently a more effective and efficient modality than "pixels."