💻 computer science

Hierarchical GRU with Input-Conditioned Slot Queries for Ball Action Anticipation

This paper proposes a hierarchical GRU model enhanced with input-conditioned slot queries and frequency-reweighted Hungarian matching to achieve state-of-the-art ball action anticipation on the SoccerNet benchmark, attaining 17.91% mAP by effectively predicting future actions from a 30-second observation window.

Original authors: Parthsarthi Rawat

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Parthsarthi Rawat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a football (soccer) game on TV. The paper describes a smart computer system designed to act like a super-observant sports analyst. Its job is to watch the last 30 seconds of the game and guess what exciting ball-related action is about to happen in the next 5 seconds.

Here is how this system works, broken down into simple steps using everyday analogies:

1. The "Eyes" (Feature Extraction)

First, the system needs to see the game. It uses a pre-trained "eye" (a frozen video backbone) that has already learned what football looks like.

The Analogy: Think of this as a camera that doesn't just record pixels, but instantly recognizes shapes like "a player running," "a ball in the air," or "a goal post." It breaks the 30-second video into six 5-second chunks and turns each chunk into a list of numbers representing what it sees.

2. The "Brain" (Hierarchical Temporal Encoder)

The system processes this information in two layers, like a team of managers and a CEO.

The Local Manager (Local Transformer): Inside each 5-second chunk, a small team analyzes the specific interactions happening right now (e.g., "The player is winding up to kick").
The CEO (GRU Aggregator): A "Chief Executive" then looks at the reports from all six chunks. It doesn't just look at the last second; it remembers the flow of the whole 30 seconds.
The Analogy: Imagine a newsroom. The "Local Managers" write short summaries of what happened in each 5-minute segment. The "CEO" reads all those summaries in order to understand the full story leading up to the present moment. The CEO also has a special trick: it can decide to ignore the boring parts of the story (like early, uneventful seconds) and focus more on the exciting parts.

3. The "Guessers" (Input-Conditioned Slot Queries)

This is the most unique part of the paper. The system has 4 special "slots" (or guessers) ready to make predictions.

The Analogy: Usually, these guessers would start with a blank slate, like a student taking a test without reading the instructions. But this system is smarter. Before the guessers start, they get a "hint" based on the CEO's summary of the last 30 seconds.
Why it matters: If the last 30 seconds showed a player running fast toward the goal, the guessers get a hint that says, "Hey, a goal might be coming!" This helps them start with a better idea of what to look for, rather than guessing blindly.

4. The "Three Questions" (Decoupled Heads)

Each of the 4 guessers answers three specific questions for every potential action:

Is there an event? (Objectness): "Is something actually happening, or is it just noise?"
What is it? (Class): "Is it a tackle, a shot, a throw-in, or a goal?"
When will it happen? (Temporal Offset): "Will it happen in 1 second, 2 seconds, or 3 seconds?"

5. The "Fair Judge" (Training Tricks)

The system learns by comparing its guesses to the real answers (ground truth). The authors added two special rules to make learning fairer:

The Rare Event Bonus (Frequency-Reweighted Matching): In football, some actions (like a "Tackle") happen all the time, while others (like a "Block") are rare. Standard systems often ignore the rare ones because they are hard to find. This system gives the rare actions a "bonus point" during training, forcing the system to pay extra attention to them so it doesn't forget them.
The "Soft" Target (Gaussian Soft Targets): Instead of saying an event happens exactly at 2.0 seconds, the system is taught that 2.1 seconds is "almost right" and 3.0 seconds is "very wrong." This is like grading a test where getting the answer "close" gets you partial credit, rather than just "right or wrong." This helps the system make smoother, more accurate time predictions.

6. The Results

The system was tested on a famous football dataset called SoccerNet.

The Score: It achieved a score of 17.91% (measured by mAP, which is like a "correctness" score).
Comparison: This is very close to the current best method (which scored 18.05%), but this new system did it without needing to re-train its "eyes" from scratch, making it efficient.
Key Finding: The authors found that if they lowered the confidence threshold (letting the system guess more often), the quality dropped significantly. It's better to be quiet and sure than to shout out guesses that are likely wrong.

In Summary:
This paper presents a football prediction system that watches 30 seconds of video, summarizes the story, uses that summary to "prime" its prediction slots, and then guesses what rare or common ball actions will happen next and exactly when. It uses special math tricks to ensure it doesn't ignore rare plays and learns to be precise about timing.

Technical Summary: Hierarchical GRU with Input-Conditioned Slot Queries for Ball Action Anticipation

Problem Definition

The paper addresses the task of Ball Action Anticipation in football broadcast videos, specifically within the context of the SoccerNet Ball Action Anticipation (BAA) benchmark. The core challenge is to predict specific ball-related actions occurring in a future 5-second anticipation window based on a preceding 30-second observation window. The system must classify these actions into 10 distinct classes and determine their precise temporal location within the future window. The evaluation metric is mean Average Precision (mAP) calculated over six temporal tolerances (ranging from 1 second to infinity).

Methodology

The proposed approach employs a hierarchical architecture combining local feature encoding, global temporal aggregation, and a query-based decoding mechanism.

1. Feature Extraction

The system utilizes a frozen video backbone (EfficientNetV2-B0 with InvertedResidual3d blocks and GeM pooling), pre-trained on the SoccerNet Ball Action Spotting task.

Input Structure: The 30-second observation is divided into six 5-second windows ( $W=6$ ).
Clip Representation: Each 5-second window (at 25 fps) is further subdivided into $N_c=33$ clips, generating a sequence of 1280-dimensional feature vectors ( $F_w \in \mathbb{R}^{33 \times 1280}$ ).

2. Hierarchical Temporal Encoder

The encoding process operates on two levels:

Local Transformer: A shared 2-layer pre-LN self-attention Transformer processes each 5-second window independently. It captures intra-window temporal patterns by projecting features to $d=256$ dimensions and adding learnable clip position embeddings. Stochastic depth (DropPath) is used for regularization.
GRU Aggregator: To model long-range dependencies across the full 30-second observation, the encoded windows are reduced to 8 summary vectors via adaptive average pooling, creating a sequence of 48 steps ( $T=48$ ). A single-layer GRU processes this sequence to produce a memory state $H$ . Crucially, a learnable parameter per window allows the model to apply uniform importance weights ( $\alpha_w$ ), enabling it to downweight uninformative early segments of the observation.

3. Input-Conditioned Slot Decoder

Instead of using static, fixed queries, the decoder utilizes K=4 learnable slot embeddings that are dynamically initialized based on the observation context.

Contextual Seeding: Each slot query $q_k$ is formed by adding a learnable embedding $e_k$ to a context vector derived from the mean of the GRU memory $H$ . This allows the slots to adapt their initial state to the specific input sequence.
Decoding: A 4-layer Transformer decoder (with cross-attention and self-attention) processes these queries against the GRU memory.
Decoupled Heads: Each slot outputs three independent predictions:
1. Objectness: A binary probability ( $p_{obj}$ ) indicating if the slot corresponds to a real event.
2. Class: A softmax distribution over the 10 action classes.
3. Temporal Offset: A softmax distribution over 32 bins to predict the precise timing within the 5-second window.
  Note: Separating objectness from class prediction prevents background-class collapse.

4. Training Strategy

The training pipeline incorporates several mechanisms to handle class imbalance and temporal precision:

Frequency-Reweighted Hungarian Matching: To address the issue where rare action classes are systematically unmatched due to high classification costs, the matching cost is divided by an inverse-frequency weight ( $w_{cm}$ ). This systematically favors the assignment of slots to rare ground-truth events.
Gaussian Soft Targets: Instead of hard one-hot bin supervision for temporal offsets, the model uses Gaussian soft targets centered on the ground-truth bin. This penalizes temporally distant errors more heavily than adjacent ones, smoothing gradients.
Class-Imbalance Handling: A weighted random sampler significantly boosts the frequency of rare classes (e.g., TACKLE is sampled 40x more often).
Data Augmentation: Feature MixUp is applied, mixing input features and combining losses from source samples.
Auxiliary Head: A 1D-CNN head predicts per-frame action presence on the last window to provide auxiliary supervision.

Key Contributions

Hierarchical Architecture: The integration of a local Transformer for clip-level features and a GRU for global temporal aggregation allows the model to capture both fine-grained interactions and long-range context efficiently.
Input-Conditioned Queries: The introduction of slot queries seeded by the GRU summary (Eq. 1) replaces static queries with context-aware initializations, improving validation performance by approximately 0.8 mAP points compared to static embeddings.
Specialized Loss and Matching: The proposal of frequency-reweighted Hungarian matching specifically targets the class imbalance problem in action anticipation, ensuring rare classes are not ignored during optimization. Additionally, Gaussian soft targets for temporal binning improve the smoothness of temporal predictions.

Results

The method was evaluated on the SoccerNet Ball Action Anticipation benchmark:

Performance: The model achieved 17.91% mAP on the test server (average across all tolerances).
Comparison: This result is competitive with the organizer-provided FAANTRA baseline (18.05% mAP), despite the proposed method not employing end-to-end backbone fine-tuning.
Ablation Insights:
- Lowering the objectness threshold ( $\tau$ ) from 0.3 to 0.05 increased the number of predictions per clip five-fold but caused mAP to drop to 13.96%, confirming that low-confidence slots generate noisy outputs.
- Removing the input-conditioned query mechanism (reverting to static slots) degraded validation mAP by ~0.8 points.

Significance and Claims

The paper positions its significance in providing a robust, modular architecture for ball action anticipation that effectively handles the inherent class imbalance and temporal precision requirements of the task. The authors claim that their method achieves competitive performance on the blind challenge split without the computational cost of fine-tuning the entire video backbone. The work demonstrates that input-conditioned slot queries and frequency-aware matching are critical components for improving anticipation accuracy, particularly for underrepresented action classes. The results suggest that a carefully designed hierarchical encoder-decoder structure, combined with specific loss modifications for imbalance, is sufficient to approach state-of-the-art performance in this domain.