💻 computer science

Temporal Feature Distillation for Label-Efficient Precise Event Spotting in Sports Videos

This paper proposes Temporal Feature Distillation, a semi-supervised framework combining supervised warm-up and a Transformer Gate Shift module to preserve motion-sensitive cues for precise event spotting in sports videos, achieving superior performance with significantly less labeled data than fully supervised baselines.

Original authors: Hao Xu, Xinyu Wei, Sam Wells, Sunil Aryal

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Xu, Xinyu Wei, Sam Wells, Sunil Aryal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a high-speed tennis match. The players are moving so fast that to a regular camera, one frame looks almost exactly like the next. But to a coach, there's a tiny, split-second difference: the exact moment the racket hits the ball. That split-second is the "event." Finding it is like spotting a needle in a haystack, but the haystack is moving, and the needle is invisible until it's too late.

This is the challenge of Precise Event Spotting (PES). It's not just about knowing what happened (a tennis serve); it's about knowing exactly when it happened, down to the single frame.

The Problem with the "Old Way"

Scientists have been trying to teach computers to do this using a method called self-distillation (think of it as a student learning from a teacher). The most famous teacher in this field is a system called DINO. DINO is great at learning general pictures; it teaches a computer to say, "Hey, this is a dog," even if the dog is blurry or turned sideways.

But here's the twist: The authors found that DINO is actually terrible at this specific sports job.

Why? Because DINO tries to make the computer see two slightly different pictures of the same scene as "the same thing." It smooths things out. Imagine if you tried to teach a student to find the exact moment a ball is hit by telling them, "The picture before the hit and the picture after the hit are basically the same." The student would get confused and miss the hit entirely! The authors showed that DINO's method "over-smooths" the video, blurring the very sharp boundaries between frames that are needed to spot the event. It treats the crucial motion cues as noise and ignores them.

The New Solution: A Specialized Sports Coach

To fix this, the team from Deakin University and Champion Data invented a new approach called Temporal Feature Distillation (TFD). Instead of a generic teacher, they built a sports-specific coach.

Here's how their three secret weapons work:

1. The "Time-Shifting" Glasses (Transformer Gate Shift)
Standard video AI looks at frames like a stack of photos. This new system uses a special module called Transformer Gate Shift (TGS). Think of it like giving the AI glasses that can peek slightly into the past and the future for every single frame.
Instead of just looking at the current moment, the AI can say, "Wait, this frame is connected to the one before it and the one after it." It splits the information into three groups: what happened in the past, what's happening now, and what's coming up. It then uses a "gate" to decide how much of that past or future info to mix in. This helps the AI understand the flow of the game, not just the still pictures.

2. The "Motion-Focus" Filter (Temporal Motion Augmentation)
In sports videos, the background (the crowd, the court lines) usually stays still, while the players and the ball zoom around. Standard AI often gets distracted by the huge, static background.
The team created a filter called TMA that acts like a spotlight. It automatically finds the parts of the video that are moving the most (like the ball or a player's foot) and tells the AI, "Ignore the boring static stuff; look here!" They do this by comparing frames to see what changed, then boosting the importance of those moving pixels. This ensures the AI pays attention to the action, not the scenery.

3. The "Smart Warm-Up" (Supervised Warm-up)
You can't just throw a student into a game with no rules and expect them to win. The authors found that if they tried to teach the AI using only unlabeled videos (videos without timestamps), the AI would ignore the tiny, fast-moving objects (like a tennis ball) because they look like noise.
So, they introduced a warm-up phase. First, they teach the AI using a small amount of labeled data (videos where humans marked the exact moments). This teaches the AI what to look for. Then, they slowly introduce the unlabeled data. It's like saying, "First, learn what a tennis serve looks like with a coach, and then go practice on your own." This prevents the AI from forgetting the important details while it learns from the massive pile of unlabeled videos.

The Results: Smarter with Less Help

The team tested this on four different sports datasets: Tennis, Figure Skating, and Diving. They wanted to see if their method could work well even when they didn't have many labeled examples (a "low-label" setting).

The results were impressive:

With only 10% of the labeled data: On the Figure Skating dataset (specifically the "FSPerf" split), their method improved the accuracy score (mAP) by 4.54 points compared to the next best method. That's a huge jump when you have so little data to start with.
With 80% of the labeled data: On two of the four datasets, their method performed just as well as, or even better than, methods that used 100% of the labeled data. This suggests they can get the same high-quality results with much less human effort.

What They Didn't Do

It's important to note what this paper didn't do. They didn't use extra tools like tracking the ball with a separate camera or using audio to help. They relied purely on the video frames (RGB). They also didn't claim to have solved every problem in video analysis; they specifically focused on the "precise event spotting" task in sports.

The Bottom Line

The authors suggest that by changing how the AI learns (aligning the internal "motion" features instead of just the final output) and by teaching it to focus on movement, we can build systems that are incredibly good at spotting split-second sports events, even when we don't have a million hours of human-labeled video to train them. It's a way to get the most out of the data we already have, making sports analysis faster, cheaper, and more precise.

Technical Summary: Temporal Feature Distillation for Label-Efficient Precise Event Spotting in Sports Videos

1. Problem Definition

The paper addresses Precise Event Spotting (PES) in sports videos, a task requiring the identification of the exact temporal location (frame index) and category of specific events (e.g., a tennis forehand or a soccer pass) within long, untrimmed sequences. Unlike coarse action recognition, PES demands frame-level precision to distinguish between visually similar yet semantically distinct adjacent frames.

The primary challenge identified is the label inefficiency of current state-of-the-art methods. Existing approaches rely heavily on fully supervised learning with dense frame-level annotations, which are time-consuming to produce, often require expert knowledge, and are inconsistent for ambiguous event boundaries. While Unsupervised Learning (USL) and Semi-Supervised Learning (SSL) have succeeded in general video recognition, the authors observe that directly applying existing self-distillation frameworks (such as DINO) to PES is ineffective. Standard self-distillation enforces semantic invariance across augmented views, which inadvertently suppresses the subtle motion cues and temporal boundary signals essential for distinguishing adjacent frames in sports scenarios.

2. Methodology

The authors propose a semi-supervised framework comprising three core components: a temporal-aware backbone, a specialized distillation objective, and motion-aware augmentation.

2.1 Transformer Gate Shift (TGS)

To enable temporal modeling within Vision Transformers (ViTs) without resorting to complex video-specific architectures, the authors introduce TGS, a multi-scale gated shifting module.

Mechanism: TGS is inserted at the end of each ViT encoder block. It processes input token features by applying layer normalization and reshaping them to apply temporal convolutions independently along the time dimension.
Multi-scale Gating: The module employs depthwise 1D convolutions with kernel sizes $k_d \in \{3, 5, 7\}$ (derived from $d \in \{1, 3, 5\}$ ) to capture temporal context at multiple scales.
Gating Strategy: Unlike previous gating modules that generate coarse gates for channel groups, TGS produces element-wise gates for every feature element. It splits features into "past," "future," and "static" groups. The past and future groups are temporally shifted and modulated by sigmoid-activated gates, while the static group remains unchanged. This allows fine-grained control over temporal information flow.
Fusion: Outputs from different scales are fused using learnable softmax-normalized weights before a residual connection.

2.2 Temporal Feature Distillation (TFD)

The authors propose TFD as a task-driven alternative to DINO-style self-distillation.

Core Innovation: Instead of aligning projection-head outputs (which encourages semantic invariance), TFD aligns temporally structured backbone features directly. This preserves motion-sensitive and boundary-aware cues that projection-level invariance would otherwise smooth out.
Architecture: A standard teacher-student framework is used where the teacher is an Exponential Moving Average (EMA) of the student. The student processes strongly augmented views, while the teacher processes weakly augmented views.
Supervised Warm-up: Recognizing that purely unsupervised pre-training may treat subtle motion cues (like a small ball) as noise, the authors implement a supervised warm-up. The model is first trained on labeled data to establish event-aware representations. The unsupervised distillation loss is then gradually introduced via a cosine ramp-up schedule, ensuring meaningful cues are learned before unlabeled data is utilized.
Loss Function: The objective combines a supervised cross-entropy loss (for labeled data) and a temporal alignment loss (minimizing cosine distance between teacher and student backbone features for unlabeled data).

2.3 Temporal Motion Augmentation (TMA)

Standard image augmentations (e.g., random cropping) are suboptimal for sports videos as they may remove critical small objects (e.g., the ball) or be dominated by static backgrounds.

Motion Map Generation: TMA computes a motion map using frame differences between adjacent frames.
Patch Selection: The map is divided into spatial patches. Patches are ranked by the average of their top 2% highest-motion pixels.
Augmentation: High-motion regions are preserved, while static background regions are down-weighted. Additionally, temporal masking randomly drops short temporal segments to force the model to infer missing information, enhancing temporal reasoning.

3. Key Contributions

Temporal Feature Distillation (TFD): A novel SSL objective that aligns temporally structured backbone features rather than projection heads, specifically designed to preserve the boundary sensitivity required for frame-level event spotting.
Transformer Gate Shift (TGS): A multi-scale gated shifting module tailored for ViTs that enables effective temporal information exchange without requiring dedicated video-transformer architectures.
Motion-Aware Augmentation Strategies: The introduction of TMA and a supervised warm-up schedule to prevent the suppression of subtle motion cues and stabilize training in low-label regimes.

4. Experimental Results

The method was evaluated on four fine-grained sports benchmarks: Tennis, Figure Skating (FSComp and FSPerf), and FineDiving.

Low-Label Performance: Under 10% supervision on the FSPerf dataset, the proposed method improved mAP by 4.54 points over the strongest competing approach (ASTRM), achieving 50.29 mAP compared to 45.75 mAP.
Label Efficiency: With only 80% labeled data, the method matched or surpassed the fully supervised (100% data) baseline on two of the four datasets.
Ablation Studies:
- TGS: Adding TGS to a ViT-Tiny backbone increased mAP from 10.96 to 44.77 on Tennis (+33.81 points).
- TFD: Combining TFD with TGS yielded significant gains over using TFD alone, confirming that aligning temporally enriched features is superior.
- TMA: Motion-aware augmentation provided a +4.87 mAP gain on Tennis, demonstrating the importance of focusing on dynamic regions.
Complexity: While TGS introduces an overhead (increasing parameters from 23.45M to 28.89M and GFLOPs from 644 to 1000 for ViT-Small), the authors argue this trade-off is favorable given the consistent performance improvements.

5. Significance and Claims

The paper claims that the proposed framework effectively addresses the scalability bottleneck of PES by leveraging unlabeled data without sacrificing the temporal precision required for sports analysis. The authors emphasize that their approach is task-driven, specifically modifying standard SSL paradigms to account for the unique characteristics of sports videos (rapid motion, subtle boundaries, and small critical objects).

The significance lies in demonstrating that task-aware temporal representation learning is crucial for fine-grained sports video understanding. By moving away from generic semantic invariance toward boundary-sensitive feature alignment, the method achieves state-of-the-art results in label-efficient settings, making it a viable solution for domains where dense annotation is impractical. The authors note that while clip length remains a dataset-specific hyperparameter, their default setting provides a practical balance across diverse sports scenarios.