Technical Summary: OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization

Problem Definition: Point-Supervised Online TAL (POTAL)

The paper addresses a critical gap in Temporal Action Localization (TAL): the intersection of label efficiency and online inference constraints. Existing TAL methods typically rely on dense temporal boundary annotations and offline access to full video sequences, which limits scalability and prevents deployment in streaming scenarios (e.g., live sports analytics, surgical assistance, robotics). Conversely, Online TAL (OnTAL) methods operate in streaming settings but require full segment-level supervision, which is costly to obtain.

The authors introduce Point-Supervised Online TAL (POTAL), a novel task setting where:

1. Supervision is sparse: Training utilizes only a single temporal point (timestamp) and class label per action instance, rather than full start/end boundaries.
2. Inference is strictly online: The model must process streaming video frame-by-frame, making predictions without access to future context or the ability to revise past decisions.

The core challenge is that training an online model with only point labels is inherently difficult due to the lack of boundary cues and the absence of future context required to define action extents.

Methodology: The OnPoint Framework

To solve POTAL, the authors propose OnPoint, an offline-to-online multi-level distillation framework. The system employs a point-supervised offline teacher (which has access to the full video) to guide a strictly online student (which processes streaming windows). The teacher generates structured supervision signals that the student learns to mimic using only past and current frames.

The framework consists of four key technical components:

1. Multi-Level Distillation Signals

The teacher transfers knowledge through three complementary signals:

• Instance-Level Pseudo-Distillation: The offline teacher generates pseudo-segment ground truth (start/end boundaries) from sparse point labels. These pseudo-labels supervise the student's anchor-based instance prediction module, providing structured boundary guidance that the student cannot infer from points alone.
• Class-Activation Sub-Sequence (CASS) Distillation: To enhance temporal representation, the student aligns its frame-wise predictions within a sliding window with the teacher's Class Activation Sequence (CAS). This dense, frame-level supervision strengthens the backbone's ability to capture subtle action transitions without future context.
• Anticipatory Window-Level Distillation: To compensate for the missing future context, the teacher's CAS is used to predict the likelihood of actions occurring in the next window. This "future knowledge" is distilled into the student's anticipation head, guiding it toward imminent action boundaries.

2. Actionness-Guided Attention Calibration

Standard anchor-based decoders often lack explicit action priors. OnPoint introduces a mechanism to calibrate the attention weights in the transformer decoder. It utilizes the actionness scores derived from the CASS predictor (specifically, the inverse of the background probability) to bias the attention logits. This forces the decoder to emphasize frames with high action likelihood and suppress background noise, improving the quality of anchor features.

3. Auxiliary Anchor-Level Point Supervision

To mitigate the risk of error propagation from imperfect pseudo-labels generated by the teacher, the framework directly incorporates the original ground-truth point annotations as an auxiliary supervisory signal. An Anchor-Level Point Prediction module classifies whether an anchor contains a point and regresses the distance between the anchor center and the point. This provides a weak but reliable signal that stabilizes training.

4. Training and Inference

• Training: The online student is trained using a weighted combination of four losses: instance-level loss (from pseudo-labels), CASS loss, anticipation loss, and point prediction loss.
• Inference: The offline teacher is discarded. The online student processes sliding windows, generates proposals via the anchor decoder, and applies Online Non-Maximum Suppression (ONMS) to output action segments immediately upon detection.

Key Contributions

1. POTAL Task Formulation: The first formal definition of temporal action localization that simultaneously enforces strict online inference constraints and point-level supervision, complete with evaluation protocols and baselines.
2. OnPoint Framework: The first offline-to-online distillation framework for TAL under point supervision. It uniquely bridges the gap between full-video context and streaming constraints via multi-level distillation (pseudo-segments, CASS, and anticipation).
3. Novel Architectural Components:
   • CASS Distillation: The first integration of CAS-based distillation into an online pipeline for fine-grained temporal guidance.
   • Window-Level Anticipation: A tailored distillation mechanism that teaches the online model to anticipate upcoming actions based on offline temporal context.
   • Actionness-Calibrated Decoding: A novel attention mechanism that uses actionness scores to refine anchor features in the absence of future frames.
4. Robustness Mechanism: The introduction of direct anchor-level point supervision to stabilize training against noisy pseudo-labels.

Experimental Results

The authors evaluated OnPoint on five datasets: THUMOS'14, EGTEA, HOI4D-Office, FineAction, and EPIC-Kitchens-100.

• Performance: OnPoint consistently outperformed strong baselines (including distillation-free approaches and other online methods trained with full supervision) across all datasets.
  • On THUMOS'14, it achieved an average mAP gain of 4.8% over the best baseline, reaching 61.9% average mAP (tIoU 0.1–0.5), surpassing several fully-supervised online methods.
  • On EGTEA and HOI4D-O, it showed consistent gains of 3.4% and 2.3% respectively over baselines.
• Ablation Studies: Removing any of the three distillation signals (CASS, Anticipation, or Instance-level) resulted in performance drops, confirming the necessity of the multi-level approach. The actionness-calibrated attention mechanism alone contributed a 2.0% improvement.
• Robustness: The model demonstrated resilience to noise in the teacher's pseudo-labels when anchor-level point supervision was active, maintaining stability where the distillation-free variant failed.
• Efficiency: The online model is computationally efficient (93M parameters, ~312 FPS on RTX 4090), offering a favorable balance between speed and accuracy compared to heavier models like HAT or MATR.

Significance and Claims

The paper positions OnPoint as a solid foundation for POTAL research. It claims to be the first work to successfully bridge the gap between the label efficiency of point annotations and the strict causality constraints of online streaming.

The authors emphasize that while Large Multimodal Models (LLMs) were tested for automatic label generation, they proved unreliable for precise temporal localization (achieving only ~29% mAP compared to the teacher's ~90%). In contrast, OnPoint demonstrates that a task-specific offline teacher trained on weak human point labels can effectively transfer knowledge to an online student, enabling scalable, real-time action localization without the prohibitive cost of dense annotation. The work establishes that high-performance online TAL is achievable even with minimal supervision, provided the right distillation architecture is employed.