Technical Summary: Prior-Adaptive Transfer for Skeleton-Based Action Recognition (PATS)

Problem Statement

While skeleton-based action recognition models have achieved strong performance on large-scale general benchmarks (e.g., NTU RGB+D), directly transferring these models to domain-specific, low-data scenarios—particularly in healthcare monitoring (e.g., Alzheimer's detection, fall detection)—often yields suboptimal results. Two primary challenges hinder this transfer:

1. Domain Shift and Relevance: Healthcare tasks often involve subtle, clinically relevant movement deviations that are underrepresented in general action datasets. Consequently, not all pretrained motion priors are equally useful; retaining irrelevant components can hinder adaptation.
2. Efficiency Constraints: Real-world healthcare deployment often faces resource constraints (limited data, computational power, and storage). Standard full-parameter fine-tuning retains redundant components, increasing computational costs without necessarily improving performance for narrow-scope tasks.

Existing transfer learning methods typically rely on full fine-tuning or generic model compression (e.g., knowledge distillation, weight pruning) that do not explicitly assess the relevance of specific pretrained motion priors to the target healthcare task.

Methodology: The PATS Framework

The authors propose Prior-Adaptive Transfer of Skeletons (PATS), a unified framework designed to adapt general skeleton-based models to domain-specific tasks by selectively retaining task-relevant motion priors while filtering redundant ones. The framework consists of three modules:

1. Skeleton Signal Extraction

Raw video data is processed to extract standardized skeleton sequences.

• Pose Estimation: The framework adopts AlphaPose as the default estimator due to its robustness in lateral views and occluded scenarios compared to OpenPose.
• Unification: Extracted skeletons undergo topological and dimensional unification, mapping them to the 25-joint representation and topology of the NTU RGB+D dataset to ensure consistency across datasets.

2. Skeleton Action Modeling

A graph-based backbone, specifically InfoGCN, is employed to encode spatio-temporal dynamics. This backbone serves as a bridge, exposing general motion priors learned from large-scale datasets (like NTU RGB+D 60) for subsequent adaptation.

3. Adaptive Prior Transfer (Core Contribution)

This module performs iterative pruning and refinement to compress the model while preserving task-relevant knowledge. The process involves two main steps:

• Redundancy Scoring:

  • The model is treated as a sequence of computational blocks.
  • To identify redundant blocks, the framework calculates the Mean Squared Error (MSE) between the input and output feature representations of each block on the target domain training set ($D_t$).
  • A block with a low MSE indicates that its output is nearly identical to its input (acting as an identity mapping) under the target distribution, suggesting it contributes minimally to the specific task.
  • The block with the lowest MSE ($k^*$) is identified as redundant and pruned.

• Compensatory Fine-Tuning:

  • Pruning can perturb the learned feature distribution. To recover performance, the pruned model undergoes fine-tuning.
  • Strategies: The authors evaluate two strategies: Full-Parameter Fine-Tuning (FFT) for maximum accuracy and LoRA (Low-Rank Adaptation) for parameter efficiency, suitable for edge deployments.
  • Alignment Losses: To further stabilize the pruned model against the original unpruned model ($M_0$), two additional loss functions are introduced during fine-tuning:
    1. Inter-block Model Alignment ($L_e$): Aligns the output features of the pruned model's blocks with the corresponding blocks in the unpruned model.
    2. Action-level Alignment ($L_{KL}$): Uses Kullback–Leibler (KL) divergence to align the class prediction logits of the pruned model with the unpruned model.
  • The total loss function is $L = L_{AR} + L_e + L_{KL}$, where $L_{AR}$ is the standard action recognition loss.

Key Contributions

1. PATS Framework: A novel framework for adapting skeleton action models to low-data, domain-specific scenarios by combining transfer learning with selective prior retention.
2. Adaptive Prior Transfer Strategy: A mechanism that uses redundancy-aware block pruning (based on input-output feature discrepancy) and compensatory adaptation to remove task-irrelevant priors while preserving useful ones.
3. Empirical Validation: Extensive experiments demonstrating that selectively transferring priors improves both accuracy and efficiency compared to standard fine-tuning and other compression baselines.

Experimental Results

The authors evaluated PATS on two healthcare tasks:

• Alzheimer's Detection: Using the EHE dataset (6 categories of morning exercises).
• Fall Detection: Using the UP-Fall dataset (11 classes including falls and daily activities).

Key Findings:

• Performance: Models trained from scratch performed poorly. Pretrained models with full fine-tuning (FFT) performed well, but PATS (FFT) consistently achieved the best results, often outperforming the uncompressed baseline. For instance, on the EHE dataset, PATS (FFT) with 1 block pruned achieved 85.52% accuracy (Fold 1), surpassing the uncompressed baseline (84.14%).
• Efficiency: The pruning strategy reduced model parameters by approximately 30% and improved inference latency by roughly 10%.
• Comparison: PATS outperformed other compression baselines, including model distillation and weight pruning (L1-norm), which degraded performance relative to the uncompressed model.
• Ablation Studies:
  • Both inter-block alignment ($L_e$) and action-level alignment ($L_{KL}$) were shown to be critical for recovering performance after pruning.
  • The MSE metric for redundancy scoring was validated; removing the block with the lowest MSE (Block 8 in the UP-Fall experiments) yielded the best accuracy, confirming the metric's effectiveness.

Significance and Claims

The paper claims that PATS addresses a critical gap in deploying skeleton-based action recognition in real-world healthcare settings. By moving beyond simple fine-tuning to selective prior transfer, the framework enables models to generalize effectively in low-data domains while simultaneously improving computational efficiency.

The authors emphasize that this approach allows for the deployment of robust action recognition systems on resource-constrained devices (e.g., edge computing in community settings) without sacrificing the diagnostic accuracy required for clinical applications like Alzheimer's monitoring or fall detection. The work suggests that "less is more" in transfer learning: removing redundant general priors can actually enhance task-specific performance.