Privacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication

1. Problem Statement

Federated Learning (FL) allows for collaborative model training without sharing raw video data, addressing privacy concerns in video action recognition. However, existing FL approaches for video tasks face two critical vulnerabilities:

• Model Exposure (Privacy Leakage): Even without raw data, the gradients exchanged between clients and the server can be exploited to reconstruct visual content or infer sensitive user actions (e.g., motion patterns). Standard Differential Privacy (DP) methods often degrade model accuracy significantly when applied to high-dimensional video networks.
• Communication Overhead: Video models (e.g., Transformers like MViT) are high-dimensional. Synchronizing full-model updates across clients and a server creates massive bandwidth consumption and latency, making FL impractical for real-world deployment.

Existing solutions either ignore communication efficiency, apply uniform noise across all layers (hurting utility), or fail to handle the temporal correlations inherent in video data effectively.

2. Methodology: FedDP-STECAR

The authors propose FedDP-STECAR (Federated Differential Privacy with Selective Tuning and Efficient Communication for Action Recognition). The framework integrates three core mechanisms:

A. Selective Fine-Tuning

Instead of updating the entire neural network, the framework partitions model parameters into:

• Frozen Parameters ($\theta_f$): The majority of the model remains fixed.
• Trainable Parameters ($\theta_t$): Only a small, task-relevant subset (specifically the final transformer stages and the classification head) is updated.
• Benefit: This confines the sensitivity of the model to a smaller subspace, limiting the propagation of noise and reducing the "surface area" for information leakage.

B. Differentially Private SGD (DP-SGD) with Top-Level Sampling (TLS)

• DP-SGD: Applied only to the trainable subset $\theta_t$. Per-sample gradients are clipped and perturbed with Gaussian noise.
• Top-Level Sampling (TLS): The framework employs sampling without replacement during local epochs. This ensures each sample is used exactly once per epoch, providing privacy amplification. This allows for stronger privacy guarantees (lower $\epsilon$) with less noise injection compared to standard sampling with replacement.

C. Communication-Efficient Aggregation

• Masked Updates: Clients transmit only the updates for the trainable subset ($\Delta^t_k = \text{Mask}_{\theta_t}(w_{k,t+1} - w_t)$) rather than the full model.
• Aggregation-Agnostic: The server can aggregate these sparse updates using standard algorithms like FedAvg or FedNova without modification.
• Result: This drastically reduces the data payload per communication round.

3. Key Contributions

1. Selective DP Strategy: The paper introduces a strategy that confines DP noise to task-relevant parameters. This preserves temporal coherence in video features while significantly reducing unnecessary perturbations.
   • Impact: Achieves up to 70.2% higher accuracy under strict privacy ($\epsilon = 0.65$) compared to full fine-tuning baselines in centralized settings.
2. Massive Communication Reduction: By transmitting only tuned layers, the framework reduces communication traffic by >99% (from ~1456 MB to ~3.1 MB per round) and reduces training runtime by 48%.
3. Scalability and Robustness: The framework maintains high accuracy (73.1%) in federated setups with varying client counts and privacy budgets, demonstrating that it scales effectively even under strict constraints.
4. Privacy Amplification via TLS: The integration of Top-Level Sampling provides additional privacy guarantees, allowing for better utility-privacy trade-offs, particularly in centralized scenarios.

4. Experimental Results

The framework was evaluated on the UCF-101 dataset using the MViT-B-16x4 transformer architecture.

• Centralized Performance (Privacy vs. Accuracy):

  • Under a strict privacy budget ($\epsilon = 0.65$), the selective tuning model with TLS (DP-Adam-SelTLS) achieved significantly higher accuracy than full fine-tuning models.
  • It outperformed full fine-tuning by 70.2% in accuracy, proving that limiting noise to specific layers preserves model utility.

• Federated Performance (Efficiency & Accuracy):

  • Accuracy: Achieved 73.1% accuracy with $\epsilon = 1.33$, whereas full fine-tuning baselines collapsed to below 22-28%.
  • Communication: Reduced traffic from 1456 MB to 3.1 MB per client per round (99.8% reduction).
  • Latency: Reduced communication delay from 174.7 seconds to 0.37 seconds (a 470x speedup).
  • Runtime: Reduced total training time by 48%.

• Scalability:

  • The framework maintained accuracy above 72% even when scaling to 20 clients and varying privacy budgets, while full fine-tuning accuracy dropped below 20%.
  • Performance was consistent across different aggregation algorithms (FedAvg and FedNova), confirming the framework is aggregation-agnostic.

5. Significance

This work addresses the "trilemma" of Privacy, Accuracy, and Communication Efficiency in federated video learning.

• Practical Deployment: By reducing bandwidth requirements by nearly 99%, FedDP-STECAR makes federated video action recognition feasible for bandwidth-constrained environments (e.g., mobile devices, edge sensors).
• Privacy Utility: It demonstrates that strong privacy guarantees do not require sacrificing model accuracy if the noise is applied intelligently (selectively) rather than uniformly.
• Generalizability: The approach is compatible with existing federated optimizers and can be applied to other high-dimensional video tasks beyond action recognition.

In summary, FedDP-STECAR provides a scalable, privacy-preserving solution for video analytics that overcomes the traditional bottlenecks of communication costs and accuracy degradation in Federated Learning.