Technical Summary: Onboarding Without Forgetting: Hypernetwork Personalization with Data-Free Replay for Personalized Federated Learning

1. Problem Formulation: Progressive Client Onboarding in FL (PCO-FL)

The paper identifies a gap in existing Personalized Federated Learning (pFL) and Federated Continual Learning (FCL) literature. While standard pFL assumes a static set of clients and FCL assumes sequential task shifts, real-world scenarios often involve a fixed task and label space where the client population evolves over time.

The authors define this setting as Progressive Client Onboarding in Federated Learning (PCO-FL). In this regime:

• Task Stability: The classification task and label space remain constant.
• Dynamic Cohorts: Clients arrive in batches ($B_t$) over time. Once onboarded, existing clients ($C_{<t}$) do not retrain locally; they remain "frozen" while new batches join.
• The Core Conflict:
  1. Proactive Adaptation (PA): Updating the shared model with new clients improves their performance (providing a strong initialization) but typically degrades the accuracy of existing clients due to gradient misalignment and capacity conflicts.
  2. Retroactive Improvement (RI): Freezing the shared model protects existing clients but prevents them from benefiting from the new knowledge acquired by later batches.

The objective is to maximize PA for new clients while ensuring non-negative RI (improvement or stability) for existing clients without retraining them or accessing their raw data.

2. Methodology: pFedDSH

To address these trade-offs, the authors propose pFedDSH (Personalized Federated Learning via Data-free Sub-Hypernetwork). The framework consists of three integrated components:

A. Central Hypernetwork for Personalization

Instead of training local models directly, a central server maintains a hypernetwork $H(\cdot; \phi)$.

• Mechanism: Each client $c$ is represented by an embedding $e_c$. The hypernetwork generates personalized weights $\theta_c = H(e_c; \phi)$ for the base architecture.
• Benefit: This allows the server to control personalization, generate instant models for new clients, and manage capacity allocation centrally without requiring local fine-tuning of the hypernetwork parameters by clients.

B. Batch-Specific Binary Masking

To prevent new updates from overwriting knowledge critical to existing clients, the method employs learnable binary masks.

• Capacity Allocation: For each onboarding step $t$ and layer $\ell$, the server learns mask logits $z^{(\ell)}_t$. A sigmoid function with a high scale $\gamma$ converts these to near-binary gates $g^{(\ell)}_t$.
• Cumulative Union: A cumulative mask $m^{(\ell)}_{\le t}$ is maintained as the element-wise maximum of previous masks and the current step's gate.
• Gradient Modulation: Gradients are modulated such that weights are only updated if they involve newly active channels at step $t$. If a connection was already allocated to a previous batch, its gradient is zeroed out, effectively freezing that capacity for existing clients while allowing new capacity to absorb conflicting updates.

C. Server-Side Data-Free Replay

Since existing clients do not retrain and their data is private, the server uses DeepInversion to synthesize data and propagate improvements backward.

• Synthesis: The server generates synthetic images $\tilde{x}$ by optimizing a loss function that matches the running statistics (mean and variance) of BatchNorm layers from the client training phase, combined with Total Variation (TV) and L2 regularization to ensure natural image structure.
• Retroactive Fine-Tuning: Using the synthetic dataset $S_t$ derived from the current batch, the server fine-tunes the hypernetwork. Crucially, it retrieves the specific subnetwork defined by the batch-specific mask of each previous batch $t' < t$ and updates those parameters.
• Result: This allows the server to "teach" existing clients the new knowledge derived from recent batches without requiring the clients to possess raw data or perform local training.

3. Key Contributions

1. Definition of PCO-FL: The paper formally introduces and analyzes the Progressive Client Onboarding scenario, distinguishing it from standard pFL and FCL by its fixed task but evolving client population.
2. Empirical Investigation: Through controlled experiments, the authors demonstrate that:
   • Progressive aggregation helps new clients (positive PA) but harms existing ones (negative RI).
   • Freezing the model protects existing clients (RI $\approx$ 0) but prevents knowledge transfer.
   • A mechanism is needed to achieve both positive PA and positive RI simultaneously.
3. pFedDSH Framework: The proposal of a novel architecture combining hypernetworks, batch-specific masking, and data-free replay to solve the onboarding dilemma.
4. Comprehensive Evaluation: Extensive experiments showing pFedDSH outperforms traditional pFL, late-client-joining hypernetworks, and FCL baselines across multiple datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet).

4. Experimental Results

The authors evaluate pFedDSH against baselines (FedPer, FedRep, FedSelect, pFedHN, PeFLL, FedWeIT, etc.) under varying onboarding schedules (2, 5, and 11 batches).

• Performance (PA & RI):
  • Existing Clients: pFedDSH is the only method that achieves consistently positive RI (e.g., +3.80% to +6.20% on CIFAR-10 depending on the schedule), meaning existing clients actually improve as new batches join. All other baselines suffer significant accuracy drops (negative RI).
  • New Clients: pFedDSH maintains strong PA (gains over isolated training), comparable to or better than other hypernetwork-based methods.
• Capacity Efficiency:
  • Analysis of neuron usage shows that pFedDSH does not allocate disjoint subnetworks for every batch. Instead, it reuses up to ~40% of channels across batches, gradually increasing total capacity only when necessary.
• Efficiency:
  • Communication: The communication overhead per round is comparable to other hypernetwork methods (differing by only a few kilobytes).
  • Latency: While server-side replay adds computation, pFedDSH runs it only once per batch (amortized cost), resulting in a modest latency increase compared to methods that run replay every round (like HR) or no replay at all.

5. Significance and Claims

The paper claims that pFedDSH offers a practical solution for real-world federated systems where client participation is dynamic. Its significance lies in:

• Solving the "Onboarding vs. Forgetting" Trade-off: It demonstrates that it is possible to onboard new clients and improve the system's global knowledge without sacrificing the performance of incumbent clients.
• Privacy Preservation: By using server-side data-free replay (synthesizing images from BatchNorm statistics rather than raw data or gradients), the method facilitates knowledge transfer without exposing client data, addressing a critical privacy concern in FL.
• Scalability: The masking mechanism ensures that the model does not grow linearly with the number of batches, making it scalable for long-term deployment.

The authors conclude that pFedDSH bridges the gap between personalization and continual learning in a setting previously overlooked, providing a robust framework for evolving client cohorts in privacy-sensitive domains like healthcare.