Technical Summary: FedKT-CSD for One-Shot Federated Learning

Problem Statement

Federated Learning (FL) enables collaborative model training across decentralized data while preserving privacy, but it typically suffers from high communication overhead and slow convergence, particularly when client data distributions are heterogeneous. One-Shot Federated Learning (OSFL) addresses communication constraints by limiting interaction to a single round. However, existing OSFL methods face a critical trade-off: they often lack formal privacy guarantees or require computationally expensive local training (e.g., training local generators or distilling models). When Differential Privacy (DP) is retrofitted to these methods, the high dimensionality of model parameters inflates sensitivity, degrading utility. Furthermore, many current approaches transmit model weights or distilled data that can leak information about individual training records.

The core challenge addressed by this work is: Can a framework be devised that leverages representation sharing in an OSFL setting to achieve formal privacy guarantees and lightweight client computation simultaneously?

Methodology: FedKT-CSD

The authors propose FedKT-CSD (Federated Knowledge Transfer via Collaborative Synthetic Data), a framework inspired by neural image compression. The method utilizes a publicly pretrained, frozen autoencoder (Encoder $E$ and Decoder $D$) as a shared latent space, eliminating the need for on-device training or adaptation.

1. Client-Side Processing (No Training)

Each client $i$ holds a private dataset $D_i$. The process involves a single forward pass:

• Encoding: Images are encoded into compact latent vectors $z_{i,j} = E(x_{i,j})$.
• Clipping: Latent vectors are clipped to a bounded $\ell_2$-norm $R$ to ensure sensitivity bounds for DP.
• Statistic Computation: For each class $c$ present in the client's data, the client computes three additive statistics:
  • Sum of clipped latents: $m_c^i = \sum \bar{z}_{i,j}$
  • Sum of outer products: $M_c^i = \sum \bar{z}_{i,j}\bar{z}_{i,j}^\top$
  • Sample count: $n_c^i$
• Transmission: Clients upload only these lightweight summaries (vectors and matrices) to the server. No raw data or model weights are transmitted.

2. Server-Side Aggregation and Noise Injection

The server aggregates the statistics using Secure Aggregation, which computes element-wise sums across clients without revealing individual contributions.

• Global Sums: $S_{\mu,c} = \sum m_c^i$, $S_{\Sigma,c} = \sum M_c^i$, and $n_c = \sum n_c^i$.
• Differential Privacy: Calibrated Gaussian noise is added to the aggregated sums ($\tilde{S}_{\mu,c}$ and $\tilde{S}_{\Sigma,c}$) based on the sensitivity of the clipping threshold $R$. The privacy budget $(\epsilon, \delta)$ is split between the mean and covariance queries.
• Post-Processing: The server recovers the noisy mean and covariance, applies bias correction (to account for the variance of the noise in the mean) and Positive Semidefinite (PSD) projection (to ensure valid sampling).

3. Synthetic Data Generation

The server samples latent vectors from the resulting class-conditional Gaussian distributions $\mathcal{N}(\mu_c^{DP}, \Sigma_c^{DP})$ and decodes them using the frozen decoder $D$ to generate a DP-protected synthetic dataset $X^*$. This dataset is then used to train a global model or for downstream tasks.

Key Contributions

1. One-Shot DP Synthetic Data Generation: The paper demonstrates that deep compression autoencoders provide a shared latent space where class-conditional statistics can be computed and aggregated with formal $(\epsilon, \delta)$-DP guarantees. This enables a pipeline requiring only a single communication round and zero on-device training.
2. Competitive Performance under DP: Despite operating under strict privacy constraints, FedKT-CSD outperforms existing non-private one-shot FL baselines (e.g., FedSD2C, DENSE, CoBoosting) across diverse datasets and heterogeneity settings.
3. Heterogeneity and Scale Invariance: Unlike model-aggregation approaches that suffer from biased local models in heterogeneous settings, FedKT-CSD is invariant to the degree of data heterogeneity and the number of clients. The additive nature of the statistics ensures the global distribution is reconstructed identically regardless of data partitioning.
4. Practical Efficiency: The method is highly efficient, with client payloads in the Kilobyte regime (e.g., ~327 KB for 10 classes) and computation times of seconds on consumer hardware, making it deployable in bandwidth- and compute-constrained environments.
5. Versatile Downstream Utility: The generated synthetic dataset consists of actual images (not fixed features), allowing it to be used for training any architecture, pretraining for personalized FL, or as auxiliary data for multi-round methods.

Experimental Results

The authors evaluated FedKT-CSD on four image classification benchmarks: ImageNette, CIFAR-100, EuroSAT, and BloodMNIST.

• Performance vs. Baselines: FedKT-CSD achieved state-of-the-art results among one-shot methods, often outperforming non-private baselines. For instance, on CIFAR-100 under pathological heterogeneity (where each client sees only 10 of 100 classes), baselines collapsed to ~21% accuracy, while FedKT-CSD maintained ~29%. On EuroSAT, it reached ~80% accuracy compared to ~55% for the best baseline.
• Privacy Evaluation: The method satisfies formal $(\epsilon=10, \delta=10^{-5})$-DP. Membership Inference Attacks (MIA) yielded an AUC of 0.501 (random guessing), confirming the inability to infer individual record participation.
• Scalability: The method scales linearly to 1,000 clients with no performance degradation, whereas baselines degrade significantly as data is spread thinner.
• Ablation Studies:
  • Heterogeneity: Performance remained stable across varying Dirichlet concentration parameters ($\alpha$), while baselines degraded monotonically.
  • Participation Rate: The method degraded gracefully even at 10% client participation.
  • Architecture: A latent dimension of $d=128$ with a 64x64 resolution autoencoder (DC-AE f32c32) offered the best trade-off between representational capacity and noise accumulation.

Significance and Claims

The paper claims that FedKT-CSD closes the gap between low communication, robustness to heterogeneity, and rigorous privacy in federated learning. By leveraging the structure of pretrained autoencoders, the method transforms the problem of private data sharing into the aggregation of low-dimensional, additive statistics.

The authors emphasize that the synthetic dataset preserves the inter-class separation necessary for classification without requiring photorealism, which is why it achieves high downstream accuracy despite the coarse appearance of individual generated images. The work suggests a broader design principle: pretrained autoencoders can serve as a shared, low-dimensional space where simple additive statistics suffice for high-quality, privacy-preserving data generation, enabling scalable and efficient federated systems without the computational burden of local training or the privacy risks of parameter exchange.