Distributed sequential federated learning

Technical Summary: Distributed Sequential Federated Learning

Paper Title: Distributed sequential federated learning
Authors: Z. F. Wang, X. Y. Zhang, and Y-c I. Chang
Date: February 2023

1. Problem Statement

Traditional Federated Learning (FL) focuses heavily on the technical aspects of information security and reducing data transportation costs. However, the authors identify a critical gap in the statistical methodology of FL. Specifically, when data is distributed across multiple sites, two major issues arise:

• Data Non-homogeneity: Differences in sample sizes and data quality across sites make standard aggregation methods (like simple or weighted averaging) inefficient or statistically biased.
• Information Loss: Traditional robust methods that treat local results as random observations often fail to retain the full information from individual analyses, leading to a loss of estimation precision.

The core challenge is: How can we effectively and legitimately integrate results from multiple independent sites to obtain a global estimate that is both precise and statistically valid, especially when sites have varying data qualities and sizes?

2. Methodology

The authors propose a novel Distributed Sequential Federated Learning (DSFL) framework. Instead of using fixed sample sizes, they leverage sequential adaptive design within each local site.

A. Local Sequential Estimation:
Each of the $M$ sites performs a local estimation using a Generalized Linear Model (GLM). The sampling process at each site is not fixed; instead, it follows a stopping rule based on two criteria:

1. Estimation Precision: The confidence set for the parameters of interest ($\theta$) must reach a prescribed accuracy ($d_1$).
2. Prediction Performance: The model's predictive power (e.g., AUC for logistic regression or MSE for linear regression) must meet a minimum threshold ($d_2$).

The stopping rule $\tilde{N}_j$ is defined such that sampling continues until the local data satisfies these precision and quality requirements. This allows sites with high-quality data to stop early and sites with low-quality data to continue sampling (if possible).

B. Global Integration (Aggregation):
Once all $M$ sites reach their respective stopping criteria, the results are integrated using a "randomly weighted" approach. The global estimate $\hat{\theta}$ is calculated as:
$$\hat{\theta} = \sum_{j=1}^{M} \rho_j \tilde{\theta}_j$$
where the weights $\rho_j = \tilde{N}_j / \hat{N}$ are proportional to the number of samples used at each site. This weighting scheme is mathematically designed to achieve asymptotic efficiency (minimal covariance) when data is homogeneous.

C. Adaptive Sampling Strategy:
To further optimize the process, the authors incorporate an A-optimal design criterion. This active learning approach selects the most "informative" samples from the local data pool at each step, accelerating the convergence toward the stopping criteria.

3. Key Contributions

• Statistical Integration Framework: Developed a method to combine results from sequential procedures with random stopping times, a task previously under-explored in FL literature.
• Data-Driven Sample Size: Unlike traditional FL, which uses pre-fixed sample sizes, this method allows the sample size to be a random variable determined by the local data's quality and the user's desired precision.
• Preservation of Statistical Properties: Proved that the proposed method maintains asymptotic consistency (the estimate converges to the truth) and asymptotic efficiency (the estimate achieves the best possible variance).
• Dual-Constraint Stopping Rule: Introduced a stopping criterion that simultaneously controls both parameter estimation error and predictive accuracy (e.g., AUC).

4. Results

The authors validated the method through two main avenues:

• Simulation Studies:
  • The DSFL method outperformed "straight average" methods, especially in scenarios with heterogeneous covariates and varying sample sizes ($p_2$ setup).
  • The Adaptive Sampling (A) strategy reached the stopping criteria significantly faster than the Random Sampling (R) strategy.
  • The coverage frequency of the confidence sets approached the theoretical $1-\alpha$ level as precision requirements increased.
• Real-World Application (COVID-19 Data):
  • Applied to a massive dataset from 32 Mexican hospitals.
  • The method successfully identified key risk factors (e.g., age, diabetes, and obesity as positive predictors for COVID-19).
  • The DSFL method achieved higher AUC values than the baseline model and demonstrated that even sites with limited data could contribute meaningfully to the global model through the sequential framework.

5. Significance

This paper shifts the focus of Federated Learning from purely computational/security concerns to statistical rigor. By allowing the data itself to dictate the sampling effort, the DSFL method provides a robust way to handle the "messy" reality of distributed data (varying sizes, quality, and heterogeneity). It is highly significant for medical research and large-scale distributed sensing, where data is often siloed, non-uniform, and requires strict precision guarantees.