Scaling Unsupervised Multi-Source Federated Domain Adaptation through Group-Wise Discrepancy Minimization

Problem Statement

Unsupervised Multi-Source Domain Adaptation (UMDA) aims to learn a model that generalizes to an unlabeled target domain by leveraging labeled data from multiple source domains. While federated learning offers a privacy-preserving solution by avoiding raw data sharing, existing federated UMDA methods struggle with scalability. Current approaches are typically designed for small-scale scenarios (2–6 sources) and suffer from high computational overhead or training instability when the number of heterogeneous source domains increases. Specifically, methods relying on pairwise alignment become computationally prohibitive ($O(N^2)$), while those using random pairwise sampling (e.g., FACT) introduce high variance and fail to converge in high-diversity settings.

Methodology: The GALA Framework

The authors propose GALA (Grouping-based Adaptive Learning), a federated UMDA framework designed to scale with the heterogeneity of diverse source domains. GALA operates through two primary mechanisms:

1. Inter-Group Discrepancy (IGD) Minimization
To address the quadratic complexity of full pairwise alignment, GALA introduces a group-level discrepancy objective.

• Mechanism: At each training round, the $N$ source predictors are randomly partitioned into two disjoint groups, $G_1$ and $G_2$.
• Aggregation: Weighted average predictors are formed for each group ($F_{G_1}$ and $F_{G_2}$).
• Objective: The model minimizes the $\ell_1$ disagreement between these two group predictors on the unlabeled target data:
  $$L_{IGD} = \mathbb{E}_{x \sim D_T} [\|F_{G_1}(G(x)) - F_{G_2}(G(x))\|_1]$$
• Theoretical Advantage: Unlike random pairwise sampling (which has high variance) or full pairwise computation (which is expensive), IGD averages many pairwise differences before taking the norm. This results in a low-variance, linear-complexity approximation of the global alignment objective. While it introduces a small bias, the variance reduction leads to more stable training.

2. Temperature-Scaled Centroid-Based Weighting (MDMGB+)
To prevent negative transfer from irrelevant or noisy sources, GALA dynamically estimates the relevance of each source to the target.

• Centroid Similarity: Clients compute soft class-wise centroids in the feature space. The server calculates cosine similarity between source centroids and the target centroid.
• Temperature Scaling: The authors introduce a modified version of the MDMGB algorithm, termed MDMGB+. It applies a softmax function with a tunable temperature parameter $\tau$ to the similarity scores:
  $$w_n = \frac{\exp(\tau \cdot S(r_T^c, r_n^c))}{\sum_j \exp(\tau \cdot S(r_T^c, r_j^c))}$$
• Function: The temperature $\tau$ controls selectivity. Higher values amplify small differences in similarity, assigning greater weight to sources better aligned with the target and suppressing irrelevant ones. These weights are used to form the group predictors in the IGD step.

Algorithm Flow:
The process involves iterative rounds where the server broadcasts the global model, clients compute centroids and update models locally, the server aggregates feature extractors using relevance weights, and finally, the target domain updates the feature extractor by minimizing the IGD loss between the two group predictors.

Key Contributions

1. Scalability Analysis: The authors identify that existing distributed UMDA methods fail to converge or become computationally infeasible as the number of heterogeneous sources grows, a gap largely unaddressed in current literature.
2. Inter-Group Discrepancy (IGD): A novel objective that approximates full pairwise disagreement at linear cost with low variance, enabling robust alignment in large-scale settings.
3. GALA Algorithm: A scalable federated UMDA framework combining IGD with a temperature-scaled, centroid-based weighting scheme to prioritize target-relevant sources and mitigate negative transfer.
4. Digit-18 Benchmark: The introduction of a new benchmark comprising 18 diverse digit datasets (spanning synthetic and real-world shifts) to rigorously test scalability and source heterogeneity, addressing the scarcity of large-scale UMDA benchmarks.

Experimental Results

The paper evaluates GALA on standard benchmarks (Digit-Five, Office-Caltech10) and the new Digit-18 benchmark.

• Standard Benchmarks: GALA achieves state-of-the-art (SOTA) or near-SOTA performance on Digit-Five and Office-Caltech10, matching or exceeding methods like FACT and KD3A.
• Scalability (Digit-18): In high-diversity settings (up to 18 sources), existing methods show significant degradation:
  • FACT: Becomes unstable with high variance and fails to converge on challenging targets as source count increases.
  • KD3A: Suffers from exponential runtime growth due to sequential consensus steps, becoming computationally infeasible beyond 9 sources.
  • GALA: Maintains stable accuracy and convergence. In the 18-source setting, GALA achieves a 5.3% average gain over FACT and outperforms all baselines on every target domain.
• Ablation Studies: Results confirm that both the IGD objective and the MDMGB+ weighting are critical. IGD alone fails to converge on difficult targets without the weighting, while weighting alone is insufficient without the group-wise alignment.
• Efficiency: While GALA has a slightly higher per-round cost than FACT due to group alignment, it avoids the exponential computational explosion of KD3A, maintaining linear scalability.

Significance and Claims

The paper claims to be the first to explicitly address scalability in distributed UMDA regarding the number of heterogeneous source domains. The significance of GALA lies in its ability to enable stable, parallelizable training across many diverse sources where current methods either fail to converge or are computationally infeasible. By combining group-wise discrepancy minimization with dynamic source prioritization, GALA establishes a new state-of-the-art for large-scale, high-diversity multi-source domain adaptation, particularly in privacy-sensitive federated settings. The authors note that while the method incurs higher communication costs per round than selective methods, it provides a centralized workflow without requiring the target to receive all source models, and future work could focus on reducing source participation frequency.