Technical Summary: Consistent Distributed Ranking of Generative Models via Kernel Distances

Problem Statement

The evaluation and ranking of generative models (e.g., GANs, Diffusion Models, VAEs) are critical for identifying the best-performing architectures. In standard centralized settings, this is achieved by comparing generated data against a single reference distribution using metrics like Fréchet Distance (FD) or Kernel Distance (KD). However, modern applications increasingly operate in distributed settings (e.g., Federated Learning), where reference data resides across multiple clients with heterogeneous data distributions.

In such scenarios, a centralized evaluation (aggregating all client data to compute a single "score-all" metric) is often infeasible due to privacy constraints and communication costs. Instead, practitioners may rely on aggregating individual client scores ("score-avg"). A fundamental question arises: Does aggregating individual client scores yield the same model ranking as a centralized evaluation? The paper investigates whether the ranking of generative models remains consistent between these two approaches across different evaluation metrics.

Methodology

The authors analyze the consistency of distributed ranking for two primary classes of distance-based metrics: Kernel Distance (KD) and Fréchet Distance (FD).

1. Theoretical Framework

The study defines two aggregate scoring mechanisms for $k$ clients with distributions $P_{X_1}, \dots, P_{X_k}$ and weights $\lambda_i$:

• Score-All ($D_{all}$): The distance between the generative model $P_G$ and the mixture distribution $\bar{P}_X = \sum \lambda_i P_{X_i}$. This represents the ideal centralized evaluation.
• Score-Avg ($D_{avg}$): The weighted average of individual client distances: $\sum \lambda_i D(P_{X_i}, P_G)$.

The authors derive theoretical relationships between $D_{all}$ and $D_{avg}$ for KD and FD.

2. Kernel Distance (KD) Analysis

The KD metric is defined as the squared Maximum Mean Discrepancy (MMD) using a kernel function $k$.

• Theorem 1: The authors prove that for any kernel function satisfying standard Reproducing Kernel Hilbert Space (RKHS) conditions, the relationship between the average KD score and the centralized KD score is:
  $$KD_{avg} = KD_{all} + \sum_{i=1}^k \lambda_i KD(\bar{P}_X, P_{X_i})$$
• Implication: The term $\sum \lambda_i KD(\bar{P}_X, P_{X_i})$ depends only on the client data distributions and is independent of the generative model $P_G$. Consequently, $KD_{avg}$ and $KD_{all}$ differ by a constant offset for all models. This guarantees that ranking models by $KD_{avg}$ yields the exact same ordering as ranking by $KD_{all}$, regardless of data heterogeneity.

3. Fréchet Distance (FD) Analysis

The FD metric is defined as the 2-Wasserstein distance between Gaussian approximations of the data distributions.

• Theorem 2: The authors show that the optimal covariance matrix for the centralized aggregation ($\bar{C}$) differs from the optimal covariance for the average aggregation ($\tilde{C}$) when client means $\mu_i$ are heterogeneous. Specifically, $\bar{C}$ includes a term accounting for the variance of the client means, while $\tilde{C}$ does not.
• Proposition 1: The authors prove that there exist scenarios where two distinct generative models receive identical individual FD scores from all clients (and thus identical $FD_{avg}$), yet their centralized $FD_{all}$ scores differ significantly. This demonstrates that no aggregation strategy of local FD scores can guarantee a consistent ranking with the centralized FD-all metric.

Key Contributions

1. Problem Formulation: The paper formally identifies and studies the task of ranking generative models in distributed settings with heterogeneous data, highlighting the gap between privacy-preserving local evaluations and centralized ground truth.
2. Consistency of KD: It provides a theoretical proof that the Kernel Distance (KD) metric allows for consistent distributed ranking. Specifically, averaging client KD scores ($KD_{avg}$) is mathematically equivalent to the centralized KD score ($KD_{all}$) in terms of model ordering for any kernel function.
3. Inconsistency of FD: It demonstrates that the widely used Fréchet Distance (FD) metric fails to preserve ranking consistency in distributed settings. The paper proves that local FD scores are insufficient to recover the global ranking, as different models can appear identical locally but differ globally.
4. Empirical Validation: The authors validate these theoretical findings through extensive numerical experiments on standard image datasets (ImageNet, CIFAR, FFHQ, AFHQ) and various generative architectures (BigGAN, StyleGAN, Diffusion models).

Experimental Results

• KD Consistency: Experiments on ImageNet and distributed StyleGAN training show a fixed gap between $KD_{avg}$ and $KD_{all}$ across different models and training configurations. The rankings derived from both metrics are identical.
• FD Inconsistency:
  • In a controlled experiment with six simulated clients, two generators ($G_1, G_2$) had nearly identical client-level FD scores (differences < 1.0) and identical $FD_{avg}$ (195.04 vs. 195.37). However, their centralized $FD_{all}$ scores differed drastically (100.49 vs. 190.93).
  • Experiments on variance-limited federated datasets (where intra-client variance is low but inter-client variance is high) further confirmed that $FD_{avg}$ and $FD_{all}$ can produce contradictory rankings, often favoring different models based on diversity vs. fidelity trade-offs.
• Other Metrics: The study notes that while Recall scores also maintain consistency (Recall-all = Recall-avg), metrics like Precision, Density, and Coverage exhibit ranking inconsistencies similar to FD.

Significance and Claims

The paper claims that Kernel Distance is a uniquely suitable metric for privacy-preserving distributed evaluation of generative models. Because $KD_{avg}$ preserves the ranking of the centralized $KD_{all}$ without requiring data sharing, it enables:

• Private Ranking: Clients can share only scalar scores, allowing a server to rank models based on the global reference distribution without accessing raw data.
• Distributed Optimization: The monotonic relationship between $KD_{avg}$ and $KD_{all}$ allows for distributed fine-tuning (e.g., minimizing MMD loss) where the local objective aligns with the global objective.

The authors modestly conclude that while KD offers a solution for consistent ranking, other popular metrics like FD do not share this property, posing challenges for distributed model selection. They suggest future work could explore differential privacy techniques to further secure the ranking process and investigate ranking under communication constraints.