FeDepth: Federated Learning for Depth Estimation under Robot Heterogeneity

Technical Summary of FeDepth: Federated Learning for Depth Estimation under Robot Heterogeneity

Problem Statement
Robot perception, particularly monocular depth estimation, relies heavily on training data from diverse environments to ensure generalization. However, existing approaches predominantly use Centralized Learning (CL), which requires transmitting high-bandwidth raw sensory data to a central server. This creates significant communication overhead, scalability bottlenecks, and privacy concerns. Federated Learning (FL) offers a distributed alternative where robots train locally and share only model weights. Yet, standard FL (e.g., FedAvg) and even existing Clustered FL (CFL) methods struggle in robotic deployments due to severe heterogeneity. This heterogeneity arises from diverse robot platforms (e.g., UGVs, UAVs, legged robots), varying environments (e.g., urban, forest, indoor), and unbalanced data quantities.

A critical gap identified in this work is that real-world robotic data distributions are often partially overlapping rather than clearly separable. A single robot may operate in multiple environments, or different robots may share similar environments but differ in platform dynamics. Conventional CFL methods assume clients can be partitioned into distinct, non-overlapping clusters (hard clustering). This assumption fails in robotic settings where domain shifts are continuous and ambiguous, leading to performance degradation when clients are forced into a single, potentially mismatched cluster.

Methodology: FeDepth Framework
The authors propose FeDepth, a descriptor-based Clustered FL framework designed to handle these overlapping and complex inter-client relationships through soft clustering. The framework operates in three main stages:

1. Soft Clustering (Client Descriptor Generation):

   • Instead of relying on raw images or model parameters alone, FeDepth utilizes a pre-trained encoder (from baseline depth models like AdaBins, NeWCRFs, or DCDepth) to extract visual descriptors from a subset of each client's local data.
   • These feature maps are compressed using Generalized Mean (GeM) pooling. The mean and variance of these pooled features are aggregated to form a client descriptor $(\mu_c, \sigma_c^2)$, representing the client's feature distribution.
   • The server computes pairwise similarities between clients using the Jeffreys divergence (a symmetric form of KL divergence) between their Gaussian approximations.
   • Unlike hard clustering, FeDepth employs a soft clustering strategy. Clients are assigned to multiple clusters based on a similarity threshold $\tau$. This allows a single client to participate in multiple relevant clusters, capturing the continuous and ambiguous transitions common in robotic environments. A merging step resolves redundant partitions to finalize the cluster map.

2. Cluster Model Update:

   • During training rounds, clients perform local updates.
   • The server aggregates models cluster-wise. For each cluster, it averages the models of the clients assigned to that cluster (weighted by data size).
   • Clients then download the models of all clusters they belong to and average them to initialize their next round of local training. This mechanism allows clients to benefit from knowledge in multiple related domains.

3. Inference:

   • For a new test sequence, the system extracts a descriptor and computes similarity to existing cluster centroids.
   • The sequence is assigned to a set of candidate clusters (those below the threshold $\tau$), and the final prediction is an average of the models from these selected clusters.

Key Contributions

1. Systematic Investigation of Robot-Driven Heterogeneity: The paper highlights that robot perception FL faces unique challenges due to platform, environment, and depth distribution shifts, which are often under-explored compared to standard image classification FL.
2. Novel Non-IID Scenarios (HPE and BMR): The authors introduce two realistic testbeds derived from existing datasets (M3ED, NYUv2, KITTI):
   • Heterogeneous Platform-Environment (HPE): Simulates combinatorial variations where clients share attributes (e.g., same environment, different robot) or have overlapping trajectories, creating ambiguous domain boundaries.
   • Bi-Modal Range (BMR): Simulates a severe shift between indoor (close-range) and outdoor (long-range) depth distributions, testing the ability to handle fundamentally different depth scales.
3. FeDepth Framework: A model-agnostic CFL framework that utilizes soft clustering to allow clients to belong to multiple groups. This addresses the limitation of hard clustering in handling overlapping data distributions.
4. Comprehensive Evaluation: The method is validated across multiple depth estimation architectures (AdaBins, NeWCRFs, DCDepth) and consistently outperforms standard FL, personalized FL, and hard-clustering baselines.

Experimental Results

• HPE Scenario: FeDepth consistently outperforms all baselines (FedAvg, FedProx, FedDyn, SCAFFOLD, FedClust, PACFL) across all tested architectures. For instance, with the NeWCRFs backbone, FeDepth achieved an Abs Rel of 0.249, significantly lower than FedAvg (0.366) and hard-clustering PACFL (0.318). The results demonstrate that soft clustering better captures the continuous nature of robotic data distributions.
• BMR Scenario: In the clearly separated indoor/outdoor scenario, FeDepth performs comparably to hard-clustering methods (PACFL) for architectures like NeWCRFs and DCDepth, where clusters are naturally distinct. However, for AdaBins, where feature representations are less separable, FeDepth still outperforms centralized learning (CL) and standard FL, showing robustness even when hard clustering is theoretically optimal.
• Ablation Studies:
  • Soft vs. Hard: Soft clustering consistently outperforms hard clustering under the same threshold, confirming that robotic data distributions are too continuous for rigid partitioning.
  • Overlap Sensitivity: An overlap ratio of 20% (clients assigned to multiple clusters) yielded optimal performance. Too little overlap limits knowledge sharing; too much (100%) collapses the system into standard FedAvg.
  • Label-Based vs. Distribution-Based: Clustering based on ground-truth labels (platform/environment) resulted in unstable performance compared to FeDepth's distribution-aware clustering, suggesting that semantic labels do not always align with feature distribution similarity.

Significance and Claims
The paper claims that FeDepth provides a practical and effective solution for federated robot perception by addressing the specific challenge of overlapping client domains. By moving away from the assumption of clearly separable clusters, FeDepth enables robust training in heterogeneous multi-robot systems where robots share multiple attributes. The authors position FeDepth not just as an improvement in accuracy, but as a necessary step toward scalable, privacy-preserving perception for diverse real-world robotic deployments. The work establishes new benchmarks (HPE and BMR) for evaluating FL under realistic robot heterogeneity, moving beyond controlled laboratory settings.