Technical Summary: Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing

1. Problem Statement

Remote sensing (RS) generates vast amounts of image data distributed across decentralized archives. However, legal regulations, privacy concerns, and bandwidth constraints often prevent the centralization of this data, rendering conventional deep learning approaches impractical. Federated Learning (FL) offers a decentralized alternative where clients train models locally and share only parameters. A critical challenge in FL for RS is data heterogeneity (non-IID): geographical, seasonal, and sensor-dependent variations cause significant distribution shifts and label imbalances across clients. This often leads to client-specific optima, slowing convergence and degrading the generalization of the aggregated global model.

While Vision-Language Models (VLMs), such as CLIP, demonstrate robustness to distribution shifts due to their transferable representations learned from large-scale image-text data, their application in FL is hindered by their massive parameter size. Full adaptation of these models incurs prohibitive communication overhead and local computational complexity. Consequently, there is a lack of guidance on which VLM adaptation strategy best balances generalization capability, communication efficiency, and computational cost in the specific context of RS image classification.

2. Methodology

The paper presents the first comprehensive comparative study of VLM adaptation strategies within an FL framework for RS. The study evaluates four distinct strategies on the BigEarthNet-S2 dataset (federated training) and assesses generalization on EuroSAT, RESISC45, and ImageNet.

Adaptation Strategies Investigated

1. Full Fine-Tuning (FFT): Updates all image and text encoder parameters ($\theta_{train} = \theta$). Offers maximum representational flexibility but incurs the highest communication and computational costs.
2. Encoder-Specific Fine-Tuning: Restricts updates to either the image encoder ($\theta_{train} = \{\theta_v\}$) or the text encoder ($\theta_{train} = \{\theta_t\}$), freezing the other. This reduces communication costs compared to FFT but limits cross-modal adaptation.
3. Prompt Learning (CoOp): Optimizes only a small set of learnable context vectors (soft prompts) prepended to class names, while keeping encoders frozen. This minimizes communication overhead and computational load but restricts adaptation capacity to the prompt level.
4. Low-Rank Adaptation (LoRA): Freezes pretrained weights and inserts trainable low-rank matrices into transformer projection layers. This allows adaptation to a low-rank subspace, preserving pretrained representations while significantly reducing the number of trainable parameters compared to FFT.

Experimental Design

• Federated Setup: Clients are partitioned by country (e.g., Austria, Belgium, Finland) to simulate non-IID data distributions.
• Baselines: Includes a ResNet-50 trained from scratch and a Zero-shot CLIP baseline.
• Metrics: Performance is measured via micro/macro-mAP (in-distribution) and top-1/top-5 accuracy/recall (out-of-distribution). Complexity is quantified via FLOPs/MACs and the number of transmitted parameters.
• Catastrophic Forgetting Mitigation: The study investigates PAINT, a method using linear interpolation between pretrained and fine-tuned weights, to analyze the trade-off between task specialization and preserving zero-shot generalization.

3. Key Results

In-Distribution Performance (BigEarthNet-S2)

All adaptation strategies significantly outperform the Zero-shot CLIP baseline and the ResNet-50 trained from scratch.

• LoRA and FFT achieve the highest performance, with LoRA improving micro-mAP by over 52% compared to Zero-shot CLIP.
• Encoder-specific strategies show moderate gains, while CoOp provides the smallest improvement, suggesting that limiting updates to prompts constrains the model's ability to align with the substantial domain shift of RS data.

Out-of-Distribution Performance & Catastrophic Forgetting

A critical trade-off is observed between task specialization and cross-domain generalization:

• Severe Forgetting: FFT exhibits the most catastrophic forgetting, dropping from 75.6% top-1 accuracy (Zero-shot) to 7.8% on ImageNet. This indicates that unrestricted adaptation to RS data destroys pretrained representations.
• Preservation: LoRA preserves the most pretrained knowledge, maintaining 75.1% top-1 accuracy on ImageNet (a <1% drop), while still achieving strong performance on RS datasets.
• Encoder Specifics: Text encoder fine-tuning preserves visual features better than image encoder fine-tuning, resulting in better out-of-distribution performance on datasets like EuroSAT and RESISC45 compared to image-only tuning.
• PAINT Analysis: Interpolating weights (PAINT) reveals that maximizing performance on the RS task inherently degrades ImageNet performance. However, intermediate checkpoints and specific interpolation coefficients can offer a balanced compromise.

Complexity and Communication

• Communication Cost: CoOp is the most efficient, transmitting only ~46.85K parameters. LoRA follows with ~1.08M parameters, which is two orders of magnitude lower than FFT (427.62M).
• Computational Complexity:
  • Without Caching: FFT and encoder-specific tuning have similar high FLOPs. CoOp is slightly more efficient due to fewer operations in text feature construction.
  • With Caching: Encoder-specific tuning benefits significantly from caching frozen features (e.g., text features in image-encoder tuning), reducing FLOPs by ~387B compared to FFT. LoRA does not benefit from caching as effectively because injected modules alter the feature flow, resulting in slightly higher complexity than FFT.

4. Significance and Guidelines

The paper establishes that while VLMs are powerful for FL in RS, the choice of adaptation strategy dictates the system's feasibility and performance. The authors derive the following guidelines based on operational constraints:

1. Text Encoder Fine-Tuning: Recommended when the primary goal is transferring knowledge to related RS datasets. It offers the best cross-dataset generalization among fine-tuned models with moderate communication costs.
2. Prompt Learning (CoOp): Recommended only when communication bandwidth is the dominant constraint. It minimizes overhead but sacrifices task accuracy and generalization.
3. LoRA: Recommended as the optimal trade-off for scenarios requiring both high performance and communication efficiency. It achieves competitive in-distribution results while reducing communication overhead by two orders of magnitude compared to FFT and mitigating catastrophic forgetting.
4. FFT + PAINT: Recommended when extending a zero-shot model to a new task while strictly preserving performance on previously learned tasks, despite the high communication cost.

5. Conclusion

The study concludes that parameter-efficient adaptation strategies, particularly LoRA, are well-suited for federated VLM training in remote sensing. They effectively mitigate communication costs and catastrophic forgetting while maintaining competitive performance. Conversely, unrestricted Full Fine-Tuning, while effective for in-distribution tasks, leads to significant loss of generalization capabilities, limiting its applicability in settings where cross-domain robustness is required. Future work aims to extend this analysis to domain-specific foundation models and multispectral sensor modalities.