💻 computer science

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

This paper proposes FedDTL, a novel federated learning framework for Vision-Language Models that decouples image and text encoders to ensure coherent global updates and employs a two-stage local fine-tuning strategy combining supervised learning with reinforcement learning to effectively balance global task adaptation and generalization under heterogeneous data distributions.

Original authors: Yuting Ma, Lechao Cheng, Xiaohua Xu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuting Ma, Lechao Cheng, Xiaohua Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of friends trying to learn a new skill together, like identifying different types of birds, but they live in different cities and cannot share their private photo albums. This is the core idea of Federated Learning: everyone learns locally on their own devices and only shares what they've learned, not their raw data.

Now, imagine these friends are using a super-smart, pre-trained "bird expert" (a Vision-Language Model) to help them. The problem is, if everyone just studies their own local photos and then tries to average their notes, things get messy. Some friends might become too obsessed with the specific birds in their own backyard (over-specialization), while others might learn completely different ways of describing birds, making it impossible to combine their knowledge into one useful guide (inconsistency).

The paper introduces a new method called FedDTL to fix this. Here is how it works, broken down into simple concepts:

1. The "Split Team" Strategy (Decoupled Training)

In most methods, every friend tries to learn both how to see the bird (Image Encoder) and how to describe it in words (Text Encoder) on their own. This leads to confusion because everyone's descriptions drift apart.

FedDTL splits the job:

The Local Artists (Clients): Each friend keeps their camera and photo album. They train their "Image Encoder" locally to recognize the birds in their own photos. This keeps their private photos safe.
The Central Librarian (Server): A central server trains the "Text Encoder." Instead of looking at photos, the server just learns the names of the birds (e.g., "Cardinal," "Blue Jay").
The Connection: The friends send their "visual understanding" to the server, and the server sends back the "textual definitions." It's like the friends send sketches to the librarian, and the librarian sends back the correct dictionary definitions. This ensures everyone is using the same "language" to describe what they see, keeping the group aligned without anyone seeing each other's private photos.

2. The "Two-Step Dance" (Two-Stage Local Fine-Tuning)

Even with the split team, if a friend studies their local photos for too long, they might start memorizing the specific lighting or background of their own garden, forgetting how to recognize the bird in a different setting. This is called "over-specialization."

FedDTL solves this with a two-step training process for each friend:

Step 1: The Warm-Up (Supervised Fine-Tuning - SFT):
First, the friend does a quick, standard study session. They look at their photos and learn to match them to the bird names provided by the Central Librarian. This gets them up to speed quickly and reliably. Think of this as memorizing the flashcards.
Step 2: The Reinforcement Boost (Reinforcement Learning - RL):
After the warm-up, the friend switches to a "trial and error" mode. Instead of just memorizing, they are rewarded for being general rather than just specific.
- The Analogy: Imagine the friend is playing a game where they have to guess the bird. If they guess correctly, they get a point. But here's the twist: they are encouraged to guess correctly even when the photo is slightly blurry or has a weird angle.
- The paper uses a technique called GRPO (Group Relative Policy Optimization). It's like having a group of friends guess the same bird at the same time. If one friend guesses correctly while others guess wrong, that friend gets a "bonus" for being more general. This stops them from just memorizing the exact pixels of their local photos and forces them to learn the true essence of the bird, making them better at recognizing birds they've never seen before.

Why is this a big deal?

Previous methods tried to fix these problems by adding complex rules or just averaging everyone's notes. But in "Full-Data" settings (where friends have lots of photos, not just a few), those old methods failed. They either made the group too inconsistent or made individuals too obsessed with their own data.

FedDTL claims to be the first to successfully balance two things at once:

Global Adaptation: The group learns the task well together (everyone agrees on what a "Cardinal" is).
Generalization: The group can still recognize birds in new, unseen environments (not just the specific backyard photos they trained on).

The Results

The authors tested this on many different datasets (like identifying flowers, pets, and food) under various difficult conditions (some friends having very different types of data than others). They found that FedDTL consistently outperformed other methods, especially when there was a lot of data to process. It managed to keep the group unified while ensuring no single friend became too "stubborn" about their local data.

In short, FedDTL is a smarter way for a distributed group to learn together: it separates the "seeing" from the "naming" to keep everyone aligned, and it uses a two-step training process to ensure they learn the general rules of the game, not just the specific details of their own backyard.

Technical Summary: Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning (FedDTL)

Problem Statement

Federated Learning (FL) combined with pre-trained Vision-Language Models (VLMs), such as CLIP, offers a promising paradigm for downstream task adaptation while preserving data privacy. However, existing approaches relying on fully local optimization with simple parameter aggregation face two critical challenges, particularly under heterogeneous data distributions and full-data regimes:

Inter-client Optimization Inconsistency: Under Non-IID settings (e.g., label skew and feature shift), clients optimize mismatched local objectives. This leads to misaligned local gradients and "client drift," where clients converge toward different representation coordinate systems. Consequently, simple parameter averaging fails to produce a coherent global semantic update, degrading global task adaptation.
Intra-client Over-specialization: In full-data regimes with long local training trajectories, lightweight parameter-efficient fine-tuning (PEFT) modules tend to become overly specialized to a specific client's biased label frequencies and feature statistics. While this improves local fitting, it severely compromises generalization to unseen classes or shifted domains.

Current methods often address these issues via regularization or alignment constraints but remain insufficient because they rely on local-only optimization. Furthermore, many evaluations are limited to few-shot settings, potentially underestimating the compounded negative effects in realistic full-data scenarios.

Methodology: FedDTL

The authors propose FedDTL, a novel framework that integrates a decoupled encoder training scheme with a two-stage local fine-tuning strategy to balance global task adaptation and generalization.

1. Decoupled Encoder Training

Inspired by the modality decoupling in CLIP, FedDTL separates the training of image and text encoders across the client-server architecture:

Local Image Encoder: The image encoder ( $V$ ) processes private raw images and is trained locally on each client to preserve data privacy.
Global Text Encoder: The text encoder ( $T$ ), which operates on category-level textual descriptions (e.g., "a photo of a [classname]"), is optimized centrally on the server.
Mechanism: Clients align their local visual representations with the server's globally consistent textual embeddings. This server-client modality alignment acts as a semantic constraint, reducing inter-client optimization inconsistency without requiring clients to maintain local text encoders.
Efficiency: Both encoders utilize LoRA (Low-Rank Adaptation) tuning on frozen backbones to minimize computational overhead and communication costs.

2. Two-Stage Local Fine-Tuning

To mitigate intra-client over-specialization, FedDTL employs a sequential local training process:

Stage 1: SFT-based Task Adaptation (Warm-start): Clients perform Supervised Fine-Tuning (SFT) on their local image encoders using cross-entropy loss. This stage provides a fast and reliable optimization warm-start, rapidly adapting the model to the local task distribution.
Stage 2: RL-based Generalization Enhancement: Following SFT, clients switch to a Reinforcement Learning (RL) stage inspired by Group Relative Policy Optimization (GRPO).
- Stochasticity Injection: Since image encoders produce deterministic embeddings, the authors inject unbiased Gaussian noise into the latent embedding space to induce stochasticity for action sampling.
- Reward & Policy: The policy model is updated based on an accuracy-based reward (correct classification = 1, else 0). The objective minimizes a policy gradient loss with $\epsilon$ -clipping and a KL-divergence regularization term against a hybrid reference model (combining the final SFT model and the latest global policy).
- Goal: This stage refines the local update trajectory to prevent over-memorization of local data, thereby enhancing generalization to unseen classes and domains.

3. Global Coordination

After local fine-tuning, clients upload updated LoRA parameters and normalized image embeddings (specifically class tokens) to the server. The server aggregates visual LoRA parameters via weighted averaging and updates the global text encoder using the collected embeddings, completing the FL round.

Key Contributions

Decoupled Training Scheme: A novel architecture where the image encoder is optimized locally for privacy, while a centralized text encoder provides a global semantic anchor. This design reduces inter-client inconsistency and improves global task adaptation across diverse FL data distributions.
Two-Stage Local Fine-Tuning: A strategy combining SFT for rapid adaptation and a GRPO-inspired RL stage to suppress intra-client over-specialization. This approach stabilizes local training and enhances generalization in heterogeneous and full-data settings.
Comprehensive Evaluation: Extensive experiments across nine benchmarks under both few-shot and full-data regimes, covering label skew and feature shift. The results demonstrate that FedDTL effectively balances global task adaptation and generalization, outperforming prior federated VLM baselines.

Experimental Results

The paper evaluates FedDTL on datasets including CIFAR-10/100, EuroSAT, Tiny-ImageNet, Oxford-Pet, Flower102, Caltech101/256, Food101, Office-Caltech10, and DomainNet.

Label Skew (Base-to-Novel Generalization): In full-data Non-IID settings, FedDTL achieves the highest accuracy on base classes (e.g., 91.64% vs. 80.56% for FedMaPLe on CIFAR-100) while maintaining competitive performance on novel classes. It successfully avoids the severe generalization degradation seen in baselines like pFedDC and pFedMMA, which suffer from over-specialization in full-data regimes.
Feature Shift: In cross-domain settings (Office-Caltech10, DomainNet), FedDTL achieves the highest average accuracy in almost all configurations, including those with combined feature shift and label skew.
Ablation Studies:
- Removing the decoupled encoder scheme significantly drops base accuracy in Non-IID settings, confirming its role in resolving inter-client inconsistency.
- Removing the RL stage leads to lower novel accuracy, confirming its effectiveness in preserving generalization.
- The two-stage strategy (SFT + RL) outperforms using SFT or RL alone, validating the complementary nature of rapid adaptation and generalization enhancement.
Privacy & Communication: The framework reduces communication costs by uploading only class token embeddings. Adding noise perturbation to these embeddings provides additional privacy protection with minimal performance loss.

Significance and Claims

The paper claims that FedDTL addresses a critical gap in federated VLM research: the inability of existing methods to simultaneously handle inter-client inconsistency and intra-client over-specialization, particularly in realistic full-data scenarios.

Systematic Resolution: Unlike prior works that rely on simple parameter averaging or local-only regularization, FedDTL systematically resolves inconsistency through server-client modality alignment and over-specialization through RL-based refinement.
Robustness: The framework demonstrates robustness across varying data distributions (IID, Dirichlet, Non-IID) and learning regimes (few-shot to full-data).
Practicality: By leveraging LoRA and a two-stage schedule, the method balances model performance with computational efficiency, making it suitable for edge devices.

The authors conclude that their approach provides a more stable and effective path for adapting pre-trained VLMs in federated environments, ensuring that global models remain both adaptable to specific tasks and generalizable to unseen scenarios.