💬 NLP

Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

This paper investigates the impact of client language composition on multilingual Federated Learning of Large Language Models, demonstrating that increasing within-client multilinguality yields fairer and more effective global models, particularly for low-resource languages, while introducing a novel Local Dynamic Early Stopping mechanism to enhance training efficiency.

Original authors: Aleix Sant, Jordi Luque, Carlos Escolano

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aleix Sant, Jordi Luque, Carlos Escolano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of 8 different chefs how to cook a single, amazing "World Cuisine" dish.

In a traditional cooking school (Centralized Learning), you would gather all the ingredients from every chef into one giant kitchen, mix them all together, and teach them all at once. This usually produces the best result, but it breaks the rule: Chefs can't share their secret family recipes.

This is where Federated Learning (FL) comes in. Instead of sharing ingredients, each chef cooks in their own private kitchen. They only send you the lessons they learned (the "weights" or "gradients"), not the recipes themselves. You then combine these lessons to create a master cookbook.

However, there's a catch in this paper: The chefs speak different languages.

The Problem: The Language Barrier

In this study, the researchers looked at what happens when the chefs have different "language diets":

The Monolingual Chef: Chef A only speaks Spanish. Chef B only speaks German. They never mix.
The Multilingual Chef: Chef A speaks Spanish, German, and French. Chef B does the same.

The big question the paper asks is: Does it matter if our chefs only know one language, or if they know many?

The Solution: A Smarter Way to Stop Cooking

The researchers also introduced a new tool called LDES-FL (Local Dynamic Early Stopping).

Think of this like a smart timer for each chef.

Old Way: The head chef says, "Stop cooking when everyone is done." If one slow chef is still struggling, everyone waits, wasting time and electricity.
New Way (LDES-FL): Each chef has their own timer. If Chef A finishes their Spanish dish and realizes, "I can't make it any better," they stop cooking. But, if Chef B sends over a new tip that helps Chef A improve their Spanish dish again, Chef A wakes up and starts cooking again!

This saves energy and makes the process much more efficient.

The Experiments: What Happened?

The researchers ran many simulations using a model called Salamandra-2b (a multilingual AI brain) and tested different scenarios:

The "Specialist" Approach (Monolingual Clients):
- Each client only had data in one language.
- Result: If you wanted the best possible Spanish model, you should just train a specialist on Spanish data alone. It's the most effective for one specific language.
- Downside: If you tried to combine them into one global model, the result was messy. The model got confused because the "Spanish" updates clashed with the "German" updates.
The "Generalist" Approach (Multilingual Clients):
- Each client had a mix of languages (e.g., 85% Spanish, 15% German).
- Result: The global model became much smarter and fairer. It learned to speak all languages well, not just a few.
- The Surprise: The "low-resource" languages (like Serbian or Basque, which have less data in the world) improved the most when clients were multilingual. It's like a small language getting a huge boost from the attention of a larger one.

The Trade-Off: Speed vs. Quality

Here is the "taste test" conclusion:

Monolingual Clients (The Sprinters): They finish training very fast. They stop early because they hit a "local peak" (they think they are done). But the final model is a bit unbalanced; it's great at some languages and terrible at others.
Multilingual Clients (The Marathon Runners): They take longer to train. They keep going back and forth, refining their skills. But the final result is a superior, balanced model that treats all languages fairly.

The Big Takeaway

The paper tells us that how we mix the data at the client level is the most important design choice.

If you want a model that is perfect at one specific language, train it alone.
If you want a fair, high-quality model that speaks many languages, you must make sure your clients are multilingual. Even if it takes more computing power and time, the result is a much better "World Cuisine" dish.

In short: To build a truly global AI, don't just ask people to teach you their own language in isolation. Ask them to teach you a little bit of everything. It takes longer, but the final result is a much more harmonious and capable world.

1. Problem Statement

The paper addresses the challenges of applying Federated Learning (FL) to Large Language Models (LLMs) in multilingual environments. Key issues include:

Data Heterogeneity (Non-IID): In real-world FL, clients often possess data in different languages (monolingual clients), creating highly skewed distributions. This leads to client drift, where local updates diverge from the global objective, hindering convergence and degrading generalization.
Resource Constraints: Training LLMs in FL is computationally expensive. Standard FL workflows often require all clients to train for the same number of rounds, leading to inefficiency when some clients converge faster than others.
Multilingual Fairness: There is a lack of understanding regarding how the language composition of client datasets (ranging from fully monolingual to fully multilingual) impacts the performance, fairness, and efficiency of the resulting global model.

2. Methodology

A. Framework Extension

The authors extended the FederatedScope-LLM framework to support multilingual instruction-tuning. Key modifications included:

Multilingual Data Pipelines: Support for flexible prompt integration and language-aware sample processing.
Client Composition Scenarios: They designed experimental settings where the language distribution within each client's local dataset was systematically varied.
- 100% Mono: Each client holds data in a single, unique language (high heterogeneity).
- Mixed/Multilingual: Clients hold a mix of languages (e.g., 85% mono, 15% multi), reducing cross-client heterogeneity and approximating an IID distribution.
- 15% Mono: Clients hold mostly multilingual data (low heterogeneity).

B. Parameter-Efficient Fine-Tuning (PEFT)

To mitigate computational and communication costs, the study utilized LoRA (Low-Rank Adaptation).

Model: salamandra-2b-instruct (a multilingual LLM).
Configuration: LoRA rank $r=16$ , scaling factor $\alpha=32$ . Only the LoRA matrices ( $A_i, B_i$ ) were trained and aggregated, while the backbone remained frozen.

C. Local Dynamic Early Stopping (LDES-FL)

A novel mechanism was introduced to improve training efficiency:

Mechanism: Each client monitors its local validation loss. If no improvement is observed for a set number of rounds (patience), the client pauses local training but continues to download and evaluate the global model.
Adaptive Rejoining: If the downloaded global model improves the client's local validation loss, the client resumes training.
Termination: The global training stops only when all clients have paused.
Benefit: This allows clients to stop training when they have converged locally, reducing unnecessary computation steps without sacrificing performance.

D. Experimental Setup

Dataset: A multilingual version of the Alpaca Cleaned dataset covering 8 languages (English, Spanish, German, Catalan, Danish, Serbian, Croatian, Basque).
Baselines:
- Base Model: Pre-trained salamandra-2b-instruct.
- Local FT (Mono): Centralized fine-tuning on a single language per client.
- Local FT (Multi): Centralized fine-tuning on the union of all languages (Upper Bound).
- FedAvg: Standard Federated Averaging with LDES-FL.

3. Key Contributions

Multilingual FL Framework: Extended FederatedScope-LLM to handle complex multilingual scenarios with flexible prompt integration and data pipelines.
LDES-FL Algorithm: Proposed a client-specific early stopping mechanism that enables dynamic pausing and resuming of training, significantly reducing optimization steps while maintaining performance.
Systematic Analysis of Client Composition: Provided empirical evidence on how varying the degree of multilinguality within client datasets affects global model quality, fairness, and convergence.

4. Key Results

A. Impact of Client Language Composition

Performance vs. Heterogeneity: As the proportion of multilingual data within clients increases (moving from 100% mono to 15% mono), average multilingual performance improves, and cross-lingual disparities decrease (higher fairness).
The Trade-off:
- Monolingual Clients (100% mono): Converge faster (fewer optimization steps) but result in a biased global model with lower overall quality and higher variance between languages.
- Multilingual Clients (15% mono): Require significantly more optimization steps (approx. 6.8x more than 100% mono) but yield a stronger, fairer, and more balanced global model.
Comparison to Local Fine-Tuning:
- Monolingual Local FT: Best for specializing in a single target language but fails to generalize well to other languages.
- Federated Training: Produces a single model that outperforms the average of monolingual local models across all languages, offering a better balance of quality and coverage.
- Centralized Multilingual FT: Remains the upper bound (best overall performance), but Federated Learning achieves a strong approximation of this without centralizing data.

B. Benefits for Low-Resource Languages

The study found that increasing client multilinguality disproportionately benefits low-resource languages (e.g., Serbian, Croatian, Basque).

Moving from 100% mono to 15% mono resulted in a +16.11% gain in ROUGE-L for low-resource languages, compared to only +4.78% for high-resource languages.
This suggests that multilingual FL effectively mitigates performance gaps between resource-rich and resource-poor languages.

C. Efficiency of LDES-FL

LDES-FL reduced the total number of optimization steps by ~22% in the 100% mono setting and ~32% in the 15% mono setting compared to standard federated early stopping, while maintaining comparable performance.
Federated training was significantly cheaper than training separate monolingual models for all 8 languages (up to 43x fewer steps for the cheapest configuration).

5. Significance and Conclusion

Design Variable: The paper establishes client language composition as a critical design variable in multilingual FL. Practitioners must choose between training efficiency (monolingual clients) and model quality/fairness (multilingual clients).
Mitigating Client Drift: The results confirm that increasing within-client multilinguality reduces client drift by aligning local objectives more closely with the global objective.
Practical Implication: For organizations aiming to deploy a single, fair, and high-performing multilingual LLM across diverse regions, federated training with multilingual clients is superior to maintaining separate monolingual models, despite the higher computational cost.
Future Work: The authors suggest using strictly controlled multilingual splits to better isolate true cross-lingual transfer effects in future research.

In summary, the paper demonstrates that while federated learning incurs a "decentralization tax" compared to centralized training, strategic client composition (making clients more multilingual) and efficient stopping mechanisms (LDES-FL) can yield robust, fair, and computationally viable multilingual LLMs.