🤖 machine learning

Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning

This paper proposes and evaluates a federated fine-tuning framework using parameter-efficient methods (LoRA, QLoRA, IA3) on private, non-IID data across healthcare and finance domains, demonstrating that it achieves performance comparable to centralized training while outperforming isolated learning and offering energy-efficient solutions for adapting large language models without sharing sensitive data.

Original authors: Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris, Joaquin del Rio, Oleksii Sliusarenko, Xabi Uribe-Etxebarria

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel M. Jimenez-Gutierrez, Enrique Zuazua, Georgios Kellaris, Joaquin del Rio, Oleksii Sliusarenko, Xabi Uribe-Etxebarria

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Unlocking the "Locked Vaults" of Knowledge

Imagine that the world's most valuable information (like detailed patient medical histories or private bank customer records) is locked inside thousands of separate, high-security vaults. These vaults belong to different hospitals and banks.

Currently, the "smartest" AI models (Large Language Models or LLMs) have been trained on public data, like books and websites available to everyone. But to get truly good at specific jobs—like diagnosing rare diseases or spotting financial fraud—they need to learn from those locked vaults.

The Problem: The vaults cannot be opened and merged into one giant room because of privacy laws and security rules. You can't move the data; it has to stay where it is.

The Solution: This paper proposes a way for these vaults to "talk" to each other without ever opening their doors. They use a method called Federated Learning. Think of it like a group of chefs who each have a secret family recipe. Instead of sending their recipes to a central kitchen (which is forbidden), they each cook a small part of the dish in their own kitchen, send only the taste notes to a head chef, who mixes the notes together to create a master recipe, and then sends that master recipe back to everyone. No one ever sees the other's ingredients, but everyone ends up with a better dish.

The Experiment: A Race Between Three Teams

The researchers set up a competition to see how well this "recipe-sharing" method works compared to two other ways of training AI. They tested this in two high-stakes fields: Medicine (answering medical exam questions) and Finance (analyzing financial news sentiment).

They compared three teams:

The "All-in-One" Team (Centralized): They broke the rules and moved all the data into one giant room to train the AI. This is the "gold standard" for performance, but it's usually illegal in real life.
The "Lone Wolf" Team (Single-Institution): Each vault trained its own AI using only its own tiny pile of data. They didn't talk to anyone.
The "Collaborative" Team (Federated): This is the paper's main focus. The vaults kept their data locked but shared their "learning notes" (model updates) to build one smart AI together.

The Results: The Collaborative Team Wins (Almost)

The findings were very encouraging:

Better than being alone: The "Collaborative Team" (Federated) almost always did much better than the "Lone Wolf" teams. By sharing their learning notes, they learned from each other's unique experiences.
Almost as good as the "All-in-One" Team: Surprisingly, the Collaborative Team performed nearly as well as the team that had all the data in one room. They didn't lose much accuracy, even though they never saw the raw data from other vaults.
The "Non-IID" Challenge: In the real world, data isn't perfectly mixed. One hospital might see mostly heart patients, while another sees mostly orthopedic cases. The researchers simulated this "messy" reality. Even with this uneven data, the Collaborative Team still managed to learn effectively.

The Tools: Choosing the Right "Cooking Utensils"

To make this work without needing supercomputers in every vault, the researchers used special "efficient" training tools called PEFT (Parameter-Efficient Fine-Tuning). They tested three different tools, which they compared to different types of kitchen gadgets:

LoRA (The Heavy-Duty Mixer): This tool gave the best results in terms of pure accuracy. It's like a powerful mixer that makes the perfect cake, but it requires a lot of electricity (memory) and sends large notes between the kitchens (communication cost).
QLoRA (The Energy-Saving Blender): This is a version of the mixer that uses less electricity. It was almost as good as the heavy-duty mixer but required significantly less memory. This is a huge win for hospitals or banks that don't have massive servers.
IA3 (The Tiny Spoon): This tool sends the smallest notes between kitchens, making it the fastest to communicate. However, it didn't make the cake taste quite as good as the other two. It's great if you have a slow internet connection, but you might sacrifice a little bit of quality.

The Green AI Angle: Saving Energy

The paper also looked at the "Green" aspect. Training AI is energy-intensive.

QLoRA was the winner for saving memory (electricity), making it a "greener" choice for running these models on smaller devices.
IA3 was the winner for saving bandwidth (communication), making it efficient for networks with limited speed.

The Bottom Line

This paper proves that we don't need to break privacy laws to build smarter AI. By using Federated Fine-Tuning, institutions can collaborate to create powerful, specialized AI models that understand medicine and finance deeply, all while keeping private data locked safely inside their own walls.

Key Takeaway: You can have your cake (high performance) and eat it too (keep data private), provided you choose the right tool (like QLoRA or LoRA) for the job.

Technical Summary: Towards the Next Frontier of LLMs, Training on Private Data

Problem Statement

The development of Large Language Models (LLMs) has historically relied on vast public datasets. However, the next frontier for LLM advancement lies in accessing private, institutionally held data, particularly in highly regulated sectors like healthcare and finance. These domains contain high-value information (e.g., patient histories, customer communications) that cannot be centralized due to privacy, regulatory, and organizational barriers. Furthermore, institutional datasets are typically non-independent and identically distributed (non-IID), exhibiting heterogeneity in population characteristics, data modalities, and label distributions.

The core challenge addressed is how to adapt LLMs to these sensitive, distributed data silos without exchanging raw data. While Federated Learning (FL) offers a paradigm for collaborative model training without data sharing, its application to LLM fine-tuning faces specific hurdles:

Non-IID Data: Statistical heterogeneity across nodes can degrade global model performance.
Resource Constraints: LLMs require significant memory and communication bandwidth, which is exacerbated in federated settings.
Lack of Controlled Comparisons: Existing benchmarks often focus on single domains or lack controlled comparisons across training scenarios (centralized vs. federated vs. isolated) and Parameter-Efficient Fine-Tuning (PEFT) strategies under identical protocols.

Methodology

Experimental Framework

The authors present a controlled experimental benchmark comparing three training scenarios across two sensitive domains (Medicine and Finance):

Centralized: All data pooled into a single repository for fine-tuning.
Single-Institution: Each node trains independently on its local data without collaboration.
Federated Learning (FL): Nodes collaboratively fine-tune a shared model by exchanging only trainable parameters, keeping raw data local.

Datasets and Tasks

The benchmark utilizes four closed-ended datasets:

Medicine: MedQA (USMLE-style questions) and MedMCQA (multi-subject medical entrance exams).
Finance: FPB (Financial PhraseBank for sentiment classification) and FiQA-SA (Financial sentiment analysis).

Non-IID Simulation

To reflect real-world institutional heterogeneity, the training splits were partitioned across five nodes using a Dirichlet distribution protocol. This creates controlled label-distribution shifts (e.g., varying proportions of answer options A, B, C, D) across nodes, with concentration parameters ( $\alpha$ ) selected to ensure a noticeable level of non-IID data (Hellinger distance $\geq 0.5$ ).

PEFT Strategies and Models

The study evaluates three representative PEFT methods:

LoRA (Low-Rank Adaptation): Updates low-rank decomposition matrices.
QLoRA (Quantized LoRA): Combines LoRA with 4-bit quantization of the backbone.
IA3 (Infused Adapters by Inhibiting and Amplifying Inner Activations): Scales internal activations via learned vectors.

Experiments were conducted on a diverse set of pretrained backbones, including domain-specialized models (e.g., II-Medical-8B-1706, FineMedLM-o1) and general-purpose instruction-tuned models (e.g., Qwen3-8B, Llama-3.1-8B-Instruct, Gemma-2-9b-it).

Evaluation Metrics

Accuracy: The primary metric for closed-ended QA and classification.
Efficiency:
- Communication Cost: Bidirectional volume of trainable parameters transmitted per round.
- Memory Footprint: Peak reserved GPU memory during training.

Key Results

1. Performance Comparison

Federated vs. Centralized: Federated fine-tuning consistently achieves performance close to centralized training, often reproducing a large fraction of the centralized accuracy. In many configurations, the gap is marginal.
Federated vs. Single-Institution: Federated learning significantly outperforms isolated single-institution learning, demonstrating that collaborative adaptation effectively leverages complementary information across nodes despite non-IID constraints.
Domain Specificity:
- Medicine: Domain-specialized backbones (specifically II-Medical-8B-1706) consistently outperformed general-purpose models across all scenarios.
- Finance: Results were more heterogeneous, with no single model dominating all benchmarks. Performance varied by task and backbone (e.g., Qwen3-8B and Gemma-2-9b-it showed strong results).

2. PEFT Strategy Effectiveness

Accuracy: LoRA generally provided the most robust and highest accuracy across both domains. QLoRA remained a close second, often matching LoRA's performance with minor degradation. IA3 adapters generally trailed behind in accuracy, particularly in medical benchmarks.
Efficiency Trade-offs:
- Memory: QLoRA was systematically the most memory-efficient among high-performing methods, reducing GPU memory usage by approximately 25-30% compared to LoRA (e.g., Qwen3-8B dropped from ~28.7 GB to ~20.7 GB).
- Communication: IA3 adapters required significantly less communication (often two orders of magnitude lower than LoRA/QLoRA) due to the smaller size of the trainable parameters, making them the most communication-efficient option.

3. Impact of Non-IID Data

Despite the explicit introduction of statistical heterogeneity via Dirichlet partitioning, the federated models did not suffer severe performance degradation. This suggests that LLM fine-tuning is robust to realistic levels of non-IID data when using appropriate PEFT strategies and backbones.

Significance and Claims

The paper positions itself as a controlled experimental benchmark rather than a proposal of new algorithms or optimization methods. Its primary contributions and claims are:

Feasibility of Federated PEFT: The study demonstrates that federated fine-tuning is a viable mechanism for adapting LLMs to private data silos. It allows institutions to benefit from collaborative learning without violating data privacy constraints or centralizing raw data.
Benchmarking Gap: It addresses a gap in the literature by providing a unified comparison of centralized, single-institution, and federated settings across multiple domains, tasks, and PEFT strategies under identical experimental protocols.
Green AI Perspective: The work highlights that efficiency in LLM adaptation is not solely about model size but also about the choice of fine-tuning strategy. It provides empirical evidence that QLoRA offers an optimal balance for memory-constrained deployments, while IA3 is superior for communication-constrained environments.
Privacy Scope Clarification: The authors explicitly state that while FL prevents raw data exchange, the benchmark does not evaluate formal privacy guarantees against model inversion or gradient leakage attacks. The focus remains on the empirical performance of data-local adaptation.

In conclusion, the paper argues that federated PEFT can enable the "next leap" in LLM development by unlocking the potential of private, heterogeneous institutional data, provided that the choice of backbone model and PEFT strategy is aligned with specific deployment constraints (accuracy, memory, or communication).