💬 NLP

FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Health

FedMentor is a privacy-preserving federated fine-tuning framework that combines Low-Rank Adaptation (LoRA) with domain-aware Differential Privacy to enable the safe, efficient, and high-utility adaptation of Large Language Models for mental health applications across heterogeneous clients.

Original authors: Nobin Sarwar, Shubhashis Roy Dipta

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nobin Sarwar, Shubhashis Roy Dipta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of therapists, each working in a different neighborhood, who want to build a single, super-smart AI assistant to help people with mental health struggles. They all have private journals (data) from their clients, but they can't share these journals because of strict privacy laws and the need to keep client secrets safe.

This is the problem FedMentor solves. It's like a clever way for these therapists to learn from each other's experiences without ever actually handing over the private journals.

Here is how it works, broken down into simple concepts:

1. The "Specialized Notebooks" (LoRA)

Usually, training a smart AI is like trying to rewrite an entire encyclopedia every time you learn something new. That takes forever and requires massive computers.

FedMentor uses a trick called LoRA (Low-Rank Adaptation). Imagine instead of rewriting the whole encyclopedia, each therapist just writes a small, sticky-note-sized "cheat sheet" (an adapter) with their specific insights.

The Benefit: These cheat sheets are tiny. They are so small that even a standard laptop (or a single graphics card) can handle them. This means the AI can be trained on smaller, less powerful devices, making it practical for real-world use.

2. The "Noise Machine" (Differential Privacy)

Even if therapists only share their cheat sheets, a sneaky observer might still be able to guess what a specific client wrote by looking at the notes. To stop this, FedMentor uses Differential Privacy.

Think of this as a static noise machine. Before a therapist sends their cheat sheet to the central server, they run it through a machine that adds a little bit of "static" or "fog" to the notes.

The Magic: This static is calculated so perfectly that the overall lesson the AI learns remains accurate, but it becomes mathematically impossible to tell which specific note came from which specific client. It's like listening to a choir: you can hear the beautiful harmony (the useful knowledge), but you can't pick out the voice of any single singer (the private data).

3. The "Smart Budget" (Domain-Aware Privacy)

Not all neighborhoods have the same level of risk. Some clients might be in crisis (high sensitivity), while others are just chatting about their day (lower sensitivity).

FedMentor is "domain-aware." It acts like a smart budget manager:

High-Risk Areas: For the most sensitive topics (like self-harm), the system adds more static noise. It's like putting a thicker layer of fog over those specific notes to ensure maximum secrecy.
Lower-Risk Areas: For less sensitive topics, it adds less noise so the AI can learn more clearly.
The Safety Net: If the AI starts getting too confused by the noise and its answers become bad, the system automatically turns down the static just a tiny bit to keep the quality high, but never enough to break privacy.

4. The Results: Safe, Smart, and Efficient

The researchers tested this system on three different mental health datasets (like different types of therapy journals). Here is what they found:

Safer Answers: The AI trained with FedMentor was much better at avoiding toxic or harmful responses. In one test, the rate of "safe" answers went up by about 2%, and the "toxicity" (bad language) went down significantly.
Just as Smart: Despite all the privacy noise, the AI was almost as smart as one trained on all the data at once (which is usually the "gold standard"). It didn't lose much ability to understand language or answer correctly.
Lightweight: Because they only shared the tiny "cheat sheets" instead of the whole AI brain, the amount of data sent back and forth was tiny (less than 173 MB per round). This means it works even on devices with limited internet or memory.

The Bottom Line

FedMentor is a practical blueprint for building mental health AI that respects privacy. It allows many different groups to collaborate and build a safer, smarter tool without ever exposing the private, sensitive stories of the people they are trying to help. It proves you don't have to choose between privacy and performance; you can have both.

Technical Summary: FedMentor

Problem Statement

The deployment of Large Language Models (LLMs) for mental health applications faces a critical triad of challenges: strict privacy requirements (e.g., HIPAA, GDPR), the heterogeneity of user data across different clinical domains, and the computational constraints of resource-limited healthcare settings. Traditional centralized training is often infeasible due to the sensitivity of mental health data, which may include explicit self-harm ideation or clinical signals requiring strong confidentiality. Furthermore, existing Federated Learning (FL) approaches often struggle with non-IID (non-independent and identically distributed) data shifts across domains and fail to provide granular privacy guarantees tailored to specific data sensitivities. There is a need for a framework that enables collaborative fine-tuning of LLMs while preserving user privacy, maintaining model utility, and ensuring communication efficiency on edge devices.

Methodology: FedMentor

The authors propose FedMentor, a federated fine-tuning framework designed specifically for heterogeneous mental health domains. The framework integrates three core components:

Federated Low-Rank Adaptation (LoRA): Instead of transmitting full model weights, FedMentor freezes the global LLM backbone and only trains and communicates lightweight LoRA adapters. This significantly reduces communication overhead and memory requirements, enabling the training of models with up to 1.7 billion parameters on single-GPU clients.
Domain-Aware Differential Privacy (DP): Recognizing that different mental health datasets possess varying levels of sensitivity, FedMentor assigns distinct privacy budgets ( $\epsilon, \delta$ $ϵ, δ$ ) to each client domain.
- Budget Allocation: High-sensitivity domains (e.g., Interpersonal Risk Factors) receive stricter budgets (lower $\epsilon$ ), while lower-sensitivity domains receive looser budgets.
- Noise Calibration: The framework employs a layer- and adapter-aware noise injection mechanism. It scales Gaussian noise based on the network layer (early vs. late) and the specific LoRA matrix (A vs. B), acknowledging that early layers and LoRA-A matrices are more sensitive to perturbation.
Adaptive Utility-Privacy Feedback Loop: To prevent excessive privacy noise from degrading model performance, the server monitors proxy utility metrics (e.g., BERTScore, Safe Rate). If the aggregate utility falls below a predefined threshold ( $\tau$ ), the server adaptively reduces the noise scale for subsequent rounds, dynamically trading off privacy strictness for clinical utility.

The training pipeline involves clients training local LoRA adapters on frozen backbones, adding domain-specific Gaussian noise to the updates, and transmitting these noisy adapters to a central server. The server aggregates updates using a dataset-weighted FedAvg algorithm.

Key Contributions

The paper outlines three primary contributions:

Domain-Aware Private Adaptation: The formulation of a federated LoRA approach with per-domain privacy budgets that protects sensitive mental health text while preserving task utility. This approach raises safety metrics (e.g., increasing Toxicity Safe Rate) while keeping utility metrics (BERTScore F1, ROUGE-L) close to non-private baselines.
Robustness to Non-IID Heterogeneity: The demonstration of stable utility and fairness across diverse datasets (Dreaddit, IRF, MultiWD) without domain collapse. The adaptive noise mechanism ensures performance is maintained even under strict privacy budgets for the most sensitive domains.
Practical Efficiency via Adapters: By exchanging only LoRA adapter weights, the framework achieves significant compression (reducing communication costs by a factor of $\approx 10^{-3}$ ), making it feasible to deploy on resource-constrained clinical devices with single-GPU limitations.

Experimental Results

The framework was evaluated on three mental health datasets (Dreaddit, IRF, MultiWD) and five LLM backbones ranging from 350M to 1.7B parameters.

Safety and Privacy: FedMentor improved safety over standard Federated Learning without privacy. On the MultiWD dataset, the Toxicity Safe Rate (TSR) increased by 2 percentage points, and mean toxicity decreased from 2.92 to 1.98.
Utility Preservation: Despite the addition of differential privacy, utility metrics remained within 0.5% of the non-private FL baseline and close to the centralized upper bound. For instance, BERTScore F1 and ROUGE-L scores showed negligible degradation.
Efficiency: The framework successfully trained models up to 1.7B parameters on single-GPU clients. Communication costs were reduced to approximately 173 MB per round for the largest model, with peak memory usage staying within the limits of 80 GB GPUs.
Ablation Studies: Experiments confirmed that domain-specific budgets and adaptive noise are critical. Uniform privacy budgets improved fairness in safety outcomes, while static noise schedules led to reduced safety and increased disparity across clients.

Significance and Claims

The paper claims to be the first study to combine domain-aware Differential Privacy with federated LoRA fine-tuning specifically for LLMs in mental health. The authors position FedMentor as a practical solution for the "privacy-utility trade-off" that plagues current mental health AI deployments. By enabling collaborative learning without centralizing sensitive data, the framework aims to enhance trust and clinical applicability in high-risk domains.

The authors maintain a modest stance regarding their claims, acknowledging limitations such as the use of English-only social media datasets (treating each as a single client), reliance on automated safety metrics rather than clinician review, and the absence of demographic subgroup audits. They note that the study reflects a specific hardware environment (NVIDIA A100) and that future work will focus on scaling federations, incorporating clinician-in-the-loop approaches, and conducting multilingual evaluations.