💻 computer science

Improving Parameter-Efficient Federated Learning with Differentially Private Refactorization

This paper proposes FedPower, a differentially private cross-silo federated learning framework that enhances parameter-efficient fine-tuning by reconstructing full-rank updates and employing a novel PowerDP mechanism to inject noise before orthonormalization, thereby preserving matrix orthogonality and achieving superior accuracy-privacy trade-offs compared to existing methods.

Original authors: Linh Tran, Ana Milanova, Stacy Patterson

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linh Tran, Ana Milanova, Stacy Patterson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, super-smart robot (a Large Language Model) how to speak like a specific group of people—say, doctors or bankers—without ever seeing their private notes. This is the world of Federated Learning. Instead of sending the private notes to a central teacher, the robot sends a small "study guide" (model updates) from each person's local computer to the teacher, who combines them to learn.

To make this fast and cheap, we use a trick called LoRA (Low-Rank Adaptation). Think of LoRA like teaching the robot new vocabulary using only two small, thin notebooks (low-rank matrices) instead of rewriting its entire massive brain.

The Problem: The "Broken Math" and the "Noise"

The paper identifies two big problems when we try to protect privacy while using this method:

The Broken Math (Aggregation Error):
Imagine a group of chefs each writing down a recipe for a cake.
- The Ideal Way: You take every chef's entire finished cake recipe, mix them all together, and then divide by the number of chefs.
- The Current Broken Way (FedLoRA): You take all the "flour" notes from every chef, mix them. Then you take all the "sugar" notes, mix them. Then you multiply the mixed flour by the mixed sugar.
- The Result: Mathematically, mixing the ingredients before combining the recipes gives a different (and worse) cake than combining the recipes first. This creates "math errors" that make the final robot less smart.
The Privacy Noise (Differential Privacy):
To stop spies from figuring out which specific doctor or banker contributed which note, we add "static" or "noise" to the data. This is like adding a little bit of fog to the recipe notes so no one can see the exact details.
- The Disaster: When you add this fog to the "Broken Way" of mixing (mixing flour separately from sugar), the fog gets amplified. It's like trying to hear a whisper in a storm; the noise drowns out the signal, and the robot becomes very dumb.

The Solution: FedPower

The authors propose a new system called FedPower to fix this. Here is how it works, using a simple analogy:

1. The "Full-Size" Reassembly (Server-Side)
Instead of mixing the small notebooks separately, the teacher (server) takes the small notebooks from every student, glues them together to reconstruct the full-size, complete recipe for each student.

Why? This fixes the "Broken Math." Now the teacher has the true, complete picture of what each student learned.

2. The "Privacy Shield" (Clipping)
Before mixing these full recipes, the teacher checks them to make sure no single student's recipe is too huge or too weird. If it is, they trim it down to a standard size. This ensures that one student's data can't accidentally leak too much information.

3. The "Smart Noise Injection" (PowerDP)
This is the paper's secret sauce. Usually, people add the privacy fog after they have already shrunk the recipe back down to the small notebooks. This ruins the recipe.

FedPower's Trick: The teacher uses a special technique called PowerDP. Imagine trying to find the most important ingredients in a messy pile of recipes. Instead of just adding fog to the final list, the teacher adds the fog while they are sorting and organizing the ingredients.
The Magic: By adding the noise during the sorting process (specifically during a mathematical step called "power iteration"), the noise gets absorbed into the structure of the recipe. It protects the privacy without destroying the important flavors (the signals) of the learning.

The Results

The paper tested this on language tasks (like understanding sentences and sentiment) with 6 different "clients" (simulating different organizations).

Smarter Robot: FedPower produced a much smarter robot than the old broken methods, even when strict privacy rules were applied. It learned faster and reached higher accuracy.
Better Privacy: They tested if hackers could figure out if a specific person's data was in the training set (a "Membership Inference Attack").
- Without privacy, the hackers were right about 60% of the time (better than random guessing).
- With FedPower, the hackers were right only 50% of the time—exactly the same as just guessing "Heads or Tails." The privacy shield worked perfectly.
Efficiency: The teacher (server) didn't have to work much harder. The extra math required was negligible (less than 10% extra time), and the students (clients) didn't have to do anything extra.

Summary

FedPower is like a new way for a teacher to combine student homework. Instead of mixing up the separate parts of the homework and adding static that ruins the work, the teacher first puts the whole homework back together, adds the static in a smart way that doesn't ruin the meaning, and then shrinks it back down for the next round. The result is a robot that is both very smart and very private.

Technical Summary: Improving Parameter-Efficient Federated Learning with Differentially Private Refactorization

1. Problem Statement

The paper addresses a critical vulnerability in Federated Learning (FL) when applied to Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) under Differential Privacy (DP) constraints. While LoRA enables efficient fine-tuning by updating only low-rank matrices ( $A$ and $B$ ) rather than full model parameters, combining it with Global DP (where a trusted server adds noise to protect client data) introduces severe utility degradation.

The core issue stems from a mathematical mismatch in aggregation:

Ideal Aggregation: The true federated average of weight updates should be the average of the products: $\text{Average}(B_i \cdot A_i)$ .
Standard FedLoRA Aggregation: Existing methods independently average the low-rank modules: $\text{Average}(B_i) \cdot \text{Average}(A_i)$ .
The Conflict: Because the average of products does not equal the product of averages, standard FedLoRA introduces "cross-term" mathematical errors. When DP noise is injected into these mismatched modules, the noise interacts multiplicatively with these errors, amplifying the negative impact and leading to suboptimal model accuracy. Existing attempts to mitigate this (e.g., freezing one LoRA module) restrict model expressiveness and convergence.

2. Methodology: FedPower and PowerDP

The authors propose FedPower, a framework that reshapes the server-side aggregation pipeline to eliminate structural errors and a novel mechanism called PowerDP for private low-rank refactorization.

FedPower Workflow:

Client-Side: Clients perform standard local training on their LoRA modules ( $A_i, B_i$ ) and send these lightweight updates to the server.
Server-Side Reconstruction: Instead of averaging $A$ and $B$ separately, the server explicitly reconstructs the full-rank equivalent updates ( $\Delta W_i = B_i A_i$ ) for each client.
Clipping and Aggregation: The server clips these full-rank matrices to bound sensitivity and computes the exact average ( $\Delta W_{global} = \text{Average}(\Delta W_i)$ ). This step eliminates the cross-term mathematical errors inherent in standard FedLoRA.
Private Refactorization: The aggregated full-rank update is projected back into a low-rank space using PowerDP.

PowerDP Mechanism:
PowerDP is a differentially private low-rank factorization algorithm designed to avoid the pitfalls of naive output perturbation (adding noise to final matrices), which often overwhelms signals in low-rank subspaces.

In-Processing Noise Injection: Instead of post-hoc perturbation, PowerDP integrates noise injection directly into a simultaneous subspace iteration (power iteration) sequence.
Process:
1. Initialize a random projection matrix $Q$ .
2. Iteratively project the full-rank matrix $W$ onto row and column spaces, orthonormalizing at each step to ensure convergence to dominant singular vectors.
3. Crucial Step: Calibrated Gaussian noise is injected into the un-normalized projection matrices ( $\tilde{B}$ and $\tilde{A}$ ) before the final orthonormalization.
4. The final orthonormalization (Gram-Schmidt) is applied after noise injection.
Benefit: By injecting noise before the final structural normalization, the mechanism preserves the orthogonality of the resulting matrices and absorbs the noise into the subspace orientation rather than destroying the magnitude of singular values.

3. Key Contributions

FedPower Framework: A novel FL framework that resolves the structural aggregation mismatch in standard FedLoRA by shifting aggregation to the full-rank space before private refactorization, thereby eliminating cross-term errors and preventing the compounding of DP noise.
PowerDP Algorithm: A differentially private low-rank refactorization mechanism that embeds calibrated Gaussian noise into the power iteration sequence. This preserves principal singular vectors and model signals while satisfying rigorous DP guarantees.
Theoretical Guarantees: The paper provides rigorous proofs establishing tight $\ell_2$ -sensitivity bounds for subspace projections. It demonstrates that FedPower satisfies both sample-level and client-level $(\epsilon, \delta)$ -Differential Privacy.
Empirical Validation: Extensive experiments on four GLUE benchmark tasks (MNLI, SST-2, QQP, QNLI) using RoBERTa-large show that FedPower outperforms existing DP-FedLoRA baselines (including FFA-LoRA) in accuracy across various privacy budgets ( $\epsilon \in \{3, 6, 9\}$ ) with negligible computational overhead.
Privacy Robustness: Evaluation against three membership inference attacks (Shadow Model, Loss Based, and Calibration) confirms that FedPower reduces attack success rates to the level of random guessing, effectively protecting client data.

4. Experimental Results

Accuracy: In non-private settings, FedPower outperforms standard FedLoRA by ~1.47 points and FFA-LoRA by ~0.71 points on average. In private settings (e.g., $\epsilon=3$ ), FedPower maintains an average accuracy of 85.18%, significantly outperforming FedLoRA (83.53%) and FFA-LoRA (84.98%).
Convergence: FedPower converges faster than baselines, reaching target accuracy in fewer communication rounds and with fewer transferred bits.
Noise Injection Comparison: The study compares PowerDP against Input Perturbation (noise before factorization) and Output Perturbation (noise after factorization). Output perturbation resulted in catastrophic accuracy failure (e.g., 60.59% at $\epsilon=3$ ), while PowerDP maintained high accuracy by preserving signal integrity during factorization.
Overhead: The computational overhead of the server-side refactorization is minimal (less than 10% increase in runtime), and clients incur no additional cost.

5. Significance and Claims

The paper claims that FedPower successfully bridges the gap between efficiency (via LoRA), privacy (via Global DP), and accuracy. By decoupling the optimization space (low-rank on clients) from the privacy-bounding space (full-rank on the server), the framework avoids the mathematical errors that typically plague DP-FL with LoRA.

The authors assert that their approach provides a provably secure defense against downstream membership inference attacks while maintaining the high utility required for practical LLM fine-tuning. The work is positioned as a solution for cross-silo FL (e.g., healthcare, finance) where data sensitivity is high, and it suggests future applicability to cross-device FL and heterogeneous (non-IID) data distributions, though these are noted as directions for future work rather than current achievements.