🤖 machine learning

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

This paper proposes FedAS-LoRA, a federated learning framework that adaptively selects whether to share the input-side or output-side LoRA factor across clients based on a novel Rank-Aware Shared-Subspace Sufficiency (RSS) metric, thereby minimizing projection residuals and enhancing fine-tuning performance.

Original authors: Xinyi Xu, Bingnan Xiao, Shuang Qin, Gang Feng, Tony Q. S. Quek

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyi Xu, Bingnan Xiao, Shuang Qin, Gang Feng, Tony Q. S. Quek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where giant, super-smart computers (called Large Language Models) are the new geniuses of our time, capable of writing stories, solving math problems, and chatting like humans. But here's the catch: these geniuses are so huge that they can't fit on a single phone or laptop, and we can't just copy-paste everyone's private diary data into one central brain to teach them new tricks. That's where "Federated Learning" comes in. It's like a group of friends trying to learn a new dance together without ever leaving their own living rooms. They each practice on their own data, send only their dance moves (not the music or the room) to a central coach, who mixes them all up to create a better group routine.

To make this dance lesson efficient, scientists use a clever trick called "LoRA" (Low-Rank Adaptation). Instead of retraining the whole giant brain, they attach two small, flexible "training wheels" (let's call them Factor A and Factor B) to the model. Factor A is like a translator that turns the input (the dance music) into a secret code, and Factor B is the dancer that turns that code into the actual moves. The big question the researchers asked was: In a group dance, should everyone share the same translator (Factor A) and keep their own dancers (Factor B), or should they share the same dancers and keep their own translators? For a long time, people just picked one way and stuck with it, assuming it was always the best. But this paper suggests that's a bit like assuming everyone should wear the same size shoes just because they're all humans.

The authors of this paper, Xinyi Xu and their team, discovered that there is no single "best" way to share these training wheels. It turns out that whether you should share the translator or the dancer depends entirely on the specific data the group is working with and how complex the dance steps are (the "rank" of the adaptation). They found that if you force everyone to share the translator when they really need to share the dancer (or vice versa), the group ends up with a clumsy, mismatched routine. To fix this, they invented a new method called FedAS-LoRA. Before the training even starts, this method uses a special "scout" (a metric they call RSS) to peek at the data and decide: "Okay, for this specific group and this specific task, we should share Factor A," or "Nope, for this one, we should share Factor B."

Think of it like a smart dance coach who, before the music starts, looks at the dancers and the song to decide who should share their shoes and who should share their choreography notes. If the song is very specific to each dancer's style, the coach might say, "Keep your own notes, but let's all use the same rhythm guide." If the song is generic but the dancers have very different moves, the coach might say, "Let's all use the same choreography, but keep your own rhythm guides." By making this choice dynamically, their method consistently outperformed the old, rigid methods. In tests on various language tasks, their adaptive approach achieved an average accuracy of 90.77%, beating the previous best method by nearly 1 percentage point. They also proved mathematically that this flexible approach is stable and won't cause the group to fall apart, even if some dancers skip practice or join late. The paper shows that the "one-size-fits-all" rule for sharing these model parts is broken, and the future of efficient AI learning lies in being flexible enough to choose the right strategy for the job at hand.

Technical Summary: Rethinking Factor Sharing in Federated LoRA

Problem Statement
In Federated Learning (FL), fine-tuning Large Language Models (LLMs) via Low-Rank Adaptation (LoRA) faces a critical challenge known as "aggregation mismatch." Standard LoRA represents model updates as the product of two low-rank factors, $\Delta W_i = B_i A_i$ . In a federated setting with $N$ clients, simply averaging the factors independently ( $\frac{1}{N}\sum B_i \cdot \frac{1}{N}\sum A_i$ ) does not equal the average of the updates ( $\frac{1}{N}\sum B_i A_i$ ).

Existing approaches to mitigate this often adopt "hard-coded" factor-sharing strategies, such as fixing a common factor $A$ across all clients while keeping $B$ local (Share-A/Local-B), or vice versa (Share-B/Local-A). However, the paper demonstrates that neither strategy is universally optimal. The choice of which factor to share depends on the specific data distributions, LoRA rank, and system settings. A fixed assignment can lead to suboptimal performance because it fails to account for the structural asymmetry between the input-side and output-side representation spaces required by different clients.

Methodology
The authors propose FedAS-LoRA (Federated Adaptive Factor Sharing Low-Rank Adaptation), a framework that adaptively selects the sharing strategy before training begins.

Theoretical Insight (Least-Squares Surrogate):
The paper analyzes the structural asymmetry of LoRA factors using a least-squares surrogate. It reveals that:
- Share-A/Local-B imposes a common rank- $r$ input-side space across clients. The performance is limited by the projection residual of the desired updates onto this shared input space.
- Share-B/Local-A imposes a common rank- $r$ output-side space. The performance is limited by the projection residual onto this shared output space.
  The preferred strategy is the one that minimizes the aggregate projection residual across clients.
Rank-Aware Shared-Subspace Sufficiency (RSS) Metric:
To determine the optimal sharing side without training, the authors design the RSS metric. This metric uses representations extracted from a frozen LLM backbone to assess whether a global rank- $r$ input subspace is sufficient to capture the local data distributions.
- It compares the "representation energy" (captured by the second-order variation) of a global rank- $r$ subspace against the energy retained by client-specific rank- $r$ subspaces.
- A threshold $\tau(r)$ is calibrated using random sample reassignment and bootstrap resampling to account for estimation errors.
- Decision Rule: If the energy loss (RSS deficit) of using a global input subspace exceeds the threshold, the system selects Share-B/Local-A. Otherwise, it selects Share-A/Local-B.
Convergence Analysis:
The paper provides a theoretical proof that FedAS-LoRA converges to a stationary neighborhood under arbitrary client participation (partial participation). The analysis holds for both sharing strategies, showing that the iterates satisfy a convergence bound dependent on the number of rounds, local steps, and client sampling drift.

Key Contributions

Projection-Based Comparison: The paper provides the first formal comparison between Share-A/Local-B and Share-B/Local-A, revealing that they correspond to input-side and output-side projection residuals, respectively. This explains why fixed strategies fail in diverse federated settings.
Adaptive Framework (FedAS-LoRA): A novel framework that selects the shared factor dynamically based on data characteristics rather than hard-coding the policy.
Training-Free Metric (RSS): The design of the RSS metric, which efficiently determines the optimal sharing side using only frozen backbone representations, avoiding the need for preliminary training.
Convergence Guarantees: Theoretical proof of convergence for the proposed method under arbitrary client participation, extending beyond the full-participation assumption common in prior works.

Experimental Results
Experiments were conducted on natural language understanding (GLUE benchmark: MNLI, SST-2, QNLI, QQP, RTE) and natural language generation (GSM8K with LLaMA3-8B) tasks.

Performance: FedAS-LoRA consistently outperformed baselines (including standard LoRA, FFA-LoRA, FedDPA-LoRA, and FedSA-LoRA) across various tasks. On the GLUE benchmark, FedAS-LoRA achieved an average accuracy of 90.77%, surpassing FedSA-LoRA by 0.93 percentage points.
Robustness: The method demonstrated superior performance under varying conditions, including:
- Client Sampling: Both uniform and non-uniform sampling settings.
- Data Heterogeneity: IID and non-IID (Dirichlet) distributions.
- LoRA Ranks: Performance remained competitive across ranks $r \in \{2, 4, 8, 16\}$ , validating that no single sharing policy is optimal for all ranks.
Generality: The approach was effective across different LoRA variants (rsLoRA, VeRA) and model sizes (RoBERTa-large, LLaMA3-8B).

Significance
The paper claims that the significance of this work lies in challenging the assumption that a single factor-sharing strategy is sufficient for all federated LoRA scenarios. By demonstrating that the optimal strategy depends on the interplay between data distribution and LoRA rank, the authors provide a principled, adaptive solution. The RSS metric offers a practical, training-free mechanism to enhance fine-tuning performance in resource-constrained federated environments, ensuring that the shared factor aligns with the underlying structural requirements of the client data.

Technical Summary: Rethinking Factor Sharing in Federated LoRA

More like this