Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?

Technical Summary: Can LoRA Fusion Support Cross-Domain Tasks in Cloud–Edge Collaboration?

Problem Formulation
Cloud-hosted Large Language Models (LLMs) increasingly rely on Low-Rank Adaptation (LoRA) for domain-specific fine-tuning. However, in realistic cloud–edge collaboration scenarios, domain data is distributed across multiple edge devices and cannot be uploaded to the cloud due to privacy constraints (e.g., GDPR). This necessitates a paradigm where edge devices train local LoRA adapters, which are then uploaded and fused in the cloud to enable cross-domain problem solving.

The paper identifies two fundamental limitations in existing approaches to this problem:

1. Resource Mismatch: Current methods implicitly assume edge devices can instantiate and fine-tune cloud-scale LLMs. In practice, edge devices (e.g., mobile phones with ~6GB RAM) lack the memory and compute to host large models (e.g., 7B+ parameters), making direct local adaptation infeasible.
2. Inadequate Evaluation: Existing LoRA fusion methods (e.g., FedAvg, FFA-LoRA) are primarily evaluated on single-domain tasks or multi-task settings where test instances belong to a single domain. They fail to address "true" cross-domain tasks where a single query requires the joint integration of knowledge from multiple distinct domains (e.g., combining philosophy and biology).

Methodology
To address these limitations, the authors propose a comprehensive framework and a new benchmark:

1. Prune-Train-Recover Framework: To resolve the resource mismatch, the authors propose a three-step pipeline:

   • Prune: The cloud-hosted LLM is structurally pruned into a compact small language model (SLM) suitable for edge deployment.
   • Train: Edge devices perform parameter-efficient LoRA fine-tuning on the pruned model using local private data.
   • Recover: The resulting LoRA adapters are uploaded to the cloud, where they are mapped back to the original parameter space via zero-padding (recovering the full dimensionality) before fusion.

2. MMLU-CD Benchmark: The authors introduce MMLU-CD, a cross-domain benchmark constructed by pairing samples from different MMLU subject domains (Humanities, Social Sciences, STEM) and using DeepSeek-R1 to synthesize coherent multiple-choice questions that require joint reasoning across domains. This allows for explicit evaluation of cross-domain problem-solving capabilities.

3. LoRA-CR (Conflict Resolution): Empirical evaluation reveals that existing fusion methods fail on cross-domain tasks, often underperforming the base model. The authors attribute this to representation conflicts among LoRA adapters trained on different domains, where updates along shared latent directions cancel each other out. To mitigate this, they propose LoRA-CR, a plug-and-play preprocessing module that:

   • Projects recovered LoRA updates into a shared latent subspace via Singular Value Decomposition (SVD).
   • Quantifies direction-wise conflicts using a cosine similarity-based metric.
   • Selectively suppresses conflicting or misaligned updates while preserving consistent domain-specific signals before the final fusion step.

Key Results

• Failure of Existing Methods: Under the proposed prune-train-recover framework, three representative LoRA fusion methods (FedAvg, FFA-LoRA, FedSA-LoRA) were evaluated on MMLU-CD. The results were negative: in 7 out of 9 cross-domain evaluation cases, the fused models underperformed the base LLM without any fusion.
• Effectiveness of LoRA-CR: Integrating the proposed LoRA-CR module significantly improved performance. Across three cross-domain settings and three fusion methods, accuracy improved in 8 out of 9 cases, with average gains ranging from 1.12% to 3.34%.
• Conflict Reduction: The introduction of LoRA-CR reduced the average direction-wise conflict score by approximately 0.2%, quantitatively validating the mitigation of representation conflicts.
• Robustness: The performance gains of LoRA-CR were observed to increase monotonically with higher pruning ratios, suggesting that conflict resolution becomes even more critical as the representational capacity of the adapters is constrained.

Significance and Claims
The paper claims that while LoRA fusion is a promising paradigm for privacy-preserving cloud–edge collaboration, existing methods are insufficient for cross-domain tasks due to unresolved parameter conflicts. The authors assert that:

• Conflict Mitigation is Critical: The inability of current methods to handle representation conflicts is a largely overlooked factor that prevents effective cross-domain knowledge integration.
• Feasibility via Pruning: The prune-train-recover framework successfully bridges the gap between cloud-scale models and resource-constrained edge devices, enabling a realistic evaluation of the fusion problem.
• Necessity of New Benchmarks: Standard single-domain evaluations are inadequate; rigorous cross-domain benchmarks like MMLU-CD are required to expose the limitations of current fusion strategies.

The work concludes that conflict-aware fusion is a fundamental challenge in cloud–edge LLM collaboration and that simple, conflict-resolution modules like LoRA-CR are necessary to unlock the potential of distributed domain knowledge for complex, cross-domain reasoning.