🤖 machine learning

The Appeal and Reality of Recycling LoRAs with Adaptive Merging

This paper investigates the feasibility of adaptively merging user-contributed LoRAs from the Hugging Face Hub, revealing that while such methods outperform the base model, their benefits are largely attributable to a regularization effect rather than cross-task transfer, as they offer limited advantage over training a new LoRA and perform similarly even with randomly initialized parameters.

Original authors: Haokun Liu, Gyung Hyun Je, Marco Ciccone, Zhenlin Xu, Prasanth YSS, Colin Raffel

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haokun Liu, Gyung Hyun Je, Marco Ciccone, Zhenlin Xu, Prasanth YSS, Colin Raffel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, super-smart robot brain (let's call it the "Base Model") that knows a little bit about everything but isn't a master of anything specific. Now, imagine thousands of people have built tiny, specialized "skill chips" called LoRAs for this robot. One chip makes it great at math, another at writing poetry, another at understanding medical jargon. These chips are floating around on the internet, free for anyone to grab.

The big question researchers asked was: If we take a random handful of these chips from the internet and glue them together, will we get a super-robot that's better than just teaching the base robot a new skill from scratch?

This is the story of "recycling" LoRAs. The researchers gathered nearly 1,000 of these user-made chips from the Hugging Face Hub (a giant library of AI models) and tried to merge them using fancy "adaptive merging" techniques. They wanted to see if they could mix and match these pre-made skills to solve new problems without needing to train a new chip from scratch.

The Big Surprise: The "Magic Glue" Might Be a Trick

Here is the twist: The fancy merging didn't actually work as well as everyone hoped.

When the researchers tried to merge these recycled chips to solve a new task, they found that the "magic" of mixing them was mostly an illusion.

The Reality Check: If you have a small dataset (about 100 examples) to teach the robot a new task, it is almost always better to just train a brand new LoRA on that specific data.
The "Random" Shock: Even more strangely, when they included the new, freshly trained LoRA in the mix, it didn't matter which other chips they added! They could grab 30 chips from the internet, or they could grab 30 chips that were just filled with random numbers (like static on a TV), and the result was almost the same.

The paper suggests that the improvement didn't come from the "knowledge" inside the recycled chips. Instead, it seems the act of merging them acts like a safety net or a regularizer. It's like trying to balance a stack of plates; adding more random plates might actually help stabilize the whole stack, not because the plates are useful, but because the process of balancing them forces the system to be more careful.

Why Did Past Studies Say It Worked?

You might wonder, "But didn't other papers say merging LoRAs is amazing?"

The authors suspect those past studies were a bit too perfect. They often used "in-house" chips—chips they made themselves in a controlled lab setting where they knew exactly which chips were relevant. It's like testing a recipe by only using ingredients you bought at the same store, knowing exactly which ones go together.

In the real world ("in the wild"), the chips are messy. They were made by different people, with different data, and for different goals. The researchers found that positive transfer (where one skill helps another) only happens if the chips are highly relevant to the task. If you mix a "math chip" with a "poetry chip" to solve a "history quiz," it doesn't help. In fact, the paper shows that if you remove the most relevant chips from a pool, the performance drops like a stone, eventually becoming no better than using random noise.

The Verdict

So, what's the takeaway for a curious teenager?

Don't expect a free lunch: You can't just grab a bunch of random AI skills from the internet, mix them, and expect a miracle.
Training is still king: If you have a specific job to do, training a fresh, small adapter (LoRA) on your own data is usually the best and most reliable way to go.
The "Recycling" is tricky: While merging can sometimes beat the base model, it rarely beats a simple, fresh training session. And if you do decide to merge, the specific choice of which chips to use matters less than you think—especially if you already have the right chip for the job in the mix.

The authors conclude that while the idea of recycling AI models is exciting, the reality is a bit more complicated. The "positive transfer" we hope for is real, but it's rare and fragile in the messy, uncurated world of public AI models. It suggests we need better ways to find the right "gems" in the pile of digital junk, or perhaps we need to rethink how we build these models in the first place.

Technical Summary: The Appeal and Reality of Recycling LoRAs with Adaptive Merging

Problem Statement

The widespread availability of fine-tuned Low-Rank Adaptation (LoRA) modules for open pre-trained models has spurred interest in "recycling" these modules to improve performance on new, unseen tasks. Existing research on model merging and adaptive merging typically evaluates methods on curated, bespoke sets of LoRAs trained under controlled conditions on specific target tasks. However, no prior work has systematically evaluated the feasibility of recycling LoRAs found "in the wild" on public repositories like the Hugging Face Hub. These real-world pools are characterized by extreme heterogeneity in training data, hyperparameters, and architectural choices, and often lack the metadata necessary for informed selection. This paper addresses the gap between the theoretical promise of adaptive merging and its practical utility in realistic, data-constrained settings using a large-scale pool of user-contributed LoRAs.

Methodology

The authors conducted a large-scale empirical study using a pool of 958 user-contributed LoRAs trained on the Llama 3.1 8B-Instruct model. The evaluation framework involved 62 diverse downstream tasks, with a strict data constraint of only 100 samples per task (80 for training/validation, 20 for validation) to tune merging coefficients.

Framework and Design Space

The study proposes a unified framework to systematically explore adaptive merging methods, varying four key design dimensions:

Selection: Strategies to choose a subset of $k$ LoRAs from the pool (e.g., Random, Evaluation-based, Cosine similarity, Quasi-FIM).
Granularity: The level at which merging coefficients are learned (Model, Layer, Sublayer, or Module).
Coefficient Activation: Transformations applied to raw coefficients (Linear, Softmax, Leaky ReLU).
Tuning: Optimization methods (Gradient-based, Gradient-free, or Joint optimization of coefficients and LoRA parameters).

The authors identified a specific configuration ("Ours") that performed best in preliminary searches: Evaluation-based selection (top- $k$ based on target-task validation accuracy), Module-level granularity, Leaky ReLU activation, and Gradient-based tuning.

Baselines and Comparisons

The study compared:

Non-adaptive methods: Simple Averaging, TIES Merging, and Task Singular Vectors (TSV) Merging.
Adaptive methods: LoraHub, $\pi$ -Tuning, AdaMerging, and the authors' proposed "Ours" method.
Critical Baselines:
- Zero-shot Prompting: The base model without any adaptation.
- Target-Task LoRA: A LoRA trained directly on the 100 available target-task samples. This is a crucial baseline often overlooked in prior literature, as adaptive merging methods also utilize this small dataset to tune coefficients.
- Randomly Initialized LoRAs: A pool of LoRAs with parameters re-initialized from a normal distribution to test if the specific knowledge in recycled LoRAs is necessary.

Key Results

1. Adaptive Merging vs. Base Model

Adaptive merging methods using recycled LoRAs consistently outperform zero-shot prompting of the base model across the 62 downstream tasks. Non-adaptive methods generally underperform adaptive methods, and selecting a subset of LoRAs (e.g., 30) performs significantly better than averaging all available LoRAs.

2. Adaptive Merging vs. Target-Task LoRA

When compared to a LoRA trained directly on the target-task data, adaptive merging methods provide limited and inconsistent benefits.

Most non-adaptive methods fail to outperform the simple target-task LoRA.
Adaptive methods (like AdaMerging and "Ours") achieve performance comparable to the target-task LoRA but rarely surpass it significantly.
The inclusion of the target-task LoRA in the merging pool is almost always necessary for adaptive methods to achieve their best performance.

3. The Role of Selection and "Positive Transfer"

A striking finding is that once the target-task LoRA is included in the merging pool, the choice of other recycled LoRAs becomes largely irrelevant.

Random vs. Evaluated Selection: The performance gap between carefully selecting LoRAs based on evaluation scores and selecting them randomly vanishes when the target-task LoRA is present.
Random Initialization: Replacing the recycled LoRAs with randomly initialized parameters yields performance similar to using actual recycled LoRAs, provided the target-task LoRA is in the pool.
Coefficient Distribution: Analysis of learned merging coefficients reveals that when the target-task LoRA is included, the optimization process assigns it a significantly higher weight than any recycled LoRA, effectively overshadowing the others.

4. Controlled vs. Realistic Settings

The authors replicated experiments using "in-house" LoRAs (trained on tasks highly relevant to the target) to simulate the controlled settings of prior work. In this scenario, positive transfer was observed, and merging outperformed the target-task LoRA baseline. However, this benefit decayed rapidly as the relevance of the source LoRAs decreased, suggesting that the "positive transfer" observed in prior literature relies on a narrow region of task similarity that is rare in "in-the-wild" pools.

Significance and Claims

The paper concludes that while adaptive merging is a valid technique for improving upon zero-shot baselines, its practical utility for recycling "in-the-wild" LoRAs is limited.

Regularization vs. Transfer: The authors suggest that the marginal gains of adaptive merging over a simple target-task LoRA likely stem from a regularization effect (via the over-parameterized coefficient space or implicit ensembling) rather than genuine "positive transfer" of knowledge from the recycled LoRAs.
Methodological Critique: The study highlights a discrepancy between controlled benchmarks (where source tasks are curated to be relevant) and realistic settings (where source tasks are heterogeneous and often irrelevant). It argues that previous claims of success may be overstated when applied to real-world model repositories.
Practical Implications: Researchers considering adaptive merging should first verify that it outperforms a simple target-task LoRA baseline. The paper advocates for "in-the-wild" evaluation settings and calls for better infrastructure (standardized metadata, licensing, and provenance) around public LoRA repositories to make recycling more viable.

The authors release their model checkpoints and code to facilitate rigorous, realistic evaluation of future recycling methods.