🤖 machine learning

Layer-wise LoRA fine-tuning: a similarity metric approach

This paper proposes a layer-wise Low-Rank Adaptation (LoRA) fine-tuning approach that systematically selects the most relevant layers based on internal representation similarity, thereby reducing trainable parameters by up to 50% while maintaining or even improving predictive performance across various model architectures and tasks.

Original authors: Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to do a specific job, like solving math problems or writing code. This robot, called a Large Language Model (LLM), has already learned a massive amount of general knowledge from reading almost everything on the internet. It's like a genius who knows a little bit about everything but needs to specialize. To teach it a new trick, you usually have to "fine-tune" it, which means adjusting its internal settings. But here's the catch: these robots are so huge that adjusting all their settings requires a computer so powerful and expensive that only giant tech companies can afford it. It's like trying to fix a single loose screw on a skyscraper by rebuilding the entire building.

To solve this, scientists invented a clever shortcut called LoRA (Low-Rank Adaptation). Instead of touching every single screw in the skyscraper, LoRA suggests you only add a tiny, lightweight patch of new screws to help the robot learn the new task. This saves a ton of money and energy. However, even with this shortcut, the robot is still so big that the "patch" can sometimes be too heavy for smaller computers. The big question is: Do we really need to patch every part of the robot to make it smart at the new job? Or are there just a few specific spots where the magic happens? This paper dives into that question, asking if we can find the "golden spots" to fix and leave the rest of the giant machine alone.

The "Layer-by-Layer" Detective Work

The authors of this paper, Keith Ando Ogawa and their team, decided to treat the robot's brain like a multi-story building. Inside a Large Language Model, information flows through a series of "layers," kind of like floors in a skyscraper. As a sentence travels from the bottom floor to the top, it gets processed and transformed. The team suspected that not every floor is equally important for every new task. Some floors might be doing the heavy lifting for math problems, while others might be more useful for writing poetry.

Their idea was simple but powerful: instead of putting a LoRA patch on every single floor, why not figure out which floors actually change the most when the robot learns something new? They came up with a way to measure this "change" using a mathematical tool called a similarity metric. Think of it like a "change detector." They looked at what the robot "thought" before it passed information up to a specific floor and what it "thought" after that floor processed it. If the input and output were almost identical, that floor wasn't doing much. But if the input and output were very different, that floor was doing a lot of work!

They called this method "Layer-wise LoRA fine-tuning." It's like hiring a detective to walk through the skyscraper, checking each floor to see where the most activity is happening. Once they identify the busiest floors, they only put the LoRA patches on those specific ones and leave the quiet, inactive floors frozen in place.

What They Found: Less is More

When the team tested this idea on several different types of robot brains, the results were surprisingly good. They found that by selecting only the most active layers—specifically, about half of the total layers—they could cut the number of trainable parameters (the "screws" they were adjusting) by 50%.

Here is the best part: even though they were adjusting half as many settings, the robots didn't get dumber. In fact, for some models like LLaMA 2-7B and Mistral-7B, the robots actually got better at solving math problems and writing code when they only focused on the most important layers. For other models, like Gemma-7B, the performance dropped just a tiny, almost unnoticeable amount.

The researchers also compared their "change detector" method against other ways people try to pick layers, like just picking the middle floors or the top and bottom floors. Their method consistently beat these simple guesses. For example, on a standard test called GLUE, their method kept the robot's performance high while cutting the work in half, whereas a method that just picked the middle layers sometimes caused the robot to stumble.

They also checked how this affected the computer's memory and speed. Because they were adjusting fewer layers, the training process became faster. On some models, they saw a speedup of about 1.25 times, and the computer needed less memory to do the job. This is a huge deal for anyone trying to run these models on smaller, cheaper computers.

Why This Matters

The paper suggests that we don't need to treat every part of a giant AI model as equally important. By using a simple measurement of how much a layer changes the information passing through it, we can find the "sweet spots" for learning. This means we can make these powerful AI tools much more efficient without losing their smarts.

The authors tested this on different kinds of robots, including those that only read text, those that only write text, and even a multimodal robot (LLaVA-1.5-7B) that can see pictures and read text. In all cases, picking just the right layers worked well. They even tried doubling the "rank" (making the patches more complex) on the selected layers to see if that would help, but found that it didn't really make a big difference. It seems that just picking the right layers is the key, not making the patches themselves more complicated.

While the paper doesn't claim to have solved every problem in AI, it offers a very practical and effective way to make fine-tuning cheaper and faster. It suggests that for many tasks, a small, targeted effort is just as good as a massive, all-out overhaul. As the authors note, this approach works well with existing tools and could help more researchers and smaller companies get in on the action of building smarter AI, without needing a supercomputer the size of a house.

Technical Summary: Layer-wise LoRA Fine-tuning via Similarity Metrics

Problem Statement
While Pre-trained Large Language Models (LLMs) have advanced general-purpose AI, their adaptation to downstream tasks via fine-tuning incurs substantial computational costs. Parameter-Efficient Fine-Tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA), mitigate this by freezing pre-trained weights and updating low-rank decomposition matrices. However, even with LoRA, the number of trainable parameters can remain prohibitive as model scales increase. Existing LoRA variants primarily focus on matrix-level optimizations (e.g., initialization strategies, decomposition formats, or rank adjustments) but often overlook the architectural level. Specifically, it remains an open question whether all transformer layers contribute equally to task adaptation or if a systematic selection of a subset of layers could further reduce computational costs without sacrificing predictive performance.

Methodology
The authors propose a layer-selection approach that is orthogonal to existing LoRA techniques. The core hypothesis is that not all layers contribute equally to changes in internal representations during task adaptation. The method operates as follows:

Representation Extraction: For a pre-trained model $F$ with $M$ transformer layers ( $L_1, \dots, L_M$ ) and a downstream dataset $D$ , the method extracts internal representations. For encoder-only architectures, the <CLS> token representation is used; for decoder-only (autoregressive) architectures, the last token representation is utilized, as these tokens are argued to contain the relevant information for prediction.
Similarity Measurement: The method employs the Centered Kernel Alignment (CKA) metric to measure the similarity between the input representation ( $R_{i-1}$ ) and the output representation ( $R_i$ ) of each layer $L_i$ .
Importance Scoring: A layer importance metric $I_{L_i}$ is defined as the dissimilarity between input and output:
$I_{L_i} = 1 - \text{CKA}(R_{i-1}, R_i)$
The authors hypothesize that layers exhibiting high dissimilarity (low CKA) between input and output have the most significant impact on internal representation changes and are therefore the most critical candidates for fine-tuning.
Selection and Fine-tuning: The top $N$ layers with the highest dissimilarity scores are selected for LoRA fine-tuning, while the remaining layers are frozen. This process occurs prior to the fine-tuning phase, relying solely on inference, thus avoiding expensive additional training steps or gradient calculations during the selection phase.

Key Contributions

Novel Selection Mechanism: Introduction of a systematic method to select transformer layers for fine-tuning based on their contribution to internal representation changes, quantified via CKA dissimilarity.
Orthogonality: The technique is compatible with existing PEFT methods (LoRA, PiSSA, etc.) and operates at the layer level rather than the matrix level, offering an additional dimension for computational efficiency.
Parameter Reduction: The method reduces the number of trainable parameters in LoRA-based techniques by up to 50% while maintaining or improving predictive performance.
Empirical Validation: Extensive experiments across encoder-only, decoder-only, and multimodal architectures demonstrate the method's robustness.

Experimental Results
The authors evaluated their approach on various models and benchmarks:

Encoder-Only Models (RoBERTa-base, DeBERTa-v3-base): On the GLUE benchmark, fine-tuning only 50% of the layers (selected via the proposed metric) resulted in a negligible drop in predictive performance (average drop of 0.27 percentage points for DeBERTa-v3-base) compared to full-layer LoRA. This significantly outperformed a competing layer-selection method based on Fisher Information Matrix (FIM) scores, which suffered a much larger performance drop (7.68 points).
Decoder-Only Models (LLaMA 2-7B, Mistral-7B-v0.1, Gemma-7B): Evaluated on mathematical problem-solving (GSM8K, MATH) and coding tasks (HumanEval, MBPP).
- For LLaMA 2-7B and Mistral-7B, the method achieved improvements in predictive performance on most tasks despite a 50% reduction in trainable parameters.
- For Gemma-7B, a small performance drop was observed, but results remained competitive.
- The method consistently outperformed naive heuristics (e.g., selecting only middle, first, or last layers).
Multimodal Models (LLaVA-1.5-7B): On the ScienceQA benchmark, the method achieved an average accuracy of 79.26% compared to 79.49% for full-layer LoRA, with a 50% reduction in trainable parameters (from 160M to 80M).
Efficiency: The selection of layers led to significant speedups in training time (up to ~1.4x for certain configurations) and reductions in GPU memory usage (approximately 15-20%).

Significance and Claims
The paper claims to be the first work to systematically explore effective subsets of layers for LoRA fine-tuning across a wide set of models and tasks, including multimodal architectures. The authors argue that their approach provides a practical path to further computational efficiency in fine-tuning large models without requiring architectural changes or complex training procedures.

The results suggest that:

Specific layers are more influential for specific tasks, and these can be identified via representation similarity metrics.
CKA dissimilarity is an effective tool for layer selection, outperforming naive strategies and existing FIM-based approaches.
The method is agnostic to the specific PEFT technique used (working with both LoRA and PiSSA) and the model architecture (encoder, decoder, or multimodal).

The authors remain modest regarding future directions, noting open questions such as whether importance metrics could capture layer interactions rather than individual contributions, whether ideal LoRA ranks could be defined per layer, and whether other approaches beyond similarity metrics could be effective. They do not claim universal optimality for all models but highlight the method's effectiveness in reducing computational burdens while preserving performance.

The "Layer-by-Layer" Detective Work

What They Found: Less is More

Why This Matters

Technical Summary: Layer-wise LoRA Fine-tuning via Similarity Metrics

More like this