💬 NLP

Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning

This empirical study demonstrates that basic LoRA performs comparably to more complex variants in multilingual instruction tuning, as neither offers a significant advantage in balancing cross-lingual transfer and knowledge retention, nor do they substantially alter layer-wise language representations.

Original authors: Thamali Wijewardhana, Napoleon H. Reyes, Surangika Ranathunga

Published 2026-06-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Thamali Wijewardhana, Napoleon H. Reyes, Surangika Ranathunga

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart encyclopedia (a Large Language Model) that was originally written entirely in English. Now, you want to teach it to understand and answer questions in other languages like Urdu, Swahili, or Hindi, but you don't have the money or time to rewrite the whole book from scratch.

Enter LoRA (Low-Rank Adaptation). Think of LoRA as a set of sticky notes you can stick onto the pages of the encyclopedia. Instead of rewriting the whole book, you just write new instructions on these sticky notes and stick them on specific pages. This is a cheap and fast way to teach the book new languages.

Recently, scientists invented "fancy" versions of these sticky notes (called LoRA variants like DoRA, VeRA, AdaLoRA, and PiSSA). These new versions claim to be smarter, more efficient, or better at organizing the information than the original, basic sticky notes.

The Big Question:
Do these fancy, complex sticky notes actually work better than the simple, basic ones when teaching the encyclopedia multiple new languages at once?

The Experiment:
The researchers from Massey University set up a test. They took the English encyclopedia and tried to teach it five different languages (Urdu, Swahili, Hindi, Bengali, and Telugu) using:

The original, basic sticky notes (LoRA).
Four different "fancy" versions (DoRA, VeRA, AdaLoRA, PiSSA).

They tested how well the book learned the new languages while making sure it didn't forget its original English skills. They also checked how much computer power (like electricity and memory) each method used.

The Findings (The "Aha!" Moment):
The results were surprisingly simple: The fancy sticky notes didn't really help.

No Magic Trick: The complex, expensive methods did not perform significantly better than the basic LoRA. In fact, for some languages, the basic method was just as good, if not better.
The "Shared" Problem: One of the fancy methods (VeRA) tried to use the same set of instructions for every single page of the book. The researchers found this actually hurt performance because it was too rigid, like trying to use the same recipe for baking a cake and fixing a car.
Cost vs. Benefit: The fancy methods often used more computer power and memory without giving better results. The basic LoRA was the most efficient "bang for your buck."
The "Where" Matters: The study found that to teach a new language effectively, you need to put these sticky notes on every single page of the book, not just the last few pages. If you only stick them on the end, the book doesn't learn as well.

Looking Under the Hood:
The researchers also peeked inside the "brain" of the encyclopedia (the hidden layers) to see how the different methods changed the way the computer "thought" about language.

They found that all the methods looked almost identical inside. Whether they used the basic notes or the fancy ones, the way the computer represented the languages didn't change much. The fancy architectural tricks didn't actually create a new way for the computer to understand the world; they just rearranged the furniture slightly without changing the house.

The Bottom Line:
If you are trying to teach a large language model new languages, you don't need to buy the most expensive, complex tools. The basic, original LoRA method is just as effective as the fancy new variants, uses less computer power, and is easier to manage. Sometimes, the simple solution is the best one.

Technical Summary: Which LoRA? An Empirical Study on the Effectiveness of LoRA Techniques During Multilingual Instruction Tuning

Problem Statement
Low-Rank Adaptation (LoRA) is a leading parameter-efficient fine-tuning (PEFT) technique that freezes original Large Language Model (LLM) parameters while training small low-rank matrices. While numerous novel LoRA variants (e.g., DoRA, VeRA, AdaLoRA, PiSSA) have been introduced to improve performance on English tasks, their efficacy in multilingual instruction tuning remains underexplored. Specifically, it is unclear whether these architectural novelties offer advantages over basic LoRA when balancing cross-lingual transfer (adapting knowledge from English to new languages) and knowledge retention (preserving English capabilities). Furthermore, existing research on LoRA in cross-lingual settings has largely focused on basic LoRA or low-resource language adaptation without comprehensively comparing these variants or analyzing their impact on internal language representations.

Methodology
The authors conducted an empirical evaluation comparing basic LoRA against four advanced variants:

DoRA (Weight-Decomposed Low-Rank Adaptation)
VeRA (Vector-Based Random Matrix Adaptation)
AdaLoRA (Adaptive Low-Rank Adaptation)
PiSSA (Principal Singular Values and Singular Vectors Adaptation)

Datasets & Languages: Experiments utilized the XNLI dataset (Natural Language Inference) and TyDiQA-GoldP (Question Answering). The study focused on five linguistically diverse, low-resource target languages (TLs): Urdu (Ur), Swahili (Sw), Hindi (Hi), Bengali (Bn), and Telugu (Te).
Experimental Design:
- Data Mixture: The LLM was fine-tuned on a mixture of English and TL data. The authors evaluated TL ratios of 0%, 1%, 10%, and 50% to determine optimal data composition.
- Parameter Control: To ensure fair comparison, most configurations were standardized to a rank ( $r$ ) of 8 and a parameter budget of approximately 0.26%. A modified DoRA configuration (DoRA*) was created to match this budget, as standard DoRA typically peaks at $r=16$ (0.36% parameters).
- Models: Primary experiments used Llama-3.1-8B. Generalizability was tested on Llama-3.2-3B and Qwen3-8B.
- Analysis: Beyond performance metrics (Micro F1), the study included a layer-wise hidden embedding analysis using CLD3 to classify language representation within the model's internal layers.

Key Results

Performance Equivalence: Across diverse target languages and model architectures, no significant advantage was found for complex LoRA variants over basic LoRA. An ANOVA test on target language performance yielded no statistically significant difference ( $p = 0.89$ ) among the variants.
English Performance Hierarchy: While target language performance was similar, English performance showed significant differences. Basic LoRA, DoRA, DoRA*, and AdaLoRA performed similarly and significantly better than PiSSA and VeRA. The authors attribute VeRA's underperformance to its shared global matrices, which may restrict updates to a single subspace, hindering knowledge preservation.
Data Mixture Sensitivity: Introducing even a small proportion (1%) of TL data significantly enhanced target language performance with minimal degradation to English performance. Increasing the TL proportion to 10% or 50% generally resulted in performance degradation for both languages.
Rank Sensitivity: Target language performance was highly sensitive to rank scaling. While Rank 8 was the standard, Rank 32 yielded peak TL performance across nearly all variants, whereas Rank 16 often showed degradation.
Computational Efficiency: No single variant dominated all efficiency metrics. Basic LoRA and AdaLoRA demonstrated favorable efficiency profiles, while DoRA and DoRA* were less efficient across most metrics (training time, GPU utilization, memory).
Internal Representations: Layer-wise analysis revealed that language representation patterns (transitioning from initial layers to English-dominant middle layers, then declining dominance in outer layers) remained largely similar across all LoRA techniques. This suggests that architectural modifications in these variants do not fundamentally alter the LLM's internal language representation.

Significance and Claims
The paper claims that for multilingual instruction tuning, architectural novelty does not necessarily translate into performance gains. The study challenges the assumption that more complex LoRA variants are superior for cross-lingual adaptation. Key takeaways include:

Simplicity Suffices: Basic LoRA is sufficient for balancing cross-lingual transfer and knowledge retention, offering a more computationally efficient alternative to complex variants.
Layer Application: Contrary to some pre-training findings, the authors find that in instruction tuning, LoRA should be applied to all layers rather than just the final layers.
Representation Stability: The lack of significant change in hidden embeddings across variants suggests that the "black box" of LLM adaptation remains stable regardless of the specific LoRA variant used, implying that the benefits of these variants may be task-specific rather than universal for multilingual settings.

The authors conclude that their work provides a broader investigation into language learning in LLMs compared to prior studies, covering a wider range of languages, model architectures, and internal representation analyses, ultimately suggesting that researchers should reconsider the necessity of complex LoRA variants for multilingual instruction tuning tasks.

More like this