🤖 AI

How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model

This study systematically evaluates the trade-offs between LoRA rank, target modules, and quantization on a 60M-parameter T5-small model for text-to-SQL, demonstrating that a rank of 16 achieves near-full fine-tuning accuracy with minimal parameter overhead while INT8 and NF4 quantization enable memory-constrained deployments with comparable performance.

Original authors: Mahendra Singh Rathor, Anagheem Azzam

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahendra Singh Rathor, Anagheem Azzam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot brain that knows almost everything. It's so powerful it can write stories, solve math problems, and even translate languages. But there's a catch: this brain is so huge and heavy that it needs a massive, expensive supercomputer to run. It's like trying to drive a semi-truck just to go to the corner store for milk. Most people can't afford the gas or the garage for a truck that big.

Scientists have found a clever trick to fix this. Instead of building a new, smaller brain from scratch, they take the giant one and give it a tiny, lightweight "training vest." This vest teaches the brain how to do one specific job, like turning English sentences into database commands (SQL), without changing the brain's entire structure. This is called Parameter-Efficient Fine-Tuning. Think of it like putting a specialized tool belt on a general contractor; the contractor doesn't need to learn a new trade, they just need the right tools for the specific job.

But there's another trick to save space: Quantization. Imagine the giant brain's knowledge is written in giant, heavy stone tablets. Quantization is like carefully carving those tablets down to lightweight plastic chips. The information is still there, but it takes up way less room. The big question scientists are asking is: If we use a tiny brain to begin with, how small can we make these training vests and plastic chips before the brain starts forgetting how to do its job?

The Great "How Small Can We Go?" Experiment

Two independent researchers, Mahendra and Anagheem, decided to answer this question by running a very careful, controlled experiment. They didn't use a massive, billion-parameter brain (which would be too expensive to test every possibility). Instead, they picked a specific, manageable 60-million-parameter model called T5-small. Think of this as a compact, efficient sedan rather than a semi-truck. They tasked it with a specific job: translating English questions into SQL queries (the language databases use to find information) using a dataset called WikiSQL.

Their goal was to turn three "efficiency knobs" one at a time to see exactly how much accuracy they would lose for every bit of memory they saved.

Knob 1: The Size of the Training Vest (LoRA Rank)

The first knob controlled the size of the "training vest" (called LoRA). Imagine the vest has different numbers of pockets. A vest with 2 pockets is tiny; a vest with 32 pockets is much bigger. The researchers tested vests with 2, 4, 8, 16, and 32 pockets.

They found something surprising: Bigger isn't always better.

When they increased the pockets from 2 to 16, the robot's performance jumped significantly. It went from getting 38.6% of answers right to 59.6% right.
However, once they hit 16 pockets, adding more didn't help much. Going from 16 to 32 pockets only boosted the score by a tiny 0.8 points (to 60.4%).
The Finding: For this specific job, a vest with 16 pockets was the sweet spot. Adding more pockets was just extra weight with no real reward. The robot had already learned everything it needed to know.

Knob 2: Where to Put the Vest (Target Modules)

The second knob decided where on the robot's body the vest went. The robot has different parts: some parts handle "attention" (focusing on specific words), and others handle "feed-forward" layers (processing the logic).

They tried putting the vest only on the "query" and "value" parts (the most critical focus areas).
They tried putting it on the whole attention system.
They tried adding the logic-processing parts too.

The Finding: It was more efficient to make the vest slightly bigger (increase the rank to 16) than to wrap it around more parts of the robot. Expanding the vest to cover extra body parts gave very little extra accuracy for the extra weight. The "16-pocket vest on the focus areas" was the most efficient combination.

Knob 3: The Material of the Brain (Quantization)

The third knob changed the material of the robot's brain from heavy stone (standard precision) to lightweight plastic (INT8 and NF4 quantization).

The Result: This was a game-changer for memory. By switching to the plastic chips, they slashed the memory needed to train the robot from 2.31 GB down to just 0.60 GB. That's a 74% reduction!
The Trade-off: The robot's accuracy dropped slightly, from 59.6% (with the heavy stone) to about 53% (with the plastic). However, the researchers noted that this tiny drop in accuracy was worth the massive saving in space. It's like trading a slightly less comfortable seat for a car that fits in a tiny garage.

The Final Verdict: The Pareto Sweet Spot

The researchers plotted all their results to find the "Pareto front"—a fancy way of saying "the best possible deal." They wanted the highest accuracy for the lowest cost.

The Goldilocks Zone: The absolute best balance for most people was using the 16-pocket vest (LoRA rank 16) on the focus areas with the heavy stone brain. This setup got 59.6% accuracy while using only 1.60 GB of memory and training less than 1% of the robot's total brain. It recovered about 83.7% of the performance of the fully trained giant robot.
The Ultra-Light Option: If you are in a situation where you have almost no memory (like running on a very old phone), the plastic brain (NF4 quantization) with the 8-pocket vest was the winner. It used only 0.60 GB of memory and still got 53.2% accuracy.

What This Means for You

The paper suggests that for small models doing specific tasks, you don't need to over-engineer the solution.

Don't go too small: A vest with only 2 or 4 pockets leaves the robot confused.
Don't go too big: Once you hit 16 pockets, you're just wasting space.
Don't over-wrap: It's better to make the vest on the right parts slightly bigger than to wrap it around the whole robot.

The researchers ran these tests three times with different random starting points to make sure the results weren't just luck. The results were consistent: the "16-pocket" rule held up every time. They also noted that while this works great for single-table questions, we don't know yet if these rules apply to much harder, multi-table puzzles. But for now, if you want to run a smart robot on a budget, this study gives you a clear, reproducible recipe: Keep the vest size at 16, focus on the right parts, and if you're really tight on space, switch to the plastic brain.

Technical Summary: How Small Can You Go?

Problem Statement
Parameter-efficient fine-tuning (PEFT) and low-bit quantization are standard approaches for adapting large language models under tight compute budgets. However, the interaction between these efficiency mechanisms is predominantly studied on billion-parameter models, where exhaustive exploration of the design space (rank, target modules, precision) is prohibitively expensive. This paper addresses a complementary question: on a specific, fully reproducible 60M-parameter encoder–decoder model (T5-small) and a single-table text-to-SQL benchmark (WikiSQL), what is the precise cost of task accuracy for each efficiency knob? The study aims to frame adaptation as a constrained trade-off between accuracy and system-level metrics (trainable parameters, memory, latency, throughput) rather than an accuracy-only objective.

Methodology
The authors conducted a controlled, single-variable ablation study using T5-small (60M parameters) on the WikiSQL dataset. The experimental setup involved:

Base Task: Natural-language-to-SQL generation, trained on 5,000 examples and evaluated on 500 validation examples.
Variables Swept:
1. LoRA Rank ( $r$ ): Tested values of $\{2, 4, 8, 16, 32\}$ .
2. Adapted Modules: Variations included attention query/value only $\{q,v\}$ , full attention $\{q,k,v,o\}$ , and attention plus feed-forward layers $\{q,v, \text{FFN}\}$ .
3. Numerical Precision: FP16 baseline, INT8 (LLM.int8()), and NF4 4-bit (QLoRA).
Metrics: The study reported Exact-Match (EM) accuracy and Execution accuracy alongside trainable parameter counts, peak training memory, inference latency, and throughput.
Analysis Framework: Results were analyzed as a constrained optimization problem (maximizing accuracy subject to memory or parameter budgets) and visualized via Pareto frontiers. All configurations were run with three random seeds (42, 123, 456) to assess variance.

Key Contributions

Reproducible Trade-off Study: A fully reproducible, single-variable analysis of LoRA rank and target-module selection on a 60M-parameter model, a regime where full design space sweeps are feasible.
Joint Metric Reporting: Simultaneous reporting of task accuracy and four system-level metrics, utilizing Pareto analysis to identify the "sweet spot" for accuracy versus cost.
Empirical Findings on Small Models: Evidence that in the T5-small/WikiSQL setting, LoRA rank saturation occurs early ( $r=16$ ) and that expanding adapted modules beyond $\{q,v\}$ yields diminishing returns compared to increasing rank.
Open Release: Public availability of all code, configurations, and training logs to facilitate further research.

Results

LoRA Rank: Increasing rank from $r=2$ to $r=16$ yielded consistent accuracy gains (38.6% to 59.6% EM). Beyond $r=16$ , accuracy plateaued sharply; $r=32$ achieved only 60.4% EM, a negligible gain of 0.8 points despite doubling trainable parameters. Peak memory remained flat (1.59–1.61 GB) across all ranks, indicating adapter weights are negligible relative to the frozen base.
Adapted Modules: Expanding from $\{q,v\}$ to full attention $\{q,k,v,o\}$ at $r=8$ recovered 5.8 accuracy points but required 71% more parameters. However, increasing the rank to $r=16$ on just $\{q,v\}$ matched the performance of the full attention set at the same parameter budget, suggesting rank is a more efficient lever than module expansion in this setting.
Quantization (QLoRA): INT8 and NF4 quantization achieved comparable accuracy to the FP16 LoRA $r=8$ baseline (52.8% and 53.2% vs. 53.4%) with dramatic memory savings. Both QLoRA configurations required only 0.60 GB peak training memory (a 62% reduction vs. FP16 LoRA and 74% vs. full fine-tuning). NF4 offered the best trade-off, providing near-FP16 accuracy at 0.60 GB with 34 tokens/second throughput.
Pareto Optimality: The configuration of LoRA $r=16$ on $\{q,v\}$ in FP16 emerged as the Pareto-optimal choice for unconstrained memory, recovering 83.7% of full fine-tuning accuracy (59.6% vs. 71.2% EM) while training only 0.97% of parameters and using 31% less peak memory.
Variance: Lower-rank configurations (e.g., $r=2$ ) showed higher variance across seeds, while quantized configurations were notably stable, likely due to quantization regularizing the optimization landscape.

Significance and Claims
The paper claims to provide a clear practical recipe for the T5-small/WikiSQL setting: use LoRA with $r=16$ on $\{q,v\}$ modules for the best balance of accuracy and cost, or QLoRA NF4 for strict memory-constrained deployments. The authors explicitly state that these findings are specific to this setting and should not be interpreted as general properties of all small language models. They note that rank saturation at $r=16$ contrasts with findings on larger models and may not transfer to multi-table benchmarks like Spider or BIRD. The study emphasizes that its goal is to characterize relative efficiency trade-offs rather than achieve state-of-the-art absolute accuracy, offering a foundation for reproducible, deployment-aware PEFT research in the small-model regime.