Technical Summary: Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation

Problem Statement

Retrieval-Augmented Generation (RAG) has become the standard approach for grounding Large Language Models (LLMs) in external corpora, particularly for documentation-oriented assistants where answers must be backed by specific configuration flags, API versions, or feature gates. However, deploying such systems involves direct tensions between competing objectives: larger generators improve answer quality but increase inference memory and latency; richer retrieval pipelines add overhead; and more expressive parameter-efficient fine-tuning (PEFT) configurations alter training costs.

Existing literature often studies these dimensions in isolation—focusing on retrieval design, general LoRA applications, or standard QA evaluation—or treats LoRA as a monolithic technique rather than a configuration space. There is a lack of systematic empirical analysis regarding how specific LoRA configurations (rank, target modules, base model size) interact with retrieval quality, latency, and training cost within a fixed RAG architecture for technical documentation.

Methodology

Benchmark Construction

The authors constructed a manually verified benchmark of 5,144 question–answer pairs derived from official Kubernetes documentation. The dataset was created through a hybrid process: 500 pairs were hand-written, and the remainder were drafted by an LLM agent (GPT-5.4) and subsequently manually verified. The pairs are categorized into EXACT (requiring literal reproduction of flags, paths, or versions) and NORMAL (factual answers tolerating paraphrasing). The data is split into train (3,614), eval (745), and test (785) sets.

RAG Pipeline

A fixed hybrid-retrieval pipeline was employed across all experiments to isolate the generator's contribution:

1. Retrieval: A hybrid approach combining dense retrieval (BGE-M3 embeddings) and native sparse retrieval (BGE-M3 sparse channel), merged via Reciprocal Rank Fusion (RRF).
2. Reranking: A pretrained cross-encoder reranker (BAAI/bge-reranker-v2-m3).
3. Generation: Two instruction-tuned base models were used: Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct.

Experimental Design

The study ablated 20 LoRA configurations across the two base models. The configuration space varied two factors:

• Rank ($r$): $\{4, 8, 16, 32, 64\}$.
• Target Modules:
  • $qv\_only$: Adapts only the query ($q$) and value ($v$) attention projections.
  • $full\_attention$: Adapts all four major attention projections ($q, k, v, o$).

Two unadapted baselines (3B and 8B) served as controls. All models were trained with fixed hyperparameters (8 epochs, learning rate $2 \times 10^{-5}$, bf16 precision) on a single hardware node (NVIDIA A100 40GB) to ensure comparable cost metrics.

Evaluation Metrics

Performance was evaluated on a multi-objective Pareto front using:

• Quality: Token-level F1 score (primary), LLM-judge groundedness pass@4, and correctness pass@4.
• Cost: Inference latency, peak inference VRAM, total training time, and peak training VRAM.
• Statistical Rigor: All point estimates include bootstrap 95% confidence intervals (1,000 resamples). Paired bootstrap tests were used for $\Delta$F1 comparisons.

Key Results

1. Structural Advantage of $qv\_only$ Adaptation

Across every Pareto front examined (F1 vs. latency, F1 vs. inference memory, F1 vs. training cost), the non-dominated points were consistently dominated by $qv\_only$ adapters.

• $full\_attention$ adapters never appeared on the training Pareto front and won at most 2 out of 10 retrieval/prompting ablation regimes on groundedness.
• Param-Matched Control: To rule out the possibility that $qv\_only$ wins simply because it has more parameters per projection, the authors compared $qv\_only$ at rank $r$ against $full\_attention$ at rank $r/2$ (equal total parameter count). The $qv\_only$ scheme remained superior or statistically indistinguishable in all cases, with a mean $\Delta$F1 of $+0.007$ [0.001, 0.012]. This suggests the advantage is structural, likely because adapting $q$ and $v$ projections is sufficient for refining context selection and re-weighting in a fixed RAG pipeline, whereas adapting $k$ and $o$ adds cost without proportional gain.

2. Base Model Size Defines the Operating Regime

The choice between 3B and 8B models primarily determines the operating regime (resource budget) rather than the achievable quality ceiling.

• The strongest adapted 3B configuration (3B r64 $qv\_only$) achieved an F1 of 0.597, which is statistically indistinguishable from the unadapted 8B baseline (F1 0.595, $\Delta$F1 $\in [-0.021, +0.026]$).
• This 3B configuration costs roughly 9 GB less in inference VRAM than the 8B baseline.
• While the 8B model ($r64$ $qv\_only$) achieved the highest absolute F1 (0.617), the gain over the 3B adapted model was small ($\Delta$F1 +0.020) and came with a significant resource penalty.

3. Groundedness vs. F1 Trade-off

The configuration maximizing F1 (8B r64 $qv\_only$) did not maximize groundedness. The highest groundedness score was achieved by 8B r16 $qv\_only$ (F1 0.615 vs. 0.617). The authors note that F1 and groundedness are not interchangeable; the optimal configuration depends on whether the deployment prioritizes exact-match agreement or strict support by retrieved context.

4. Robustness Across Retrieval and Prompting

The study tested 10 different regimes by varying retrieval strategies (dense-only, sparse-only, BM25 hybrid, reranker off) and prompting modes (neutral vs. explicit grounded).

• The $qv\_only$ scheme consistently yielded the best F1 configuration in 10/10 regimes.
• The $qv\_only$ scheme yielded the best groundedness configuration in 8/10 regimes.
• While the specific optimal rank or model size shifted slightly, the structural dominance of the $qv\_only$ adaptation scheme remained stable.

Significance and Contributions

The paper claims four primary contributions:

1. A Manually Verified Benchmark: A license-clean, multi-criteria RAG benchmark over Kubernetes documentation designed for evaluating quality-cost trade-offs.
2. Systematic Configuration Study: A structured ablation of LoRA configurations within a fixed hybrid-retrieval pipeline, providing statistical inference on all reported trade-offs.
3. Param-Matched Control: A comparison isolating the structural effect of target-module choice from the effect of parameter count, demonstrating that $qv\_only$ is structurally superior for documentation RAG.
4. Reproducible Artifacts: The release of the benchmark, selected adapters, and code to serve as a starting point for future work on retrieval-augmented fine-tuning.

The authors conclude that for documentation-oriented RAG, backbone size sets the regime, but the adapter scheme determines the efficiency. Targeted LoRA adaptation (specifically $qv\_only$) allows smaller models to reach the quality of larger unadapted baselines at a fraction of the cost, and broader adaptation (full attention) is generally not justified by the marginal quality gains in this specific domain.