Technical Summary: PreFT: Prefill-only finetuning for inference efficiency

Problem Statement

Large Language Models (LLMs) are increasingly personalized for specific users or tasks using Parameter-Efficient Finetuning (PEFT) methods like LoRA. While PEFTs reduce training memory footprints, serving thousands of user-specific adapters simultaneously creates a significant bottleneck in inference throughput.

The core issue lies in the architectural mismatch between the two phases of LLM inference:

1. Prefill: Processing the input prompt in parallel. This phase is compute-bound.
2. Decode: Generating tokens autoregressively, one by one. This phase is memory-bound due to the need to load large Key-Value (KV) caches and adapter weights for every request.

Current multi-adapter serving systems (e.g., S-LoRA, Punica) attempt to optimize this by using custom kernels and memory paging. However, even with these optimizations, serving multiple PEFT adapters during the decode phase imposes substantial memory overhead. Since the decode phase is memory-bound, the cost of loading unique adapter weights for each request in a batch erodes the throughput gains achieved by batching requests together.

Methodology: PreFT (Prefill-only Finetuning)

The authors propose PreFT, a paradigm shift that restricts the application of PEFT adapters to the prefill phase only. Once the prompt is processed, the adapter is discarded, and the model generates subsequent tokens using only the frozen base model weights.

The paper implements this concept by modifying two existing PEFT methods:

• LoRAP (LoRA Prefill-only): A variant of Low-Rank Adaptation (LoRA) where the low-rank update is applied only to the input prompt tokens.
• DiReFTP (DiReFT Prefill-only): A variant of Representation Finetuning (ReFT), specifically the DiReFT parametrization, which intervenes on residual streams only at prompt positions.

Technical Implementation:

• The authors forked the vLLM inference engine to integrate these prefill-only adapters.
• They implemented a position-mask mechanism that routes adapter interventions strictly to the prefill phase.
• Crucially, for the decode phase, the engine bypasses the adapter logic entirely, treating the request as if it were using the base model.
• This approach eliminates the memory-bound cost of loading and computing with adapter weights during token generation, allowing the system to batch more requests without the per-request adapter overhead.

Key Contributions

1. Conceptual Shift: The paper argues that for personalized serving, the optimization goal should shift from "performance per parameter" to "performance per throughput." It identifies that adapter operations should be placed in the compute-bound prefill phase rather than the memory-bound decode phase.
2. System Implementation: The authors provide an efficient implementation of PreFT (LoRAP and DiReFTP) within vLLM, including support for multi-adapter serving, memory paging, and weight synchronization for on-policy Reinforcement Learning (RL).
3. Theoretical Justification: The paper demonstrates that the down-projection of LoRA and ReFT is memory-bound even in half-precision. By removing these operations from the decode phase, PreFT avoids the primary bottleneck of multi-adapter serving.

Results

Inference Efficiency

PreFTs demonstrate significant throughput improvements over traditional all-position PEFTs when serving multiple adapters:

• Llama 3.1 70B: Serving 512 adapters on 4 H100 GPUs, PreFT achieved 1.9× the throughput of standard multi-LoRA for DiReFTP and 1.87× for LoRAP.
• Smaller Models: Similar gains were observed on Llama 3.1 8B (2.21× / 1.83×) and Qwen 2.5 0.5B (2.39× / 1.63×).
• These gains were achieved without custom kernels for ReFT, relying instead on the architectural removal of the decode-time cost.

Performance (Accuracy)

The authors evaluated PreFTs against all-position PEFTs across Supervised Finetuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR):

• SFT (Instruction Following): PreFTs exhibited higher evaluation loss than all-position PEFTs on training data. However, on downstream benchmarks (IFEval, MMLU, GSM8K), there was no statistically significant difference in accuracy between PreFTs and standard PEFTs when parameters were matched.
• RLVR (Math and Code):
  • On MATH and HumanEval (code generation), PreFTs approached parity with standard PEFTs.
  • On GSM8K, PreFTs consistently lagged behind all-position PEFTs, though the gap was often small.
• Long-Form Generation: In experiments with LongWriter (generating up to 20k tokens), LoRAP maintained length-following capabilities and quality comparable to standard LoRA. Conversely, DiReFTP failed to maintain conditional behavior over long generations, reverting to a bias toward writing excessively long outputs.

Significance and Claims

The paper claims that PreFT offers a more favorable accuracy–throughput tradeoff for personalized LLM serving than existing PEFT methods.

• Efficiency: By moving adapter operations to the compute-bound prefill phase, PreFTs significantly reduce the memory overhead during the decode phase, enabling higher throughput in multi-user scenarios.
• Practicality: The authors assert that for many use cases (style adaptation, per-user memory), PreFTs sacrifice little to no downstream performance while delivering substantial efficiency gains.
• Limitations: The paper modestly acknowledges that PreFTs are not a universal replacement. Specifically, DiReFTP struggles with long-form generation, and GSM8K performance remains slightly lower than all-position adapters. The authors suggest that future work could explore novel architectures specifically designed for prefill-only constraints to close these remaining gaps.

In conclusion, PreFT represents a systems-level optimization that leverages the distinct computational characteristics of LLM inference phases to enable scalable, personalized model serving.