Technical Summary: Cross-Model KV Cache Transfer in LLM Families

1. Problem Formulation

Production Large Language Model (LLM) serving increasingly relies on multi-model orchestration for cost-quality cascading, mid-conversation switching, and routing. These workflows frequently swap between different-sized members of the same model family (e.g., Qwen3 14B to 32B). While these models share core architectural choices and pretraining data, they differ in scale, layer count, and training recipes.

The primary bottleneck in such workflows is the prefill cost. Every time a conversation context is handed off from a source model to a target model, the target must re-process the entire accumulated context to populate its own Key-Value (KV) cache. This forward pass scales with both model size and prompt length, negating the efficiency gains of model swapping. Existing solutions like prefix caching only work within a single model, and prior cross-model approaches (e.g., neural fusers, latent adapters) typically require gradient-based training or strict architectural constraints (e.g., identical layer counts).

This paper proposes Cross-Model KV Cache Transfer, a method to reuse the source model's pre-filled KV cache directly in the target model, skipping the target's prefill entirely. The goal is to find a mapping $f: C_S \to \hat{C}_T$ such that the target model decoding from the mapped cache $\hat{C}_T$ produces outputs equivalent to decoding from its own ground-truth cache $C_T$.

2. Methodology

The authors hypothesize that cross-model KV relationships possess substantial linear structure, allowing for a closed-form, gradient-free mapping. The proposed framework operates on "matched-KV" pairs, where the source and target share the same number of KV heads and per-head dimension, even if their total layer counts or parameter sizes differ.

The mapper is designed with three core components:

A. Per-Head Ridge Regression

Instead of a global neural network, the method fits an independent linear mapping for each target layer and head.

• Input: Concatenated KV features from a selected subset of source layers.
• Output: The target's expected Key ($K$) and Value ($V$) vectors.
• Optimization: The weights are solved via ridge regression (closed-form solution) on a small calibration set (500 sequences of 1,024 tokens from FineWeb-Edu). Ridge regularization ($\lambda=0.01$) is used to handle the near-singular nature of the feature matrix when source layers are correlated.

B. Cross-Layer Source Selection

Since source and target models have different depths ($L_S \neq L_T$), the method must determine which source layers inform each target layer.

• Selection Strategy: For each target layer, the top-$k$ source layers are selected based on their predictive power (measured by head-averaged $R^2$ on a calibration set).
• Mechanism: The KV features from these top-$k$ source layers are concatenated to form the input for the ridge regression. Empirical analysis shows that complementary information is distributed across multiple source layers; a single layer is insufficient, but performance saturates quickly (e.g., $k=6$ captures most of the variance of $k=all$).

C. Content-Space Mapping (RoPE Factoring)

Standard LLMs use Rotary Position Embeddings (RoPE), which apply position-dependent rotations to keys.

• Problem: Fitting a mapper directly on rotated keys ties the weights to the specific position distribution seen during calibration, limiting transferability to different context lengths.
• Solution: The method strips RoPE from the source keys before mapping and re-applies the target's RoPE after mapping.
  • $K_{stripped} = K_{source} \cdot R^{-1}_{\Theta_s}$
  • $\hat{K}_{target} = (K_{stripped} \cdot W + b) \cdot R_{\Theta_t}$
• This decouples positional rotation from semantic content, making the linear fit position-free and reusable across context lengths (up to 32k tokens).

D. Nonlinear Extension (MLP)

For pairs where the linear ridge mapper fails to retain sufficient accuracy, the authors propose a drop-in replacement using a small Multi-Layer Perceptron (MLP). This MLP redistributes residual error away from attention-sensitive subspaces, recovering performance on difficult pairs.

3. Key Contributions

1. Closed-Form Mapping Framework: A gradient-free, per-head ridge regression framework that enables cross-model KV transfer without training. It relies on top-$k$ source layer selection and RoPE-stripped content mapping.
2. Multi-Family Validation: Validation across three model families (Qwen3, Llama 3.1, Ministral 3) and six matched-KV pairs.
3. Nonlinear Recovery: Demonstration that a lightweight MLP can recover performance on "hard" pairs where linear mapping fails, specifically by redistributing error away from attention-critical dimensions.
4. Diagnostic Metric: Identification that attention-output cosine similarity is a better predictor of downstream transfer retention than standard reconstruction metrics ($R^2$). The authors show a Pearson correlation of $r=+0.57$ between attention-output cosine and HellaSwag retention, compared to $r=-0.20$ for $R^2$.

4. Experimental Results

The framework was evaluated on six matched-KV pairs across three families using five accuracy benchmarks (ARC-C, HellaSwag, WinoGrande, MMLU, GSM8K) and WikiText-2 perplexity.

• Accuracy Retention:
  • Tier 1 (Success): Four pairs (Qwen3 14B→32B, 8B→32B; Llama 3.1 8B→70B; Ministral 3B→8B) retained 73–98% of the target model's standalone accuracy.
  • Tier 2 (Failure): Two pairs (Ministral 3B→14B, 8B→14B) degraded significantly (42–44% retention). However, replacing the linear mapper with an MLP recovered HellaSwag retention by +24.3 to +36.8 percentage points, bringing these pairs above 90%.
• Latency:
  • The mapper runs 2.7–25× faster than re-prefilling the target model.
  • Speedups are most pronounced for small-to-large transfers and longer context lengths (e.g., 25× speedup at 32k tokens for Qwen3 14B→32B).
• Multi-Turn Stability:
  • In multi-turn handoff scenarios (CoQA), drift remains small. Small-to-large drift widens by only 1.7 pp over 10 turns; large-to-small drift grows linearly at 0.33 pp/turn, suggesting stability for typical session lengths.
• Calibration Efficiency:
  • The mapper requires only 500 sequences for calibration.
  • Fitting takes ~47–87 minutes per pair on a single 8×H100 node, with no gradient-based training.

5. Significance and Claims

The paper claims that cross-model KV cache transfer within an LLM family admits a closed-form, training-free fit. The authors argue that the cross-model KV relationship is largely linear, challenging the assumption that complex neural adapters are necessary for such transfers.

The significance of this work lies in:

1. Practicality: It enables cost-quality cascading and mid-conversation switching without the prohibitive cost of re-prefilling, making dynamic model orchestration feasible in production.
2. Shift in Evaluation: The findings suggest that evaluation criteria for cross-model mappers should shift from raw reconstruction metrics ($R^2$) to subspace-aware diagnostics (like attention-output cosine), as the latter better predicts downstream task performance.
3. Error Redistribution: The work highlights that transfer failure is often due to where the error lands (in attention-sensitive subspaces) rather than the magnitude of the error, motivating future attention-aligned objectives.

The authors remain modest regarding limitations, noting that the approach is currently restricted to within-family, dense full-attention models with matched KV configurations, and that calibration is performed on a single domain (FineWeb-Edu). They leave cross-family transfer and mismatched-KV conditions as open questions for future work.