Technical Summary: Semantic Cache Distillation (SCD)

1. Problem Statement

The paper addresses a critical bottleneck in disaggregated Large Language Model (LLM) serving, where the prefill (compute-bound) and decode (memory-bound) stages are separated across different devices to maximize resource utilization. While this architecture alleviates memory constraints, it introduces a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches from the prefill producer to the decode consumer often dominates the Time-to-First-Token (TTFT).

Existing solutions face two primary limitations in this context:

1. Bandwidth Constraints: Raw KV transmission is bandwidth-prohibitive, especially over slow interconnects. Compression techniques like quantization or sparsification reduce payload size but assume the producer and consumer share identical weights (same latent space).
2. Semantic Drift in Heterogeneous Models: In practical scenarios (e.g., a base model serving as a producer for multiple fine-tuned specialists), the producer and consumer have different weights. Directly reusing raw or compressed KV caches from a base model to a fine-tuned model causes semantic misalignment. These representation mismatches compound over layers, leading to significant degradation in generation quality.

Current approaches like DroidSpeak attempt to mitigate this via selective recomputation (recomputing critical layers while reusing raw caches for others), but this enforces a rigid trade-off: prioritizing reuse risks quality, while prioritizing recomputation sacrifices the latency benefits of disaggregation.

2. Methodology: Semantic Cache Distillation (SCD)

The authors propose Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV transmission with compact semantic codes. SCD is designed for producer-consumer pairs that share the same Transformer architecture but differ in weights. It operates via two complementary mechanisms:

A. REUSE (Low-Rank Reconstruction)

For the majority of layers where cross-model compatibility is high, SCD leverages the low-rank structure of KV tensors.

• Mechanism: The producer encodes its high-dimensional KV states into low-rank semantic codes ($Z$) using a learned linear projection. The consumer reconstructs its native-space KV states by decoding these codes using a layer-specific decoder.
• Offline Calibration: A small set of paired prefixes is used to perform joint low-rank factorization on the producer and consumer states, identifying a shared latent space and training layer-specific encoders/decoders.
• Goal: Minimize transfer bandwidth by transmitting low-dimensional codes ($r \ll d$) instead of full KV caches.

B. PATCH (Selective Semantic Correction)

For a sparse set of critical "transition layers" where feature mismatches amplify and cause significant drift, SCD employs a targeted correction mechanism.

• Mechanism: Instead of transmitting raw KV states or full recomputation, the producer transmits a compressed code of the normalized pre-attention input ($H_{in}$). The consumer uses a lightweight, non-linear aligner (an MLP) to map this code to its native pre-attention space.
• Regeneration: Once the aligned pre-attention state is recovered, the consumer regenerates the KV pairs using its own native weights.
• Goal: Truncate error propagation. Theoretical analysis shows that without Patch, errors accumulate multiplicatively across layers. Patch acts as a spectral truncation operator, resetting the error trajectory at specific intervals.

C. Theoretical Foundation

The paper formalizes the problem as a rate-distortion optimization. Theoretical analysis (Lemma 5.1 and Theorem 1) demonstrates that:

• Without Patch, reconstruction errors compound multiplicatively ($\prod \alpha_i$), leading to large drift in deep stacks.
• Inserting Patch layers effectively resets the error accumulation, bounding the total divergence by the local reconstruction error of the patch rather than the accumulated history.

3. Key Contributions

1. SCD Framework: An end-to-end design integrating REUSE for bandwidth efficiency and PATCH for semantic alignment, enabling low-latency state transfer between weight-mismatched models.
2. Heterogeneous State Transfer Formulation: A formalization of the cache reuse problem under representation mismatch, identifying why static compression and raw transfer fail in disaggregated settings.
3. Selective Correction Mechanism: The introduction of the PATCH module, which applies lightweight non-linear alignment at sparse transition layers to truncate error propagation with minimal computational overhead.
4. Empirical Validation: Demonstration that SCD achieves superior latency-quality trade-offs in bandwidth-constrained environments, outperforming quantization and selective recomputation baselines.

4. Experimental Results

The authors evaluated SCD on Mistral-7B and Qwen-32B pairs (base model as producer, fine-tuned variant as consumer) across QA (CMRC2018, HotpotQA) and Language Modeling (WikiText-2) tasks.

• Latency vs. Quality: SCD achieves up to a 2.65× speedup in TTFT compared to the "Oracle" (full consumer-side prefill) while maintaining generation quality within 5% F1 of the Oracle.
• Comparison with Baselines:
  • Raw KV: Fast but suffers from massive quality collapse (F1 ~0.19 vs Oracle 0.81) due to semantic drift.
  • Quantized KV (4-bit): Fastest but results in catastrophic quality failure (F1 ~0.13).
  • DroidSpeak-style Selective Recompute: Improves quality but incurs higher latency (1.70× speedup) and larger payload sizes than SCD.
  • SCD (Reuse-only): Improves quality over raw reuse but still lags behind Oracle.
  • SCD (Full): With Patch, SCD recovers nearly Oracle-level quality (F1 0.785 vs 0.810) with a 2.65× speedup, dominating the Pareto frontier.
• Ablation Studies:
  • Patch Budget: Increasing the number of patched layers ($k$) monotonically improves perplexity (PPL), confirming that error propagation is concentrated in a few critical layers. A budget of $k=6$ was found to be optimal.
  • Rank Sensitivity: The framework allows for tunable compression; reducing the rank for Keys ($r_K$) yields substantial compression with minimal quality loss, suggesting Keys carry less fine-grained semantic information than Values.

5. Significance and Claims

The paper claims that SCD provides a practical solution for shared-prefill, specialized-decode serving, a deployment pattern where a single base model processes a common prefix (e.g., user history, instructions) and routes the state to multiple task-specialized consumers.

• Bridging the Gap: SCD successfully bridges the semantic gap between heterogeneous models without requiring full recomputation or identical weights.
• Efficiency: It shifts the adaptation cost to a bounded offline calibration phase (one-time per model pair), allowing online serving to proceed with minimal overhead.
• Scalability: The approach scales to larger models (32B parameters) and maintains effectiveness across different model families.
• Limitations: The authors modestly note that SCD is currently limited to shared-architecture pairs, does not yet handle extreme long-context regimes (>32K tokens) or arbitrary cross-architecture transfers, and requires one-time calibration per pair.

In summary, SCD transforms the state transfer problem from a raw data transmission challenge into a semantic alignment problem, enabling efficient, high-quality disaggregated serving for heterogeneous LLM ecosystems.