💬 NLP

WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs

This paper introduces WnW, a dynamic KV cache management strategy that classifies attention heads into anchor, tidal, and fixed roles to enable efficient long-form speech processing by maintaining only 20% of audio tokens on the GPU while preserving near-full-cache accuracy through selective CPU-GPU recall.

Original authors: Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, Jinsong Su

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiming Yao, Chenyang Lyu, Xuanfan Ni, Longyue Wang, Weihua Luo, Yazheng Yang, Jinsong Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers have become remarkably adept at listening. They can take hours of raw sound—meetings, lectures, long conversations, or even entire audiobooks—and turn them into written text or translate them into other languages. This ability relies on powerful artificial intelligence models that process audio by breaking it down into tiny, rapid snapshots called tokens. As these models listen, they build a temporary memory of what they have heard so far, storing the most important details to help them understand the context of the conversation. This memory is essential; without it, the model would forget the beginning of a sentence by the time it reached the end. However, as the audio gets longer, this memory requirement grows rapidly, eventually becoming so large that it overwhelms the computer's available space, causing the system to crash or produce gibberish.

The core problem is that the way these models decide what to remember changes as the conversation progresses. When a model first hears a long recording, it tends to focus heavily on the very beginning, assuming that the start holds the most critical clues. But as the model starts to speak or translate, its attention shifts. It begins to look at different parts of the audio, sometimes far away from the start, to make sense of the current moment. Older methods tried to solve the memory problem by making a permanent decision at the very start: they would look at the beginning of the audio, pick the parts they thought were important, and throw the rest away forever. This approach works fine for short clips, but for long recordings, it is a gamble that often fails. The model discards the very information it later needs, leading to errors or a complete inability to finish the task.

A team of researchers has developed a new approach called WnW, short for Waxing-and-Waning, which treats memory not as a static list but as a dynamic flow. Instead of making a final decision about what to keep before the model starts speaking, this new method allows the system to change its mind as it goes. The researchers found that not all parts of the model's brain are equally important for remembering audio. Some parts are deeply connected to the sound itself and are critical for accuracy, while others are less sensitive. Using a careful calibration process, they sorted the model's internal components into three groups. One small group, called "anchor" heads, stays fully active and acts as a guide, constantly watching the audio to see what is important right now. A second group, called "tidal" heads, keeps a partial memory on the main computer chip but stores the rest on a slower, secondary storage drive. As the model speaks, the anchor heads signal which parts of the audio are currently relevant, and the system quickly fetches those specific chunks from the secondary drive, swapping them in to replace the ones that are no longer needed. The third group, "fixed" heads, keeps only a small, permanent slice of the audio, discarding the rest, because these parts of the model do not need the full picture to function.

This strategy proved to be a game-changer for long-form audio. When tested on recordings lasting several minutes, the new method allowed the model to run smoothly on standard computer hardware, keeping only about 20 percent of the audio data in the fast, primary memory at any given time. Despite this drastic reduction, the model's accuracy remained nearly identical to a system that kept every single piece of data. In contrast, the older methods that made permanent decisions at the start failed completely under the same conditions, often unable to finish the transcription at all. The researchers also tested the system on different languages, such as French, and different tasks, like translating speech from English to French, and it performed just as well. The system was able to handle medical consultations and diverse accents without needing to be retrained for each new situation.

The success of this approach lies in its flexibility. By deferring the decision of what to keep until the moment it is needed, the system avoids the trap of guessing wrong at the beginning. The "tidal" heads act like a tide, rising and falling to bring in the right information exactly when it is required and letting it go when it is no longer useful. This movement happens so quickly that it adds almost no delay to the process, making it practical for real-world use. The findings suggest that for machines to truly understand long conversations, they must be allowed to remember and forget dynamically, rather than being forced to make a permanent choice before the story has even begun. This shift from a static snapshot to a flowing memory could be the key to unlocking reliable, long-form speech recognition on everyday devices.

Technical Summary: WnW (Waxing-and-Waning KV Cache) for Long-Form Speech LLMs

1. Problem Statement

Long-form audio inputs have shifted the inference bottleneck in Speech Large Language Models (LLMs) from model parameters to the Key-Value (KV) cache. At typical audio token rates (12.5–25 tokens/sec), a ten-minute clip occupies 7,500–15,000 KV positions, with audio accounting for 70–80% of the total cache. Existing KV compression methods for text LLMs (e.g., H2O, SnapKV, Ada-KV) and their audio adaptations (e.g., AudioKV) rely on a prefill-only paradigm: they score positions once at the start of inference and permanently discard low-ranked positions.

The authors identify a critical structural flaw in applying this paradigm to long-form audio: the prefill-decode attention mismatch.

Prefill Attention: Concentrates heavily near the start of the audio (an "attention-sink" effect), often ignoring the bulk of the input.
Decode-Time Attention: Distributes broadly across the entire clip as the model generates text, reflecting the temporal progression of the transcription.
Consequence: The overlap between positions selected by prefill attention and those actually queried during decoding is weak (Jaccard overlap $\approx$ 0.187 for top-100 positions). Consequently, methods that permanently evict positions based on prefill scores fail to terminate generation or produce Word Error Rates (WER) exceeding 100% under tight memory budgets.

2. Methodology: WnW (Waxing-and-Waning KV Cache)

WnW addresses this mismatch by deferring retention decisions to decode time and introducing a mechanism to recall evicted positions from CPU memory. The method operates through two coordinated mechanisms: Offline Head Triage and Online Recallable Chunk Swap.

2.1 Offline Head Triage

The authors classify KV-heads into three functional roles based on an offline calibration using two metrics:

Voice Score (VS): Measures how strongly a head's attention is grounded in audio content (hit ratio of attended positions within the word-aligned time span of the answer token).
Head Sensitivity (HS): A gradient-based metric measuring the $\ell_2$ -norm of the answer-token loss gradient with respect to the head's audio key vectors.

Based on the product $VS \times HS$ , heads are assigned to:

Anchor Heads (Top 5): These are the most audio-grounded and loss-critical. They retain 100% of their audio KV on the GPU. They serve as a "decode-time importance observer," providing a reliable signal of which audio chunks are currently relevant.
Tidal Heads: These heads have high importance but are not critical enough to keep all KV on GPU. They maintain a portion of their KV on the GPU and offload the complement to CPU-resident memory. They dynamically swap chunks in and out based on the Anchor Heads' signals.
Fixed Heads: These heads exhibit stable attention patterns or low loss sensitivity. They retain only a static, prefill-time top- $k$ subset on the GPU; the rest is permanently discarded with no recall pathway.

2.2 Online Recallable Chunk Swap

During decoding, the system manages the "Waxing-and-Waning" of data between GPU and CPU:

Score Aggregation: At each decode step, the attention weights of the Anchor Heads are aggregated to generate a per-position importance score.
Chunk Selection: The audio positions are partitioned into overlapping chunks (e.g., 4-second windows). The top- $k$ chunks based on the aggregated anchor scores are identified.
Dynamic Swap:
- Recall: For Tidal Heads, segments belonging to the selected chunks but currently on the CPU are fetched into the GPU.
- Eviction: Segments present on the GPU but unselected for $\delta$ consecutive steps (e.g., 3 steps) are released back to the CPU (the CPU copy persists).
- This allows the system to correct early prefill mistakes once the true attention pattern emerges during generation.

3. Key Contributions

Quantification of the Mismatch: The paper provides empirical evidence that prefill attention is a poor proxy for decode-time attention in long-form audio, demonstrating that any compression scheme lacking a recovery pathway is structurally limited.
WnW Architecture: The design of a three-way head triage (Anchor/Tidal/Fixed) driven by audio-grounding and gradient sensitivity, combined with a decode-time recall signal aligned to the temporal structure of audio.
Performance and Efficiency: WnW achieves near-full-cache accuracy while keeping only 20% of audio tokens on the GPU, a regime where prefill-only baselines fail. It generalizes across languages, tasks, and domains with minimal CPU-GPU recall overhead.

4. Experimental Results

The method was evaluated on LibriSpeech-Long (English ASR), LongSpeech (Multilingual ASR and Translation), and PriMock57 (Medical domain), using two 3B parameter backbones: Voxtral-mini-3b and Qwen2.5-Omni-3B.

Accuracy: On LibriSpeech-Long at $r_{GPU} = 0.2$ (20% retention), WnW maintains a WER within $\sim$ 1.6 points of the Full Cache baseline. In contrast, prefill-only baselines (Ada-KV, AudioKV) fail to terminate generation, producing WERs >100%.
Generalization: WnW transfers effectively to French ASR, English-to-French translation, and medical dialogues, maintaining robust performance where baselines collapse.
Efficiency: The CPU-to-GPU recall overhead is negligible. With up to 195 tidal heads, the median decode time increases by less than 5%, and the data transfer volume remains low ( $\le$ 1.04 MB/step).
Ablation:
- Increasing the number of Tidal heads significantly improves performance under aggressive compression ( $r_{GPU}=0.1$ ).
- The combination of Voice Score and Head Sensitivity ( $VS \times HS$ ) is superior to using either metric alone.
- Per-segment prefill selection is critical to ensure uniform temporal coverage before decoding begins.

5. Significance and Claims

The paper claims that WnW is the first recallable, decode-driven KV management system specifically designed for audio LLMs. Its significance lies in:

Solving the Structural Limitation: By acknowledging that prefill attention cannot predict long-horizon decode needs in audio, WnW introduces a mechanism to recover discarded information dynamically.
Practical Deployment: It enables the deployment of long-form speech LLMs under realistic memory constraints (e.g., fitting a 10-minute clip on a single GPU) without sacrificing transcription quality.
Domain Adaptability: The method's reliance on temporal alignment and head-specific sensitivity rather than modality-agnostic query similarity allows it to generalize across different languages and tasks without retraining the model.

The authors note limitations, including the straightforward scheduling of CPU-GPU traffic (which could be optimized with CUDA streams) and the current focus on offline decoding rather than streaming recognition. However, they assert that the core mechanism is well-founded for architectures that maintain a length-dependent audio KV cache.