Technical Summary: MURMUR – An Efficient Inference System for Long-Form ASR

Problem Statement

Long-form automatic speech recognition (ASR) faces a fundamental trade-off between accuracy and latency. Existing approaches fall into two categories, each with significant limitations:

1. Chunk-based Pipelines (e.g., WhisperX): These systems process audio in fixed, short windows (e.g., 30 seconds) to enable parallel inference and low latency. However, they suffer from context fragmentation, leading to errors in speaker attribution and timestamp generation at chunk boundaries. They rely on brittle heuristics and external modules (like separate diarization) to stitch segments, often resulting in degraded performance on metrics like time-constrained WER (tcpWER).
2. Long-Context Single-Pass Models (e.g., VibeVoice-ASR): These models process entire recordings in a single autoregressive pass, maintaining cross-chunk context for superior accuracy and native speaker/timestamp output. However, they are computationally expensive, suffering from high inference latency due to quadratic attention scaling and KV-cache management costs. Furthermore, they are prone to catastrophic failures, such as repetition loops, which can cause the entire inference request to fail.

The core challenge is to achieve the high accuracy and robust speaker/timestamp alignment of single-pass models while retaining the low latency and robustness of chunk-based systems.

Methodology

The authors propose MURMUR, an inference system that operates at two granularities to overcome this trade-off:

1. Inter-Chunk Optimization: Tunable Chunk Size

Rather than adhering to fixed short windows or processing the entire recording at once, MURMUR treats chunk size as a tunable hyperparameter.

• Analysis: The authors analyzed the relationship between chunk size and performance on the AMI-IHM dataset. They found that accuracy plateaus beyond a moderate context length, while extremely long contexts can degrade performance in noisy conditions due to accumulated acoustic confusion.
• Strategy: MURMUR adopts an intermediate chunk size of 300 seconds. This size strikes a balance, matching or exceeding single-pass accuracy across various conditions while enabling batched parallel inference.
• Stitching: The system uses Voice Activity Detection (VAD) to identify speech boundaries, merges segments into 300s chunks, and processes them in parallel. Post-inference, it performs cross-chunk realignment. Crucially, instead of forcing external diarization labels, it treats the model's internal speaker assignments within chunks as ground truth and only uses diarization to reconcile speaker identities across chunk boundaries.

2. Intra-Chunk Optimization: Sliding Window KV Cache Eviction

To further reduce latency within the 300s chunks, MURMUR exploits attention sparsity through a custom KV cache eviction policy.

• Observation: Profiling revealed that speech tokens exhibit extreme attention sparsity. In VibeVoice-ASR, fewer than 25% of speech tokens account for 99% of the total attention weights across most layers. Additionally, attention patterns show a diagonal sparsity that shifts as decoding progresses.
• Policy: The system implements a sliding window eviction strategy applied separately to output tokens and speech tokens:
  • Output Tokens: Follows a StreamingLLM-style approach, retaining "sink" tokens and a sliding window of recent tokens.
  • Speech Tokens: Introduces a "delay-and-shift" policy. A fixed window of speech tokens is retained initially to allow the model to attend to early audio context. As decoding proceeds, the window shifts forward, evicting older speech tokens that fall outside the active attention band.
• Implementation: This requires custom KV cache management to handle the distinct eviction logic for pre-filled speech tokens versus generated text tokens.

Key Contributions

1. Systematic Analysis of Chunk Size: The paper provides a detailed characterization of chunk size as a hyperparameter for long-context ASR, demonstrating that intermediate sizes (300s) optimize the trade-off between transcription accuracy, speaker attribution, temporal alignment, and latency.
2. MURMUR Architecture: The design of an inference system that combines VAD-guided chunked parallel inference with intra-chunk KV cache eviction. This system produces end-to-end transcriptions with timestamps and speaker labels without relying on external post-processing modules for alignment.
3. Exploitation of Speech Token Sparsity: The identification and utilization of extreme attention sparsity in speech tokens within long-context ASR models, enabling a novel eviction policy that reduces inference costs with minimal accuracy degradation.

Results

Evaluated on the VibeVoice-ASR model across AMI-IHM, AMI-SDM, Tedlium3, and Earnings21 datasets:

• Accuracy vs. Latency: On the AMI-IHM dataset, MURMUR matches the single-pass accuracy (tcpWER of ~25.29% with speech token eviction vs. 25.68% baseline) while reducing latency by 4.2× (from 370.7s to 88.43s).
• Robustness: Unlike single-pass inference, which failed catastrophically (repetition loops) on 2 out of 14 AMI-IHM recordings, MURMUR contains failures to individual chunks, ensuring the entire request does not fail.
• Eviction Impact:
  • Output token eviction alone yielded a 3.85× speedup.
  • Speech token eviction alone yielded a 4.20× speedup.
  • Combining both yielded a 3.79× speedup (slightly lower due to management overhead).
  • The speech token eviction strategy resulted in less than 1% relative degradation in tcpWER compared to the no-eviction baseline.
• Comparison: MURMUR significantly outperforms chunk-based baselines like WhisperX in tcpWER (25.73% vs. 35.54%) while maintaining comparable latency.

Significance and Claims

The paper claims that speech recognition possesses a "sharp and well-defined optimal context length" (around 300 seconds), driven by the compounding effects of acoustic noise and speaker complexity. This contrasts with text LLMs, where longer contexts are often strictly beneficial.

The significance of MURMUR lies in demonstrating that single-pass inference is not necessary to achieve single-pass accuracy. By identifying the optimal operating point for chunk size and exploiting the specific attention sparsity patterns of speech tokens, MURMUR achieves the best of both worlds: the high fidelity of long-context models and the efficiency of chunked pipelines. The authors position this work as a step toward context-aware inference strategies for long-form speech, suggesting that future systems can generalize these findings as end-to-end long-context ASR models become more prevalent.

The paper modestly notes limitations, including its current applicability only to models that natively produce speaker-attributed transcriptions and its evaluation being restricted to English. It does not claim to solve all long-form ASR challenges but rather offers a specific, efficient inference architecture for the emerging class of long-context ASR models.