💬 NLP

Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition

The paper proposes SAP $^{2}$ , a novel framework utilizing a Speech-Driven Attention-based Pooling mechanism to dynamically prune and integrate relevant contextual keywords, thereby achieving state-of-the-art performance and significantly reducing keyword error rates in long-context Automatic Speech Recognition scenarios.

Original authors: Yiming Rong, Yixin Zhang, Ziyi Wang, Deyang Jiang, Yunlong Zhao, Haoran Wu, Shiyu Zhou, Bo Xu

Published 2026-01-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiming Rong, Yixin Zhang, Ziyi Wang, Deyang Jiang, Yunlong Zhao, Haoran Wu, Shiyu Zhou, Bo Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Noisy Library"

Imagine you are trying to listen to a friend tell a story in a library. Usually, this is easy. But now, imagine the library has suddenly been filled with thousands of books (context) that might contain the names or words your friend is about to say.

The problem is that your friend only mentions a few specific words from those thousands of books. If you try to read every single page of every book while listening to your friend, two things happen:

You get overwhelmed: Your brain (the computer model) gets too full and slows down.
You get distracted: You start hearing words from the books that aren't actually being spoken, leading to mistakes.

This is the challenge for Automatic Speech Recognition (ASR) systems when they try to understand complex scenarios, like a conference presentation where the speaker is using slides full of text. The system has too much text to process and doesn't know which parts are actually important for what is being said right now.

The Solution: SAP2 (The Smart Librarian)

The authors propose a new method called SAP2. Think of SAP2 as a super-smart librarian who helps the listener. Instead of handing the listener a whole stack of books, the librarian does two things:

The "Pruning" Stage (Cleaning the Shelf):
The librarian looks at the speech (what is being said) and the massive list of keywords from the slides (the books). They quickly scan the list and throw away 99% of the words that don't matter. They only keep the few specific words that are actually relevant to the current sentence.
- Analogy: If the speaker says, "I am talking about glaucoma," the librarian throws away thousands of words about "finance" or "weather" and keeps only "glaucoma."
The "Integration" Stage (The Handoff):
Once the librarian has the short, clean list of relevant words, they hand it to the listener. The listener then uses this short, focused list to help them understand the speech perfectly.

How It Works: The "Speech-Driven Attention"

The paper introduces a special trick called Speech-Driven Attention-based Pooling.

Imagine you are trying to summarize a long movie script into a one-page cheat sheet.

Old way: You just cut the script into small chunks and paste them together. This is messy and loses the flow.
SAP2 way: You listen to the audio while reading the script. You only highlight the words in the script that match the sounds you hear. Then, you compress those highlighted words into a tiny, dense summary.

This allows the computer to handle huge amounts of text (like a whole presentation deck) without getting confused or running out of memory, because it only keeps the "speech-salient" (important to the sound) information.

The Results: Getting It Right

The researchers tested this on two main datasets:

SlideSpeech: Recordings of conference talks with slides.
LibriSpeech: General audiobook recordings.

What they found:

Better Accuracy: SAP2 made fewer mistakes than previous top methods. On the SlideSpeech dataset, it reduced errors by about 30% compared to the previous best method.
Handling the "Noise": Even when they fed the system text from 5 slides instead of just 1 (making the "library" 5 times bigger), SAP2 didn't get confused. In fact, it got slightly better at finding the right words because it had more clues to choose from, whereas other systems got worse.
Speed: It didn't take much longer to run than the older methods, proving that this "cleaning" process is efficient.

The Bottom Line

The paper claims that by acting like a smart filter that prunes (cuts out) irrelevant text and integrates (combines) only the relevant parts based on what is actually being spoken, we can make speech recognition systems much better at understanding complex, real-world situations like lectures and presentations.

Key Takeaway: It's not about giving the computer more information; it's about giving it the right information at the right time, filtered by what it hears.

Technical Summary: Speech-Aware Long Context Pruning and Integration for Contextualized Automatic Speech Recognition

Problem Statement

Automatic Speech Recognition (ASR) systems have achieved near-human accuracy in routine scenarios but struggle in contextualized environments requiring domain-specific knowledge, such as conference presentations with accompanying slides. Two primary challenges hinder performance in these settings:

Constrained Context Windows: Large-scale pre-trained models and SpeechLLMs have limited context windows, making it difficult to process extensive textual inputs (e.g., hundreds of thousands of words from multiple slides).
Information Sparsity and Noise: In long-form contexts, relevant keywords are often sparse amidst significant irrelevant information. Simply concatenating all OCR-derived text as prompts introduces computational overhead and risks diluting the model's focus with noise, leading to recognition errors in critical domain terms.

Existing approaches, such as deep biasing E2E models, often suffer from architectural rigidity and performance gaps compared to large pre-trained models. Conversely, SpeechLLM-based approaches that treat all contextual keywords as prompts face challenges in effectively filtering noise and processing long contexts without truncation or semantic loss.

Methodology: The SAP2 Framework

The authors propose SAP2 (Speech-Aware Context Pruning with Speech-Driven Attention-based Pooling), a novel framework designed to dynamically prune and integrate relevant contextual keywords. The method operates on the premise that human speech recognition actively integrates prior knowledge, and machines should similarly filter noisy context to align with speech content.

Core Components

Two-Stage Pruning-Integration (TPI) Pipeline:
- Stage 1 (Context Pruning): A SpeechLLM acts as a pruning model. It takes the full set of long-form contextual keywords ( $Z$ ) and the acoustic features ( $X$ ) as input. Using a Speech-Driven Attention-based Pooling mechanism, it dynamically selects a subset of core keywords ( $\tilde{Z}$ ) that are most relevant to the speech content.
- Stage 2 (Contextualized ASR): The pruned keywords ( $\tilde{Z}$ ) are concatenated with the speech features and fed into the ASR model for final transcription.
- Note: While the pruning and recognition models can be trained jointly, the authors find that a two-stage approach (SAP2-TPI) yields superior results by avoiding interference between the objectives of keyword selection and speech recognition.
Speech-Driven Attention-based Pooling:
- To address the issue of context lengths exceeding LLM window limits, this mechanism compresses extensive textual inputs.
- It calculates attention scores between speech embeddings and contextual keyword embeddings.
- Keywords are weighted by these scores, and a window-wise pooling operation is performed on the weighted text features. This reduces the context length while preserving speech-salient information, effectively combining the cross-attention benefits of deep biasing models with the flexibility of SpeechLLMs.

Key Contributions

Effective Long-Context Processing: Introduction of a Speech-Aware Context Pruning framework that instructs a SpeechLLM to dynamically filter OCR-derived textual contexts, retaining only keywords directly relevant to the speech.
Cross-Modal Context Compression: Development of a Speech-Driven Attention-based Pooling mechanism that leverages cross-modal alignment to compress extensive textual inputs into concise, speech-relevant context embeddings.
State-of-the-Art Performance: Demonstration of superior performance on SlideSpeech and LibriSpeech datasets, particularly in maintaining robustness under extensive contextual input conditions.

Experimental Results

The framework was evaluated on the SlideSpeech (conference slides) and LibriSpeech datasets.

SlideSpeech Performance:
- Achieved a Word Error Rate (WER) of 7.71% on the L95 subset (trained on 5-slide contexts), outperforming the previous SOTA (MaLa-ASR) by a relative reduction of 10.45%.
- Biased Keyword Error Rate (B-WER): Reduced by 41.1% compared to non-contextual baselines, indicating a significant improvement in recognizing domain-specific terms.
- Recall: Maintained a high contextual keyword recall of 95.59%.
- Scalability: The model demonstrated robust performance even when the context was expanded from 1 slide to 5 slides, and further generalized to unseen context lengths (up to 25 slides) without retraining, showing only marginal WER degradation.
LibriSpeech Performance:
- Achieved a WER of 1.12% on the test-clean subset with a biasing list of 100 words, surpassing previous SOTA methods.
- Showed strong robustness to increasing biasing list lengths (up to 1000 words), with minimal performance degradation.
Ablation Studies:
- Pruning Strategy: The two-stage TPI approach significantly outperformed direct prompt concatenation (PC) and joint pruning-integration (JPI) models, particularly in B-WER and keyword recall.
- Pooling Mechanism: The Speech-Driven Attention-based Pooling was critical for handling long contexts, improving performance on 5-slide scenarios compared to models without pooling.
- Efficiency: The two-stage pipeline did not introduce significant computational complexity; inference time remained comparable to baseline methods, with pruning time significantly reduced due to the compression mechanism.

Significance and Claims

The paper claims that SAP2 addresses the critical limitations of current contextualized ASR systems regarding context window capacity and noise resilience. By simulating the human brain's active integration of prior knowledge, the framework successfully filters noisy long-form contexts to extract salient information.

The authors emphasize that their method achieves state-of-the-art performance while maintaining robust scalability. Unlike previous methods that degrade when context length increases, SAP2-TPI consistently maintains performance under extensive contextual input conditions. The work suggests that dynamic, speech-aware pruning is a more effective strategy than static concatenation for integrating domain knowledge into ASR systems.

Limitations noted by the authors: The current implementation uses contexts split as single words. Future work is suggested to explore semantically meaningful phrasal contexts and multimodal fusion with visual embeddings.