💬 NLP

Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

This paper proposes "Abstract Compression," a method that replaces prior-turn audio with fixed latent tokens while retaining transcripts to efficiently leverage conversational context for improved LLM-based ASR, particularly in recognizing contextual entities, without the high computational cost of processing raw audio sequences.

Original authors: Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but you only have a single clue in front of you. You might guess the answer, but you'd likely miss the big picture. Now, imagine if you could remember everything that happened in the room before that clue appeared. Suddenly, the mystery becomes much easier to solve.

This is the core problem the paper addresses: How do we teach computers to "remember" a conversation so they can understand what is being said right now?

Here is a simple breakdown of their solution, "Abstract Compression," using everyday analogies.

1. The Problem: The Computer Has a Short Memory

Current speech-to-text systems (like the ones in your phone) are like people who have amnesia. They listen to one sentence, write it down, and then immediately forget everything that was said before.

Why this is bad: If someone says, "I'll meet you at the bank," the computer doesn't know if you mean a river bank or a money bank. But if the previous sentence was "Let's go deposit our savings," the computer would know it's a money bank.
The "Raw" Solution: The researchers tried giving the computer the entire history of the conversation (the audio of every previous sentence) to help it.
The Catch: Audio files are huge. Imagine trying to read a book where every previous page is printed out in giant, high-definition photos. It takes up too much space, slows everything down, and the computer gets overwhelmed. It's like trying to carry a library in your backpack just to read one page.

2. The Solution: "Abstract Compression"

The researchers realized they didn't need the entire library to understand the story. They needed a summary.

They invented a method called Abstract Compression. Think of it like this:

The Old Way (Raw Context): To understand the current sentence, the computer reads the entire audio recording of the last 10 conversations. It's heavy, slow, and expensive.
The New Way (Abstract Compression):
1. The Transcript (The "What"): They keep the text of what people said exactly as it is. This is like keeping the written notes of the conversation.
2. The Audio (The "How"): Instead of keeping the full audio recording (the voice, the tone, the background noise), they compress it into a tiny, magical "summary token."

The Analogy:
Imagine you are describing a movie to a friend.

Raw Audio: You play the entire 2-hour movie for them.
Abstract Compression: You give them the script (the text) and a 30-second highlight reel (the compressed audio) that captures the vibe and the voice of the actors, but not every single frame.

3. How They Trained the Computer

You can't just hand a computer a summary and expect it to understand it immediately. It needs to learn how to read these summaries.

The researchers used a Two-Step Training Process:

Step 1 (Learning to Summarize): They taught the computer how to turn a full audio clip into that tiny "summary token" so it could still understand the words. It's like teaching a student how to write a perfect book report from a full novel.
Step 2 (Learning to Use the Summary): Once the computer knew how to make the summaries, they taught it how to use those summaries to understand new conversations. They practiced with short conversations first, then slowly added more history, like a student learning to read longer and longer chapters.

4. The Results: Smarter and Faster

The results were impressive:

It works: The computer got much better at understanding tricky words (like names, places, or specific products) because it could "remember" the context.
It's efficient: By swapping the heavy audio files for tiny summary tokens, they saved a massive amount of computer memory and speed.
The Trade-off: It wasn't quite as perfect as listening to the full raw audio, but it was 90% as good while using only a fraction of the space.

The Bottom Line

This paper is about teaching AI to be a better listener. Instead of forcing the AI to carry a heavy backpack full of old audio recordings, they gave it a notebook of summaries.

This allows the AI to remember the "vibe" and the "who" of the conversation without getting bogged down by the heavy data, making voice assistants and meeting transcribers much smarter and faster.

1. Problem Statement

Automatic Speech Recognition (ASR) systems, particularly those based on Large Language Models (LLMs), typically process utterances in isolation. However, in conversational settings (e.g., voice assistants, meetings), accurate transcription often relies on conversational context from prior turns to resolve ambiguities, identify named entities, and establish speaker-specific pronunciations.

The core challenges identified are:

Ineffectiveness of Raw Context: Simply prepending raw audio and text from prior turns to the current prompt during inference degrades performance if the model hasn't been explicitly trained on multi-turn data.
Computational Cost: In LLM-based ASR, audio is represented as long sequences of tokens. Conditioning on raw multi-turn audio causes the prompt length (and KV cache size) to grow rapidly with conversation length, leading to high latency and memory bottlenecks.
The Trade-off: There is a need to determine if conversational context actually improves ASR and, if so, how to represent that context efficiently without the prohibitive cost of raw audio tokens.

2. Methodology

The authors propose a framework involving Supervised Fine-Tuning (SFT) and a novel compression technique called Abstract Compression.

A. Model Architecture

Backbone: The study uses PHI-4-MULTIMODAL, a multimodal LLM capable of processing interleaved audio, text, and images.
Input Representation: Audio waveforms are converted into a sequence of acoustic tokens ( $X$ ) via an audio encoder and projection module.

B. Training Strategy

The authors employ a two-stage training approach to handle the complexity of learning from compressed context:

Stage 1: Compressed-Audio-to-LLM Alignment
- Goal: Align the compressed audio representations with the LLM's input space.
- Process: The model is trained on single-turn ASR tasks where the raw audio tokens are replaced by a fixed number of latent tokens ( $K$ ).
- Optimization: The base LLM is frozen; only the compression module (learnable query matrices and cross-attention parameters) is optimized.
- Data: Uses DefinedAI (in-domain) and LibriSpeech (for broader acoustic coverage).
Stage 2: Contextual Fine-Tuning
- Goal: Teach the model to utilize compressed context for current-turn transcription.
- Process: The model is fine-tuned on multi-turn inputs where prior turns consist of compressed audio tokens ( $Z$ ) and explicit transcripts ( $Y$ ). The current turn remains at full-resolution audio.
- Curriculum Learning: Training starts with zero context turns and progressively increases the maximum number of available context turns (up to 10) to stabilize learning.

C. Abstract Compression Mechanism

Instead of feeding the entire sequence of raw audio tokens from prior turns, the method compresses them:

Audio Compression: A learnable cross-attention mechanism maps the variable-length raw audio tokens of a prior turn ( $X_i$ ) into a fixed number of latent tokens ( $Z_i$ ) of size $K$ .
Text Retention: The transcripts of prior turns ( $Y_i$ ) are kept in their original textual form (not compressed), as text is already compact compared to audio tokens.
Result: The prompt for the current turn includes the full-resolution audio of the current turn, but only a fixed-size "summary" (latent tokens) + full text for all prior turns.

3. Key Contributions

Empirical Validation of Context: The paper demonstrates that multimodal context from prior turns does improve LLM-based ASR, but only after supervised multi-turn training. Without SFT, raw context degrades performance. The primary gains are observed in the recognition of contextual entities (names, locations, products).
Abstract Compression: Introduction of a method that replaces prior-turn audio with a fixed-size latent representation while retaining explicit transcripts. This drastically reduces the token footprint of the context.
Two-Stage Training Strategy: A novel curriculum-based training pipeline that first aligns compressed audio to the LLM (Stage 1) and then fine-tunes for multi-turn context usage (Stage 2), overcoming the difficulty of training on latent representations directly.
Efficiency Analysis: Detailed analysis showing that the method recovers a significant portion of the performance gains of raw-context conditioning while maintaining a constant, small audio token budget regardless of conversation length.

4. Experimental Results

The experiments were conducted on DefinedAI (in-domain) and WoW (out-of-domain) datasets.

Baseline Performance:
- Raw context without SFT: Performance degradation.
- Raw context with SFT: Significant improvement, especially in Bias-WER (Word Error Rate on contextual entities). On WoW, Bias-WER dropped from 25.6% (single-turn) to 23.3% (raw multi-turn).
Abstract Compression Performance:
- The compressed model achieved 13.3% Bias-WER on DefinedAI and 24.5% on WoW.
- While slightly behind the raw-context model (13.1% and 23.3% respectively), the compressed model significantly outperformed the single-turn baseline (13.5% and 25.6%).
- Conclusion: The compressed representation retains the most critical information needed for entity recovery.
Compression Rate:
- The method achieves a high compression ratio for audio tokens. As the number of turns increases, the ratio of retained tokens decreases and stabilizes, making inference costs predictable.
Ablation Studies:
- Bottleneck Size ( $K$ ): Larger $K$ (e.g., 16 latent tokens) yields better performance than aggressive compression ( $K=4$ ).
- Stage 1 Data: Pre-training the compression module on a large dataset (LibriSpeech) improved single-turn reconstruction but did not necessarily translate to better multi-turn contextual ASR, suggesting task-specific alignment is more important than general acoustic compression.
- Context Length: Most performance gains saturate after 5 context turns, indicating that a moderate amount of compressed context is sufficient.

5. Significance

This work addresses a critical bottleneck in deploying LLM-based ASR for real-world conversational agents. By proving that compressed audio context is a viable alternative to raw audio, the paper offers a practical solution to the "context vs. cost" dilemma.

Scalability: It enables LLM-based ASR systems to handle long conversations without exploding memory usage or latency, as the context size grows linearly with a small constant factor rather than the variable length of raw audio.
Entity Recognition: It specifically targets the weakness of ASR systems in recognizing rare entities, showing that conversational context is the key to solving this, provided it is represented efficiently.
Future Direction: It establishes a new paradigm for "context-aware" multimodal modeling, suggesting that future systems should focus on distilling the essential information from history rather than storing raw history.

In summary, the paper demonstrates that Abstract Compression allows LLM-based ASR systems to leverage the benefits of conversational history for entity recognition while maintaining a fixed, low computational overhead.