💬 NLP

An Exploration of Mamba for Speech Self-Supervised Models

This paper explores Mamba-based HuBERT models as efficient alternatives to Transformer architectures for speech self-supervised learning, demonstrating their superior performance in long-context and streaming ASR tasks, competitive results on probing benchmarks, and enhanced ability to capture speaker features and quantized representations.

Original authors: Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tzu-Quan Lin, Heng-Cheng Kuo, Tzu-Chieh Wei, Hsi-Chun Cheng, Chun Wei Chen, Hsien-Fu Hsiao, Yu Tsao, Hung-yi Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand human speech. For years, the gold standard for doing this has been a type of AI architecture called the Transformer. Think of a Transformer like a super-organized librarian who, to understand a sentence, reads the entire book from cover to cover every single time a new word is mentioned. This works incredibly well for accuracy, but it gets slow and expensive very quickly. If the book is short, it's fine. But if the book is a whole novel (a long speech), the librarian gets overwhelmed, runs out of desk space (memory), and the process grinds to a halt.

Enter Mamba, the new contender. If the Transformer is the librarian reading the whole book, Mamba is a skilled storyteller who remembers the story as they go. They don't need to re-read the whole novel to understand the current sentence; they just keep a running mental note of what happened before. This makes them incredibly fast and efficient, especially for long stories.

This paper, titled "An Exploration of Mamba for Speech Self-Supervised Models," asks a big question: Can we replace the "librarian" (Transformer) with the "storyteller" (Mamba) for teaching computers to speak?

Here is what the researchers found, broken down into simple concepts:

1. The Long-Story Test (Long-Context ASR)

The Problem: When you try to transcribe a 5-minute speech or a whole lecture using the old Transformer method, the computer's memory explodes. It's like trying to hold a 500-page book in your hands while trying to read one page; your arms get tired, and you drop the book.
The Mamba Solution: The researchers built a "Mamba-based HuBERT" (a speech-learning model). Because Mamba is efficient, it could process entire documents without running out of memory.
The Result: The Mamba model successfully transcribed long speeches with fewer mistakes than the Transformer could even attempt. The Transformer literally crashed (ran out of memory) on the long documents, while Mamba kept going smoothly.

2. The Live Broadcast Test (Streaming ASR)

The Problem: Imagine a live news broadcast. The AI needs to type what the anchor is saying right now, without waiting for the sentence to finish. It can only use information from the past, not the future.
The Mamba Solution: Mamba is naturally built to look only at the past (it's "causal"). The researchers tested this in a streaming setting.
The Result: The Mamba model, which was actually smaller (had fewer "brain cells" or parameters), performed better than the larger Transformer model. It was faster and more accurate, proving it's perfect for real-time applications like live captioning.

3. The "Voice Print" and "Accent" Test (Representation Analysis)

The Problem: Before the AI can speak, it needs to understand the building blocks of sound (phonemes) and who is speaking (the speaker's identity).
The Mamba Solution: The researchers looked inside the AI's "brain" to see how well it learned these features.
The Result:

Phonemes: Mamba was better at grouping similar sounds together. It created a clearer "map" of speech sounds.
Speaker Identity: Mamba was surprisingly good at distinguishing who was talking. It seemed to hold onto the speaker's unique "voice print" more distinctly than the Transformer did.
Analogy: If the Transformer is a generalist who knows a little about everything, Mamba is a specialist who really understands the texture of the voice and the specific sounds being made.

4. The Catch: The "Two-Way Street" Problem

The Problem: Sometimes, to understand a sentence, you need to look at the words after the current word (bidirectional context). For example, in the sentence "The bank of the river," you need to know "river" to know "bank" isn't a financial institution.
The Result:

One-way (Causal): Mamba wins easily.
Two-way (Bidirectional): Mamba struggled a bit more. When the researchers tried to make Mamba look both forward and backward, it didn't perform as well as the Transformer, especially in larger models. It seems Mamba is still learning how to handle "looking back and looking forward" simultaneously as well as the old librarian can.

The Big Picture

This paper is a major step forward because it shows that Mamba is a viable, and often superior, alternative to Transformers for speech.

Why it matters: It means we can build speech AI that is cheaper to run, faster, and capable of understanding very long conversations without crashing.
The Future: While Mamba isn't perfect at everything yet (specifically the "two-way" looking), it opens the door for a new generation of speech models that are efficient enough to run on your phone or in real-time applications, rather than just in massive data centers.

In short: The researchers took a new, efficient engine (Mamba) and put it into a speech-learning car. They found that while the old engine (Transformer) is great for short sprints, the new engine is a champion for long-distance marathons and live races, even if it's still learning how to drive in reverse.

1. Problem Statement

While Mamba (a Selective State Space Model) has demonstrated superior efficiency and performance in language modeling compared to Transformers, its application in Speech Self-Supervised Learning (SSL) remains underexplored.

Limitations of Transformers: Transformer-based SSL models (e.g., HuBERT) suffer from quadratic computational complexity ( $O(N^2)$ ) relative to sequence length. This leads to high deployment costs, memory bottlenecks (Out-of-Memory errors) on long sequences, and inefficiency in real-time streaming applications.
Gap in Research: Prior studies on Mamba in speech have been limited to isolated downstream tasks (often lagging behind Transformers) or hybrid designs that sacrifice Mamba's linear-time scaling. There is no unified, comprehensive evaluation of Mamba as a foundational speech model for pre-training and feature extraction.

2. Methodology

The authors propose Mamba-based HuBERT, a systematic exploration of replacing Transformer blocks in the HuBERT architecture with Mamba blocks while retaining the standard feature extraction modules.

Architecture:
- Input: Standard 7-layer CNN feature encoder (25ms receptive field) + Convolutional positional encoder.
- Backbone: Replaces Transformer layers with Mamba layers.
- Variants:
  - Causal: Uses standard Mamba (unidirectional).
  - Bidirectional: Implements ExtBiMamba (two independent Mamba branches with distinct projections) and InnBiMamba (shared projections, duplicated SSM modules). ExtBiMamba is used as the default.
- Training: Follows the HuBERT two-stage pre-training pipeline:
  1. Iteration 1: MFCC targets (250k steps).
  2. Iteration 2: K-means pseudo-labels from the 6th layer of Iteration 1 (400k steps).
- Hardware: Trained on a single NVIDIA V100 GPU (smaller batch size than original HuBERT to ensure reproducibility and cost-efficiency).
Evaluation Strategy:
- Efficiency: Measured MACs/second and Real-Time Factor (RTF) across varying sequence lengths (5s to 320s).
- Fine-tuning: Tested on Long-Context ASR (TEDLIUM3, full documents) and Streaming/Causal ASR (LibriSpeech).
- Representation Analysis: Used Phone Purity (k-means clustering quality) and Canonical Correlation Analysis (CCA) to measure similarity to phoneme, speaker, and emotion labels.
- Downstream Probing: Evaluated on the SUPERB benchmark (Phoneme Recognition, Speaker ID, Emotion Recognition, Intent Classification).

3. Key Contributions

Systematic Exploration: The first comprehensive study of Mamba-based HuBERT as both a speech foundation model and a feature extractor, covering pre-training, fine-tuning, and probing.
Efficiency Validation: Demonstrated that Mamba-based models maintain linear computational complexity, enabling processing of long sequences (up to 5 minutes) where Transformers fail due to memory limits.
Superior Causal Performance: Showed that Mamba's inherent causal architecture is naturally suited for streaming ASR, outperforming larger Causal Transformer models with fewer parameters.
Representation Quality: Found that Mamba-based models produce higher-quality quantized representations (higher phone purity) and capture speaker-related features more distinctly than Transformers.
Scalability Insights: Identified a trade-off: Mamba excels in causal settings and small-scale bidirectional models but faces scalability challenges in large-scale bidirectional settings compared to Transformers.

4. Key Results

A. Computational Efficiency

Linear vs. Quadratic: Mamba-based HuBERT maintains nearly constant MACs and RTF as sequence length increases. In contrast, Causal Transformer MACs and RTF grow sharply.
Memory Limits: Causal Transformers encountered Out-of-Memory (OOM) errors beyond 80 seconds of audio. Mamba models successfully processed sequences up to 320 seconds (5 minutes).

B. Fine-tuning Performance

Long-Context ASR (TEDLIUM3):
- ExtBiMamba achieved a Word Error Rate (WER) of 11.08% on full documents.
- Transformer Base failed to run on full documents (OOM).
- Note: At the utterance level, Transformer (11.86%) slightly outperformed ExtBiMamba (13.37%), but the document-level context advantage of Mamba was decisive.
Streaming/Causal ASR (LibriSpeech):
- Mamba Base (78.2M params) achieved 15.77% WER.
- Causal Transformer Base (94.7M params) achieved 16.66% WER.
- Mamba achieved better accuracy with ~17% fewer parameters.

C. Representation Analysis

Phone Purity: Mamba models showed higher peak phone purity in causal settings and comparable/slightly better purity in bidirectional settings, indicating better phonetic encoding.
Speaker/Emotion Features: Mamba models maintained higher CCA similarity with speaker embeddings across layers, suggesting they capture speaker identity more distinctly.

D. SUPERB Probing Benchmarks

Causal Setting: Mamba-based models outperformed Causal Transformers across almost all metrics (PR, SID, ER, IC), achieving the highest overall SUPERB scores.
Bidirectional Setting:
- Small Scale: Mamba (ExtBiMamba) outperformed Transformer in most tasks (PR, SID, ER).
- Base Scale: Transformer outperformed ExtBiMamba. However, an ablation study revealed that InnBiMamba (a different bidirectional design) significantly outperformed ExtBiMamba at the Base scale (PR 9.62% vs 10.65%), suggesting the architecture design is critical for scaling.

5. Significance and Conclusion

This paper establishes Mamba-based HuBERT as a promising alternative to Transformers for speech processing, particularly for applications requiring:

Long-context modeling: Handling hours of audio without memory bottlenecks.
Real-time/Streaming ASR: Achieving state-of-the-art performance with lower latency and parameter counts.
Speech Unit Extraction: Generating high-quality discrete units for spoken language modeling.

Limitations & Future Work:
The study notes that Mamba's scalability in bidirectional settings is currently limited compared to Transformers, likely due to training instability in large-scale configurations. Future work should explore scaling laws on larger cluster resources and refine bidirectional Mamba architectures (e.g., InnBiMamba) to close this gap.

Code Availability:
The codebase is publicly available at: https://github.com/hckuo145/Mamba-based-HuBERT.