💬 NLP

Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

The paper proposes Multimodal Depth Upscaling, a method that inserts and trains new transformer layers (such as E-Branchformer) within a frozen text LLM to adapt it for speech tasks, achieving competitive ASR performance while significantly preserving original text capabilities and reducing trainable parameters compared to full fine-tuning and LoRA.

Original authors: Kazuki Yano, Jun Suzuki, Shinji Watanabe

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kazuki Yano, Jun Suzuki, Shinji Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class Chef (the Text LLM) who has spent years mastering the art of cooking with fresh vegetables, spices, and meats (text data). This Chef can write recipes, explain flavors, and even tell jokes about food.

Now, you want this Chef to also learn how to cook with Sound (speech data). You want them to listen to a voice and write down what was said (Automatic Speech Recognition).

The Problem: The "Cramming" Disaster

Usually, when you try to teach a master chef a new skill, you might force them to stop cooking their old dishes and spend all their time practicing the new one.

Full Fine-Tuning: This is like making the Chef forget everything about vegetables to focus entirely on sound. They get great at listening, but they forget how to write recipes or tell jokes. They've lost their original magic.
LoRA (Low-Rank Adaptation): This is like giving the Chef a small, cheap notebook to write down new notes while they keep their old books. It helps a little, but the notebook is too small to hold all the complex rules of cooking with sound. The Chef still struggles to learn the new skill properly without messing up the old ones.

The Solution: "Multimodal Depth Up-Scaling"

The authors of this paper propose a clever new strategy called Multimodal Depth Up-Scaling. Think of it as building a specialized extension onto the Chef's kitchen without touching the original room.

Here is how it works, step-by-step:

1. The Frozen Core (The Original Kitchen)

The original Chef and their main kitchen (the pre-trained text layers) are frozen. They are locked in place. No one is allowed to change how they think or cook. This guarantees that the Chef never forgets their original skills.

2. The New Extension (The Added Layers)

Instead of changing the Chef, you build a new, specialized wing onto the kitchen.

You insert new "rooms" (Transformer layers) into the kitchen.
Only these new rooms are trained on the sound data.
The Chef walks through their old kitchen (unchanged), then steps into the new wing to process the sound, and then comes back out.

3. The Magic Trick: The "Pop-Up" Feature

This is the coolest part.

When the Chef needs to listen: They use the whole building, including the new wing. They are great at understanding speech.
When the Chef needs to write a recipe: You can literally fold up and remove the new wing. The Chef walks back into their original, untouched kitchen. Because the original kitchen was never touched, the Chef is 100% back to their original, perfect self. They haven't forgotten a single word of their original training.

The Secret Sauce: The "E-Branchformer"

The researchers also tried different designs for these new "rooms."

Standard Rooms: These are just like the old kitchen rooms. They work okay.
E-Branchformer Rooms: These are custom-built specifically for sound. Imagine a room with two parallel counters: one for long-range patterns (like the rhythm of a sentence) and one for local details (like the specific crackle of a voice).
The Result: Using these custom "E-Branchformer" rooms made the Chef even better at listening, while still keeping the original kitchen perfectly safe.

Why This Matters

In the past, trying to teach an AI to speak often made it "forget" how to read and write.

Old Way: Teach it to speak, and it becomes bad at writing.
This Paper's Way: Teach it to speak by adding a specialized module. It becomes great at speaking, and if you need it to write, you just turn off the module. It's like having a Swiss Army Knife where you can snap off the screwdriver to use the knife, and snap the screwdriver back on later without damaging the blade.

In short: This paper shows us how to give a text AI the ability to hear and understand speech without forcing it to forget who it is. It's a way to upgrade a model's "ears" without breaking its "brain."

1. Problem Statement

The paper addresses a critical challenge in adapting pre-trained Large Language Models (LLMs) to the speech domain: Catastrophic Forgetting.

Context: While Continual Pre-Training (CPT) on speech data allows text LLMs to learn speech tasks (like Automatic Speech Recognition, ASR), it often severely degrades the model's original text capabilities (e.g., reasoning, instruction following).
Limitations of Existing Solutions:
- Text Data Replay: Replaying text data during speech training preserves text skills but requires access to proprietary pre-training data (often unavailable) and significantly increases training costs.
- Parameter-Efficient Fine-Tuning (PEFT) / LoRA: While LoRA is widely used to reduce forgetting, the authors argue it lacks sufficient capacity to learn a new modality (speech) effectively, especially on smaller models.

2. Methodology: Multimodal Depth Up-Scaling

The authors propose Multimodal Depth Up-Scaling, a strategy that expands the model's capacity by inserting new layers rather than modifying existing ones.

Core Architecture

Frozen Base: The original pre-trained text LLM layers are kept frozen throughout training.
Layer Insertion: $m$ new transformer layers are inserted into the frozen $n$ -layer network.
Training Scope: Only the added layers and the speech token embeddings are trained on speech data.
Vocabulary Expansion: The tokenizer vocabulary is expanded to include discrete speech tokens (semantic and acoustic), and the embedding layer is resized accordingly.

Key Innovations

Function-Preserving Initialization: To ensure the model behaves identically to the pre-trained version at initialization (avoiding immediate perturbation), the added layers are initialized as identity functions.
- For standard Transformers: Output projection matrices of Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN) are zero-initialized.
- For E-Branchformer: A specific initialization strategy ( $W_{Merge} = [I; 0]$ ) is used to ensure the merge module outputs zero initially, preserving the identity function $Y=X$ .
Layer Dropping at Inference: A unique feature of this method is that the added layers can be completely removed during inference for text-only tasks. This guarantees zero degradation of the original text capabilities by design, as the model reverts exactly to the pre-trained state.
Architecture Variants: The authors test two architectures for the inserted layers:
1. Standard Transformer Blocks: Identical to the frozen base layers.
2. E-Branchformer: A speech-specific architecture featuring parallel branches (Global MHSA for long-range dependencies and Local Convolutional Spatial Gating Units for local acoustic patterns).

Placement Strategies

The paper investigates five strategies for inserting the new layers:

INTERLEAVED: Evenly distributed across all original layers.
BOTTOM, MIDDLE, TOP: Concentrated in specific network regions.
SANDWICH: Placed in both the bottom and top quarters.

3. Experimental Setup

Base Models: SmolLM2-360M (32 layers) and SmolLM2-1.7B (24 layers).
Dataset: 48,000 hours of English ASR data from the OWSM v3.2 suite.
Baselines: Full Fine-Tuning (updates all parameters) and LoRA (Low-Rank Adaptation).
Evaluation Metrics:
- ASR: Word Error Rate (WER) on LibriSpeech (test-clean, test-other).
- Text Capability: Average accuracy across 8 benchmarks (ARC-e/c, BoolQ, HellaSwag, etc.).
- Instruction Following: Zero-shot translation, simplification, and summarization tasks conditioned on speech input.

4. Key Results

ASR Performance

Depth Up-Scaling vs. Full Fine-Tuning: Depth up-scaling achieves ASR performance comparable to full fine-tuning. On the larger 1.7B model, it matches or even surpasses full fine-tuning.
Depth Up-Scaling vs. LoRA: Depth up-scaling significantly outperforms LoRA, particularly on the smaller 360M model, indicating LoRA lacks the capacity to learn the speech modality effectively.
Placement Impact: The INTERLEAVED strategy yielded the best WER results.

Text Capability Preservation

Degradation: Full fine-tuning and LoRA caused severe text degradation (e.g., -22.2% and -22.0% on the 360M model, respectively).
Depth Up-Scaling: Even with added layers active, text degradation was minimal (e.g., -1.9% for INTERLEAVED on 360M).
Zero Degradation Guarantee: When added layers are dropped at inference, text performance returns exactly to the pre-trained baseline ( $\Delta = 0.0$ ).
Instruction Following: Full fine-tuning completely lost the ability to follow text instructions when conditioned on speech (producing degenerate outputs). In contrast, the Depth Up-Scaling model successfully performed zero-shot translation, simplification, and summarization directly from speech inputs.

Architecture & Initialization

E-Branchformer: Using E-Branchformer as the inserted layers further improved WER compared to standard transformers, especially on the 1.7B model, while maintaining text preservation.
Initialization: The proposed function-preserving initialization ( $[I; 0]$ ) for E-Branchformer outperformed random initialization, confirming its necessity for stable training.

5. Key Contributions

Novel Adaptation Strategy: Introduced Multimodal Depth Up-Scaling, a method that adapts text LLMs to speech by inserting trainable layers into a frozen backbone, avoiding catastrophic forgetting without needing text data replay.
Inference Flexibility: Demonstrated a mechanism to drop added layers at inference, allowing a single model to switch between high-performance speech processing and pristine text generation.
Architecture Integration: Successfully integrated E-Branchformer (a speech-native architecture) as the inserted layers, showing that domain-specific architectures can be effectively "plugged in" to LLMs to boost modality-specific performance.
Empirical Validation: Proved that this approach achieves state-of-the-art ASR results while preserving text capabilities far better than Full Fine-Tuning or LoRA, even on smaller models (360M).

6. Significance

This work provides a robust solution for building Speech Language Models that do not sacrifice the linguistic intelligence of their text-based predecessors. By decoupling the learning of the new modality (speech) from the preservation of the old modality (text) through architectural expansion rather than weight modification, the authors offer a scalable path for multimodal LLM development. The ability to "drop" the speech layers makes this approach highly practical for deployment scenarios where a model needs to handle both speech and text tasks efficiently without maintaining separate model instances.