Fast Speech Foundation Model Distillation Using Interleaved Stacking

Technical Summary: Fast Speech Foundation Model Distillation Using Interleaved Stacking

Problem Statement
Knowledge distillation (KD) is a standard technique for deploying efficient student models from large Speech Foundation Models (SFMs), such as HuBERT, to reduce inference latency and computational costs. While existing approaches often employ "deep-and-narrow" student architectures to maintain performance under memory constraints, these models suffer from slow training speeds compared to "shallow-and-wide" architectures due to limited parallelization. Although stagewise training via model stacking (progressively increasing model depth) has been explored to accelerate training in other domains, its application to SFM distillation remains uninvestigated. Furthermore, existing stacking methods (e.g., gradual stacking, MIDAS) often cause performance degradation in SFMs because they disrupt the consistency of layer positions across training stages. Since SFMs encode distinct, layer-specific knowledge, relocating layers during the stacking process hinders the effective transfer of this knowledge.

Methodology
The authors propose Interleaved Stacking, a novel training acceleration framework designed specifically for SFM distillation. The method operates through the following mechanisms:

1. Stagewise Training with Interleaving: The training process is divided into $B$ stages. Initially, a shallow student model with $K$ layers ($K = N/B$, where $N$ is the target depth) is trained. At the end of each stage, instead of appending copied layers to the top or middle of the network, the method selects every $b$-th layer from the current $bK$-layer model, copies it, and inserts the copy immediately after its original layer. This results in a $(b+1)K$-layer model for the next stage.
2. Layer Position Consistency: Unlike gradual stacking (which copies the last $K$ layers) or MIDAS (which copies intermediate blocks), interleaved stacking ensures that layers retain their relative proportional positions (early layers remain early, late layers remain late) throughout the training stages.
3. Integration with Intermediate-Level KD: The method naturally supports intermediate-level Knowledge Distillation losses. Because layer positions remain consistent, the student's $k$-th layer can be consistently aligned with specific teacher layers (e.g., every $M/K$-th teacher layer) across all stages. This allows for the use of layer-to-layer Mean Squared Error (MSE) losses without the training instability observed in other stacking methods where layer indices shift.
4. Loss Function: The total loss combines output-level KD loss (MSE between teacher and projected student outputs) and intermediate-level KD loss. The intermediate loss targets fixed teacher layer indices relative to the student's current depth, ensuring stable supervision.

Key Contributions

• First Investigation of Stacking for SFM Distillation: The paper is the first to explore the effectiveness of stacking-based training acceleration specifically for distilling Speech Foundation Models.
• Interleaved Stacking Algorithm: The authors introduce a stacking strategy that preserves layer position consistency, addressing the performance degradation caused by layer misalignment in existing methods.
• Compatibility with Intermediate Supervision: The proposed method enables the seamless integration of intermediate-level KD losses, which are critical for SFM performance, whereas existing stacking baselines struggle to incorporate them without causing training divergence.
• Empirical Validation: The method is validated on the SUPERB benchmark across four downstream tasks: Phoneme Recognition (PR), Automatic Speech Recognition (ASR), Slot Filling (SF), and Speaker Identification (SID).

Experimental Results
The authors evaluated the method using a 12-layer Transformer student model distilled from a HuBERT base teacher.

• Performance vs. Baselines: Interleaved stacking significantly outperformed existing stacking baselines (Gradual Stacking and MIDAS) across all tasks and scheduling strategies (Equal and Prop-1). For instance, on the PR task with equal scheduling, Interleaved Stacking achieved a Perplexity Error Rate (PER) of 9.08, compared to 11.50 for Gradual Stacking and 10.75 for MIDAS.
• Performance vs. Full Training: Remarkably, under the Prop-1 scheduling strategy, the proposed method achieved performance comparable to or better than a full-depth model trained without stacking (Full L2L) on PR, SF, and SID tasks, despite requiring reduced training time (approx. 1.16x speedup).
• Impact of Intermediate Loss: The inclusion of intermediate-level KD losses significantly boosted performance. The method maintained superiority over baselines even without intermediate losses, but the gains were most pronounced with them. In contrast, applying intermediate losses to Gradual Stacking caused training instability and performance drops.
• Layer Similarity Analysis: Analysis of cross-layer similarity confirmed that adjacent layers in SFMs exhibit high correlation. The interleaved approach preserved these relationships, resulting in a "block-like" similarity structure where copied layers and their originals retained high similarity, validating the design choice of adjacent insertion.

Significance and Claims
The paper claims that Interleaved Stacking offers a practical solution to the trade-off between training efficiency and model performance in SFM distillation. By maintaining consistent layer positions, the method allows for the preservation of layer-specific knowledge inherent in SFMs, which is often lost in other acceleration techniques. The authors assert that this approach not only accelerates the deployment of efficient student models by reducing training costs but also mitigates the performance degradation typically associated with stagewise training. The work highlights that training efficiency is a critical, yet underexplored, factor in real-world SFM deployment and demonstrates that careful architectural manipulation during training can yield models that are both fast to train and highly effective.