From Diet to Free Lunch: Estimating Auxiliary Signal Properties using Dynamic Pruning Masks in Speech Enhancement Networks

Technical Summary: Estimating Auxiliary Signal Properties Using Dynamic Pruning Masks in Speech Enhancement Networks

Problem Statement
Speech Enhancement (SE) systems in embedded audio devices (e.g., hearing aids) increasingly rely on deep learning. To ensure robustness and a seamless user experience, these systems often require auxiliary modules for tasks such as Voice Activity Detection (VAD), Signal-to-Noise Ratio (SNR) estimation, Acoustic Scene Classification, and Speech Quality Prediction (SQP). However, deploying separate deep learning models for these auxiliary tasks is computationally prohibitive for edge devices, while cloud-based inference introduces latency and privacy concerns. While Dynamic Channel Pruning (DynCP) has been proposed to reduce SE computation by adaptively disabling channels, it remains unclear whether the internal mechanisms driving this pruning contain sufficient information to perform these auxiliary tasks without dedicated models.

Methodology
The authors propose a framework to estimate auxiliary signal properties directly from the binary pruning masks generated by a DynCP-based SE model, specifically the Conv-FSENet architecture.

• Data Pipeline: The system treats training and testing data as continuous streams. Clean speech and noise excerpts are mixed to form noisy inputs ($x$), which are processed by the SE model to produce enhanced speech ($\hat{s}$). Ground truth labels for various tasks (VAD, gender, accent, noise category, SNR, SI-SDR, PESQ, F0) are extracted from metadata or computed using standard libraries (e.g., librosa, pyworld) and aligned with the Short-Time Fourier Transform (STFT) frame rate.
• Feature Extraction: The SE model generates a tensor of binary pruning masks ($G$) indicating which convolutional channels are active in each processing block. To reduce dimensionality and remove uninformative constant masks, the authors filter $G$ based on a minimum standard deviation threshold ($\tau$), resulting in a subset of dynamic features ($\tilde{G}$).
• Prediction Models: The authors employ simple, interpretable linear models (logistic regression for classification, linear regression for regression) to predict targets from $\tilde{G}$. The use of binary inputs allows predictions to be computed as weighted sums, minimizing computational overhead.
• Baselines & Comparisons: Performance is compared against baselines using raw STFT log-magnitudes and predicted suppression masks. The study also evaluates the impact of using features from specific network blocks, the top-$N$ most informative features, and raw (non-binarized) gating scores.
• Speaker Verification (SV): As an additional experiment, the authors aggregate pruning masks over voice-active frames to create utterance-level embeddings, which are then evaluated using a standard linear backend (WCCN and LDA) for speaker verification.

Key Contributions

1. Emergent Behavior Analysis: The paper investigates the latent representations learned by DynCP gating subnets, demonstrating that they implicitly encode relevant acoustic and speech characteristics despite being trained solely for computational efficiency.
2. Holistic Solution: The authors repurpose DynCP masks as a unified source of information for multiple auxiliary tasks (VAD, noise classification, SNR, quality metrics, etc.), eliminating the need for separate, compute-heavy predictor models.
3. Efficiency and Interpretability: By utilizing simple linear predictors on binary masks, the approach achieves high performance with negligible computational overhead. The linear coefficients provide a degree of interpretability regarding which parts of the network contribute to specific signal properties.

Results

• Classification: Simple predictors achieved up to 93% accuracy on VAD, 84% on noise classification, and 59% on noise classification (using top-64 features). Gender classification was successful, while accent classification performed near chance levels, likely because SE behavior is not heavily influenced by accent.
• Regression: The model achieved an $R^2$ of 0.86 on F0 estimation. For quality metrics, it achieved a Mean Absolute Error (MAE) of 0.2 for PESQ and 3.2 dB for SI-SDR.
• Feature Efficiency: Using only the top-64 most informative features retained performance comparable to using the full set of ~202 features, suggesting the auxiliary models can be highly compact.
• Raw Scores vs. Binary Masks: Raw gating scores (pre-binarization) generally outperformed binary masks, particularly for noise classification and F0 estimation, as they retain more internal state information. However, binary masks offered the lowest computational cost.
• Speaker Verification: The pruning masks yielded only modest Equal Error Rate (EER) results, performing worse than STFT baselines. The authors suggest this may be due to temporal pooling or the SE model learning speaker-agnostic representations.
• Computational Cost: The overhead for predicting 21 targets using the full feature set is approximately 0.6% to 0.93% of the total compute per frame, depending on active channels.

Significance and Limitations
The paper claims that DynCP models exhibit "emergent behavior" where gating subnets learn to allocate resources based on signal characteristics like noise levels and voice activity. This suggests that DynCP can serve as a dual-purpose mechanism: optimizing SE efficiency while simultaneously providing auxiliary data for robust system behavior and user feedback.

The authors remain modest regarding the maturity of these findings. They acknowledge several limitations:

• Linear models assume additive contributions and may struggle with correlated inputs.
• Feature collinearity complicates the interpretation of coefficient importance.
• Temporal smoothing within gating subnets may hinder the estimation of rapidly changing targets.
• Windowed ground truths (e.g., PESQ) may not fully represent instantaneous signal quality.

The work positions itself as a preliminary step toward single-model pipelines that can enhance speech, save compute, and generate auxiliary data simultaneously, encouraging further research into dynamic models and multi-task training in audio processing.