Efficiency-Performance Trade-offs in Neural Speaker Diarization via Structured Pruning and Low-Bit Quantization

Technical Summary: Efficiency-Performance Trade-offs in Neural Speaker Diarization

Problem Statement
Streaming speaker diarization is essential for time-critical applications such as emergency medical dispatch, where accurate speaker attribution supports downstream processing. However, deploying these systems on resource-constrained hardware presents significant challenges. Modern neural diarization pipelines, which often rely on scaling transformer encoders, increase parameter counts and inference-time memory, creating a conflict between accuracy and efficiency. Furthermore, the lack of public, domain-specific datasets (due to privacy concerns regarding protected health information) limits systematic evidence regarding diarization behavior under realistic deployment constraints. Existing literature often treats latency and runtime budgets as post-hoc constraints rather than explicit experimental factors, and there is a gap in end-to-end characterizations of how compression techniques (pruning and quantization) affect diarization accuracy within fixed, reproducible medical conversational pipelines.

Methodology
The study utilizes SIMSAMU, a public corpus of simulated French medical-dispatch phone dialogues annotated for diarization and transcription. The authors employ a fixed pipeline based on the SIMSAMU reference implementation, consisting of a PyanNet speaker-segmentation model (initialized from pyannote/segmentation-3.0 with a SincNet front-end and BiLSTM layers), a pretrained speaker-embedding extractor (pyannote/wespeaker-voxceleb-resnet34-LM), and agglomerative clustering.

The experimental approach involves three main components:

1. Latency Characterization: The authors sweep streaming parameters to quantify latency-accuracy trade-offs. They vary the right-context (look-ahead, $R$) and chunk length ($C$) using log-spaced values while measuring the Diarization Error Rate (DER) and Real-Time Factor (RTF).
2. Structured Pruning: Iterative, magnitude-based structured pruning is applied exclusively to the segmentation model parameters. Two structures are tested:
   • Hidden-unit pruning: Removing BiLSTM hidden units (grouped across gates and directions).
   • Linear-channel pruning: Removing neurons/channels in the post-LSTM linear stack.
     Pruning is performed gradually over iterations with recovery fine-tuning on a training subset to maintain stability.
3. Low-Bit Quantization: The best-performing pruned model (Linear–40) is subjected to various precision modes: FP32 (baseline), FP16, INT8 (Post-Training Quantization [PTQ] and Quantization-Aware Training [QAT]), and INT4 (PTQ and QAT). Activation bit-widths are kept at 8-bit for INT4 settings to mitigate distribution shifts.

Key Contributions
The paper makes three primary contributions to the field of efficient speech processing:

1. Latency-Performance Characterization: It demonstrates that additional buffering (increasing right-context) is not consistently beneficial for diarization accuracy in this domain. Conversely, very low-latency operating points (small chunk lengths) can substantially degrade performance.
2. Pruning Structure Impact: The study reveals that the choice of pruning structure significantly influences the accuracy-efficiency trade-off. Hidden-unit pruning drastically reduces model size but causes sharp accuracy degradation, whereas linear-channel pruning preserves accuracy better at low-to-moderate levels with only modest size reductions.
3. Compression Trade-offs: The authors identify a specific operating point where applying FP16 quantization to a linear-channel pruned model (Linear–40) reduces the model size by half (from ~5.55 MB to ~2.78 MB) with essentially unchanged RTF, at the cost of a ~40% relative increase in DER compared to the baseline.

Results

• Latency: Performance remains stable for small-to-moderate right-context values ($R$) but degrades sharply as $R$ increases beyond ~1 second, likely due to blurring speaker-change boundaries in dispatch calls. Similarly, chunk lengths below 0.1 seconds cause significant error increases, while lengths above 0.1 seconds reach a stable plateau.
• Pruning:
  • Hidden-unit pruning: Reduces parameters by up to ~77% (from 1.47M to 0.33M) and size by ~78%, but DER increases from 10.70% to 50.47% at 80% pruning.
  • Linear-channel pruning: Reduces size by only ~2% even at 80% pruning, but DER increases more gradually. The Linear–40 variant (40% pruning) achieves the best balance among pruned models, with a DER of 12.66% (vs. 10.70% baseline) and minimal size reduction.
  • RTF: Structured pruning yields negligible gains in Real-Time Factor (RTF remains ~0.022–0.023 across all variants), suggesting that end-to-end throughput is dominated by non-pruned pipeline stages (e.g., feature extraction, I/O, clustering) and kernel overheads.
• Quantization:
  • FP16: Achieves the lowest DER among quantized models (15.02%) and reduces model size by 50% compared to the pruned FP32 baseline.
  • INT8/INT4: INT8 QAT outperforms INT8 PTQ (19.02% vs. 23.67% DER), but both suffer significant accuracy loss compared to FP16. INT4 variants result in severe degradation (DER > 44%).
  • Stability: FP16 shows the tightest interquartile range in per-recording error changes, while INT4 exhibits systematic accuracy loss across the distribution.

Significance and Claims
The paper positions itself as a first step toward making efficiency claims in speaker diarization more interpretable and comparable. The authors argue that configuration choices (chunk length, look-ahead, precision) significantly influence accuracy, meaning systems with similar high-level architectures may not be directly comparable without reporting these deployment settings.

A key finding is that model compression does not automatically translate to end-to-end throughput gains in a complete diarization stack. While pruning and quantization reduce storage footprint, they do not necessarily improve RTF if the runtime is bottlenecked by other pipeline components. Consequently, the authors advocate for evaluating efficiency at the pipeline level rather than solely at the checkpoint level. They call for the community to explicitly report deployment settings (latency budgets, precision, chunking) to facilitate meaningful comparisons in resource-constrained inference scenarios.

The study acknowledges limitations, noting that SIMSAMU is a specialized, small-scale corpus and that results may not generalize to other languages, interaction conditions (e.g., high overlap), or hardware environments. Future work is suggested to include diverse interaction conditions and joint optimization of latency parameters with compression techniques.