On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models

Technical Summary: On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models

Problem Statement
Generative Spoken Language Modeling (GSLM) aims to train language models (LMs) on discrete speech representations without relying on textual transcriptions. While this paradigm enables applications for unwritten languages and infant language acquisition, current GSLM approaches face efficiency challenges. The conversion of speech signals to discrete unit sequences (speech-to-unit, or s2u) typically results in sequences significantly longer than text, leading to quadratic increases in computational costs for Transformer-based LMs. Furthermore, existing literature suggests that performance in spoken language understanding can be improved by reducing sequence length and increasing cluster size, but it remains unclear whether these findings hold for speech synthesis and continuation tasks, which are critical for generative applications.

Methodology
The authors investigate the impact of varying bitrates on GSLM performance by manipulating two key parameters in the s2u pipeline:

1. Segmentation Width ($N$): The continuous speech representations extracted from an SSL model (HuBERT-base) are segmented into fixed-length windows of $N$ milliseconds.
2. Cluster Size ($K$): These segments are quantized using K-means clustering with varying numbers of clusters ($K$), ranging from $2^7$ (128) to $2^{14}$ (16,384).

By combining eight values of $N$ (20ms to 280ms) and eight values of $K$, the study creates 64 distinct bitrate settings. The baseline is defined as $N=20$ (matching the original HuBERT frame shift) with standard cluster sizes.

The experimental framework evaluates two primary tasks:

• Speech Resynthesis: Reproducing input speech from discrete units.
• Speech Continuation: Generating semantically and acoustically coherent speech conditioned on an input segment.

The system employs a standard GSLM architecture: an s2u module, a Transformer-based autoregressive unit LM (uLM, based on OPT), and a unit-to-speech (u2s) synthesis module. Two distinct u2s models were tested to ensure robustness: Tacotron2 with a Parallel WaveGAN vocoder and VITS (an end-to-end model).

Evaluation Metrics
The study utilizes a multi-faceted evaluation strategy:

• Resynthesis: Word Error Rate (WER) for intelligibility, UTMOS for naturalness, and Mel-Cepstral Distortion (MCD) and LogF0 RMSE for acoustic feature reconstruction.
• Continuation: Perplexity (PPL) and diversity (VERT) calculated over a range of sampling temperatures. The authors note that the traditional "area under the curve" metric for PPL/VERT trade-offs is often invalid due to non-monotonic behavior; instead, they use Euclidean distance to an oracle point after min-max normalization.
• Advanced Evaluation: Pairwise judgments using an LLM-as-a-Judge (GPT-4.1-mini) and human evaluations using Meaningfulness Mean Opinion Score (MMOS).

Key Results

• Speech Resynthesis: While smaller segmentation widths ($N$) are generally preferred, moderately large values (e.g., $N=40$ or $80$) achieve performance comparable to the baseline ($N=20$) without substantial degradation in intelligibility or naturalness. This suggests that lower bitrate settings can preserve speech quality.
• Speech Continuation: Contrary to resynthesis, where the baseline often yields the best results, several settings with larger $N$ (lower bitrates) perform on par with or slightly better than the baseline in continuation tasks. Specifically, settings such as $(N=120, K=2^{13})$ for Tacotron2 and $(N=80, K=2^{12})$ for VITS demonstrated high performance.
• Model Dependency: Tacotron2 consistently outperformed VITS in intelligibility (WER), while VITS showed superior acoustic fidelity (MCD, UTMOS). However, the choice of u2s model had limited impact on the semantic content quality of the continuation, as evidenced by high Spearman rank correlations in PPL and VERT scores between the two models.
• Evaluation Metrics: LLM-based pairwise judgments showed a higher correlation with human subjective scores (MMOS) than conventional metrics (PPL, VERT). However, the correlation remained low overall, indicating that current automatic metrics are insufficient for reliably evaluating speech meaningfulness.

Significance and Claims
The paper claims that the conventional GSLM setting, which typically employs high-resolution discrete units (small $N$, large sequence length), may contain redundant information for effective speech generation. The authors demonstrate that intelligible and natural speech resynthesis, as well as high-quality speech continuation, can be achieved at substantially lower bitrates by increasing segmentation width and optimizing cluster size.

This finding suggests a path toward more efficient training and inference for GSLMs, potentially reducing computational costs without sacrificing output quality. Additionally, the study highlights a critical gap in evaluation methodology: while LLM-based metrics offer an improvement over traditional metrics, the field still lacks robust, high-correlation automatic evaluation methods for speech continuation, necessitating further research in this area.