Tracking the emergence of linguistic structure in self-supervised models learning from speech

1. Problem Statement

Self-supervised speech models (S3Ms), such as Wav2Vec2 and HuBERT, have demonstrated remarkable success in learning representations of spoken language from unlabelled audio. While it is established that these models encode various linguistic structures (from phonemes to syntax), two critical questions remain unanswered:

1. When do different levels of linguistic structure emerge during the training process?
2. Where (across which model layers) are these structures encoded, and how does this vary with different pre-training objectives?

Existing literature often focuses on single structural levels or specific languages (mostly English), lacking a systematic, comparative analysis of the hierarchical organization and learning dynamics across multiple levels of abstraction (acoustic, sublexical, lexical, and syntactic) within the same training framework.

2. Methodology

Models and Training Setup

The authors trained six models on 831 hours of spoken Dutch (combining the Spoken Dutch Corpus, CommonVoice, and Multilingual LibriSpeech). The study compared three architectural/objective variations:

• Wav2Vec2: Uses a contrastive objective (predicting the correct latent representation among negative samples).
• HuBERT-I1: Uses a masked prediction objective with pseudo-labels derived from k-means clustering of acoustic features (MFCCs).
• HuBERT-I2: Uses a masked prediction objective with pseudo-labels derived from k-means clustering of hidden layer representations from HuBERT-I1 (iterative refinement).
• Baseline: A Wav2Vec2 model trained on non-speech acoustic scenes (AudioSet) to distinguish linguistic structuring from general acoustic pattern recognition.

The models were trained for up to 200,000 steps, with intermediate checkpoints saved to track learning trajectories.

Representational Probes

The authors developed a suite of nine probing techniques to analyze internal representations across 12 Transformer layers and 512-dimensional CNN outputs. The probes target different levels of abstraction and timescales:

1. Acoustic Alignment: Representational Similarity Analysis (RSA) against MFCCs.
2. Phonetic Structure: Clustering of phone categories (using Linear Discriminant Analysis - LDA).
3. Syllable Structure: Clustering of syllable forms and syllable types.
4. Lexical Structure: Clustering of word forms.
5. Homophone Disambiguation: An ABX task to distinguish words with identical pronunciation but different meanings.
6. Semantic Alignment: RSA against text-based FastText embeddings.
7. Part-of-Speech (POS): Clustering of POS categories.
8. Syntactic Structure: A structural probe (Hewitt & Manning, 2019) measuring the ability to reconstruct dependency trees (UUAS score).

Data

Analysis was performed on a held-out subset of Multilingual LibriSpeech (single speaker to control for speaker variability), with time-aligned annotations for phones, syllables, and words, and manually annotated sentences for syntactic analysis.

3. Key Contributions

• Comprehensive Hierarchy Mapping: The first study to simultaneously track the emergence of nine distinct linguistic levels (from acoustic to syntax) across model layers and training time in speech SSL models.
• Learning Dynamics Analysis: Systematic characterization of learning trajectories, revealing that different linguistic structures follow distinct temporal patterns.
• Impact of Pre-training Objectives: A comparative analysis showing how iterative pseudo-label refinement (HuBERT-I2) fundamentally alters layerwise organization and learning parallelism compared to contrastive (Wav2Vec2) and single-iteration predictive (HuBERT-I1) objectives.
• Cross-Layer vs. Temporal Emergence: Evidence that while acoustic features emerge early, higher-level structures (syntax) emerge later, but the order of emergence can be manipulated by the pre-training objective.

4. Key Results

Layerwise Organization (Where)

• Wav2Vec2 & HuBERT-I1: Exhibit a sequential pattern where acoustic and phonetic features peak in early layers, while lexical and syntactic information concentrates in middle layers (around Layer 7). Final layers show a drop in linguistic abstraction, specializing in the acoustic pre-training objective.
• HuBERT-I2: Shows a markedly different pattern. Due to iterative refinement, higher-level structures (semantics, syntax) peak in earlier layers than lower-level features (phonemes, syllables). Crucially, the final layers of HuBERT-I2 maintain high-level linguistic structures, unlike the other models.
• General Finding: All speech-trained models outperform the non-speech baseline in encoding linguistic structure, confirming that the structuring is a result of speech training, not just acoustic processing.

Learning Trajectories (When)

• Acoustic First: Acoustic alignment scores reach maximum performance first (within the first 10k steps).
• Sublexical to Lexical: Phone, syllable, and word form structures emerge next, generally outperforming the baseline by 10k–25k steps.
• Syntax Last: Syntactic structures (POS and dependency parsing) emerge significantly later (25k–50k steps), suggesting models must first learn to segment and identify units before learning their combinatorial relationships.
• Parallelism in HuBERT-I2: The learning curves for HuBERT-I2 show greater parallelism. Higher-order prediction tasks (iterative refinement) induce a state where multiple levels of structure emerge simultaneously rather than strictly sequentially.

Training Duration

• Most linguistic structures reach a performance ceiling around 100k steps.
• Extending training to 200k steps yields minimal gains for most probes, with the notable exception of syntactic structure, which continues to improve, suggesting that contextualization beyond the word level benefits from longer training.

5. Significance and Implications

• Cognitive Modeling: The findings support the hypothesis that speech processing involves a hierarchy of abstraction, but challenge the strict "bottom-up" sequential acquisition view. The ability of HuBERT-I2 to learn higher-level structures earlier suggests that iterative refinement of prediction targets can accelerate the acquisition of abstract linguistic knowledge.
• Model Architecture Design: The study highlights that the choice of pre-training objective (contrastive vs. predictive vs. iterative predictive) drastically changes the internal representational geometry. This has implications for selecting models for downstream tasks; for instance, HuBERT-I2's final layers are more linguistically specialized but may lose some acoustic fidelity.
• Linguistic Theory: The results provide empirical evidence that purely distributional learning on continuous signals is sufficient to induce complex syntactic and semantic structures, supporting the viability of S3Ms as models for human language acquisition.
• Future Directions: The paper suggests that causal experiments (e.g., suppressing word-form encoding) are needed to confirm the causal link between unit identification and syntactic learning.

In summary, this paper provides a granular "X-ray" of how self-supervised speech models build linguistic understanding, demonstrating that the timescale of information integration and the abstraction level of the pre-training objective are the primary drivers of how and when linguistic structure emerges.