Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

1. Problem Statement

Self-supervised speech models (S3Ms) like wav2vec 2.0, HuBERT, and WavLM have achieved state-of-the-art performance by learning contextualized representations from unlabeled audio. While it is known that these models encode phonological features (e.g., voicing, nasality) in a compositional manner, the mechanism by which they integrate phonetic context (information from neighboring phones) into a single frame-level representation remains unclear.

Previous work established that S3Ms represent individual phones as linear combinations of phonological vectors. However, it was unknown whether a single frame representation encodes only the aligned phone or also the surrounding context (previous and next phones), and if so, how the model distinguishes between these sources of information without confusion.

2. Methodology

The authors propose a hypothesis that a single frame-level S3M representation encodes a sequence of phones (previous, current, and next) via position-dependent orthogonal subspaces. To test this, they formulated four predictions and designed experiments using three monolingual S3Ms (wav2vec 2.0, HuBERT, WavLM) on the TIMIT and VoxAngeles datasets.

Core Techniques:

• Phonological Analogies: The study uses vector arithmetic (e.g., $v_b - v_d + v_t \approx v_p$) to test if phonological relationships hold in the representation space. They compare mean pooling (averaging frames over a phone) vs. center pooling (using only the central frame) to determine if compositionality exists at the frame level.
• Contextual Probing: They extend the analogy test to neighboring phones ($p_{-1}, p_{+1}$) to see if a center frame encodes context.
• Orthogonality Analysis: They extract phonological vectors for specific relative positions (e.g., "voicing of the previous phone" vs. "voicing of the current phone") and measure the cosine similarity between them. The hypothesis predicts these vectors should be orthogonal ($\cos \approx 0$).
• Phonetic Boundary Detection: They analyze how the cosine similarity between frame representations and position-dependent vectors changes across annotated phonetic boundaries to see if the model implicitly segments speech.

3. Key Contributions & Predictions

The paper advances the understanding of S3M interpretability through four main findings:

1. Frame-level Compositionality: Phonological compositionality is not an artifact of temporal averaging (mean pooling). A single center frame representation is sufficient to support linear phonological analogies, confirming that compositionality exists at the frame level.
2. Contextual Encoding: A single frame-level representation encodes phonological information not just for the aligned phone, but also for its immediate neighbors ($p_{-1}$ and $p_{+1}$). This is evidenced by successful phonological analogies for neighboring phones in later transformer layers.
3. Position-Dependent Orthogonality: To avoid ambiguity when encoding multiple phones, the model utilizes orthogonal subspaces. Phonological vectors associated with different relative positions (e.g., "current voicing" vs. "previous voicing") are approximately orthogonal. This allows the model to disentangle information from different temporal positions within the same vector space.
4. Emergent Phonetic Boundaries: The position-dependent subspaces are not fixed to a symmetric time window. Instead, the model implicitly tracks phonetic boundaries. The transition of which subspace is active (e.g., shifting from "next phone" to "current phone" vectors) aligns precisely with annotated phonetic boundaries in the audio.

4. Results

• Compositionality: Center-pooled representations achieved success rates comparable to or higher than mean-pooled representations in phonological analogy tasks, outperforming spectral baselines (MFCC/MelSpec).
• Context Window: S3M representations (particularly in later layers) successfully encoded phonological features of $p_{-1}$ and $p_{+1}$. The effective context window was found to be roughly $\pm 1$ phone, with WavLM showing the clearest trapezoidal decay pattern.
• Orthogonality: Cosine similarity analysis revealed that vectors from the same relative position (e.g., all "current" features) are highly correlated, while vectors from different positions (e.g., "current" vs. "previous") are nearly orthogonal. This orthogonality holds across layers and models.
• Boundary Alignment: When analyzing frame-level representations across phonetic boundaries, the cosine similarity curves for "previous," "current," and "next" vectors intersected exactly at the annotated boundaries. This indicates the model dynamically switches its focus based on phonetic segments rather than fixed time windows.
• Model Differences: WavLM and HuBERT showed consistent performance in encoding context and maintaining orthogonality. Wav2vec 2.0 showed weaker context encoding and less consistent orthogonality, likely due to differences in training objectives (e.g., masking strategies).

5. Significance and Implications

• Unified Interpretability: The paper provides a unified framework explaining how S3Ms handle context. It suggests that S3Ms do not merely refine local acoustics but actively construct a structured, disentangled representation space where phonetic context is encoded via orthogonal subspaces.
• Unsupervised Segmentation: The discovery that position-dependent subspaces align with phonetic boundaries offers a concrete mechanism for unsupervised speech segmentation. This explains why S3Ms can be used to detect syllable and word boundaries without explicit supervision.
• Architectural Insights: The findings suggest that future neural architectures could explicitly model these position-dependent orthogonal subspaces to improve efficiency and interpretability.
• Discrete Units: The results help explain why discrete units derived from S3Ms (via clustering) often align better with phonetic categories than acoustic tokens from neural codecs; the S3M representations inherently encode the phonetic context necessary for linguistic categorization.

In conclusion, the paper demonstrates that transformer-based S3Ms encode phonetic context by superposing phonological vectors from neighboring phones into a single frame representation, utilizing position-dependent orthogonal subspaces to disentangle this information, with the boundaries of these subspaces emerging naturally from the phonetic structure of the speech signal.