💬 NLP

Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English

This paper demonstrates through layer-wise probing of wav2vec 2.0 and Whisper that modern speech models encode African American English consonant cluster reduction not as simple segmental deletion, but as structured gradient phonological variation that preserves cues to underlying stop identities.

Original authors: Hamid Mojarad, Kevin Tang

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hamid Mojarad, Kevin Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two very smart, high-tech "listening robots." One robot, wav2vec 2.0, learned by listening to thousands of hours of audio without anyone telling it what the words were (like a baby learning to speak). The other, Whisper, learned by listening to audio while reading the exact text that went with it (like a student studying with a textbook).

The researchers wanted to see how these robots handle a specific quirk of African American English (AAE) called Consonant Cluster Reduction (CCR).

The "Missing Letter" Mystery

In everyday speech, people often drop the last sound in a group of consonants. For example, instead of saying "test," they might say "tes." Instead of "hand," they might say "han." To a standard computer program, this looks like a mistake or a missing piece of a puzzle.

The big question was: Do these robots think the sound is actually gone forever, or do they still "know" it was there?

The Experiment: X-Ray Vision

To find out, the researchers didn't just ask the robots to transcribe the words. Instead, they used a technique called "layer-wise probing."

Think of the robots as having 12 layers of "thinking" (like layers of an onion or floors in a skyscraper).

Bottom floors: Hear the raw sound waves (noise, pitch).
Middle floors: Recognize specific sounds (like "t" or "d").
Top floors: Understand the meaning of the whole word.

The researchers peeked inside each floor to see what information was stored there. They ran two main tests:

Test 1: The "Spot the Difference" Game

They showed the robots a mix of words with the full sound (e.g., "test") and the reduced sound (e.g., "tes").

The Result: Both robots were very good at telling the difference. They could hear that one had a "t" and the other didn't.
The Twist: However, the robots didn't treat the reduced sound as "empty." Even when the "t" was missing, the robot's internal "ears" still picked up subtle clues left behind by the missing sound. It was as if the robot could smell the "ghost" of the missing letter.

Test 2: The "Magic Restoration" Game

This was the more impressive test. The researchers gave the robots only the reduced sound (e.g., just "tes" or "han") and asked: "Can you guess what the original word was supposed to be?"

The Result: The robots were incredibly accurate (over 90% success). Even though the "t" or "d" was physically missing from the audio, the robots' internal representations still contained enough information to reconstruct the original word perfectly.

What This Means (The Big Picture)

The study concludes that these modern AI models are not just treating these speech patterns as simple errors or deleted sounds.

Instead, they are treating them like structured, gradient variations.

Analogy: Imagine you are looking at a photo of a person wearing a hat. If you blur the photo, you can't see the hat clearly. A simple computer might say, "No hat detected." But these advanced robots are like a detective who says, "I can't see the hat, but the shape of the head and the shadow suggest a hat is there." They understand the context and the rules of the language, not just the raw audio.

Why It Matters for Bias

The paper notes that these robots often make more mistakes with African American English speakers. This study suggests that the robots do understand the sounds, but perhaps they struggle because they are trained mostly on "standard" English where those sounds are always present. The robots know the "ghost" of the missing sound, but they might not be as practiced at using that clue to predict the word as well as they do with standard speech.

Summary

In short, the researchers found that these AI listening robots are surprisingly sophisticated. When an African American English speaker drops a consonant, the robots don't just hear a gap; they hear a patterned variation that still holds the key to the original word. They have learned that language is fluid and that missing sounds often leave behind a "fingerprint" that smart models can read.

Technical Summary: Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English

Problem Statement
Automatic Speech Recognition (ASR) systems exhibit significant performance disparities across demographic groups, with notably higher error rates for speakers of African American English (AAE). A primary source of this bias is Consonant Cluster Reduction (CCR), a systematic phonological process in AAE where word-final clusters are simplified (e.g., test /tEst/ $\rightarrow$ [tEs]). While prior behavioral studies have documented these error rates, the internal mechanisms within modern speech encoders that lead to this disparity remain underexplored. Specifically, it is unclear whether models treat CCR as a simple segmental deletion or as a structured, gradient phonological variation where cues to the underlying form are preserved.

Methodology
The authors conducted a speaker-independent, layer-wise probing study using two prominent speech models: wav2vec 2.0-base (self-supervised) and Whisper-small (supervised). Both models utilize 12 Transformer layers with 768-dimensional embeddings, allowing for direct architectural comparison.

Data: The study utilized the Corpus of Regional African American Language (CORAAL), selecting 6,760 tokens from 156 speakers. The dataset focused on seven frequent two-consonant clusters (/ft/, /nd/, /nt/, /st/, /sk/, /pt/, /mp/) in monomorphemic words, balanced between canonical and reduced forms.
Feature Extraction: Forced alignment was performed using the Montreal Forced Aligner (MFA) to identify temporal boundaries of consonant clusters. Hidden states were extracted from all 12 transformer layers, mean-pooled to create fixed-length token embeddings.
Probing Tasks: Two domain-informed tasks were performed using frozen encoders and trained Multi-Layer Perceptron (MLP) classifiers:
1. Segmental Reduction Detection: Classifying tokens as reduced vs. canonical.
2. Segmental Restoration: Specifically testing /nt/ and /nd/ clusters. The model was trained on reduced nasal-only inputs (/n/) and tested on its ability to distinguish the underlying canonical cluster identity (/nt/ vs. /nd/).
3. Coarticulatory Encoding (Supplementary): A "gating" paradigm tested how much information about the reduction contrast could be recovered from the first consonant (C1) alone, in 25% increments up to 100%.

Key Results

Reduction Detection: Both models successfully distinguished reduced from canonical forms with accuracies significantly above chance (typically 70–80%).
- wav2vec 2.0 exhibited a complex layer-wise pattern with multiple peaks (notably around layers 5 and 9) and a moderate decline in later layers.
- Whisper-small generally followed a rising-then-plateau trajectory, peaking in mid-to-late layers.
- Cluster Sensitivity: Performance varied by cluster type, aligning with linguistic constraints. For instance, /st/ clusters (where /s/ precedes the stop) yielded the highest detection accuracy, consistent with the high probability of reduction in this context. Whisper demonstrated greater robustness on low-frequency clusters compared to wav2vec 2.0.
Segmental Restoration: In the restoration task, both models achieved very high accuracy (93–96%) in distinguishing the underlying identity of reduced segments. This indicates that reduced forms are not encoded as "deleted" or empty, but retain substantial cues to the missing stop.
Coarticulation: The gating analysis revealed that C1 frames alone contained significant information about the reduction contrast, with accuracy increasing as more of the C1 segment was included. This suggests that the models encode strong coarticulatory cues between the remaining consonant and the deleted stop.

Key Contributions

Internal Representation Analysis: The study provides the first computational analysis of how modern speech encoders internally represent AAE-specific CCR, moving beyond surface-level error metrics to examine encoded linguistic information.
Gradient vs. Deletion: The results challenge the notion that models treat CCR as simple segmental deletion. Instead, the high accuracy in restoration tasks demonstrates that reduced realizations are encoded as structured, gradient phonological variations that remain close to their canonical counterparts in embedding space.
Model Comparison: The paper highlights distinct layer-wise behaviors between self-supervised (wav2vec 2.0) and supervised (Whisper) models, noting that Whisper's supervised training on massive labeled data may confer greater robustness in detecting dialectal variations, particularly in low-resource cluster types.

Significance and Claims
The authors claim that their findings demonstrate that modern speech models encode AAE CCR patterns as structured phonological variation rather than simple noise or deletion. The fact that models can "restore" the underlying stop identity from reduced inputs suggests that these systems possess an internal representation of the phonological rules governing AAE, even if they fail to transcribe them perfectly in end-to-end ASR tasks.

The study concludes that the observed ASR disparities are not necessarily due to a total lack of phonetic sensitivity to AAE features, but rather that the models treat these features as a continuum of gradient variation. This nuanced encoding explains why binary classification of reduced vs. canonical forms is difficult (due to the lack of a sharp phonetic boundary) while the underlying phonological identity remains recoverable. The authors maintain a modest scope, noting that their findings are specific to the two models and cluster types examined, and that further research is needed to generalize these patterns to other architectures and more complex morphological contexts.