⚡ electrical engineering

Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

This paper introduces GRIDS, a framework that leverages layer-wise Local Intrinsic Dimensionality (LID) in self-supervised speech models to detect anomalies by identifying how adversarial and noisy perturbations uniquely deform local geometric structures, thereby correlating LID shifts with automatic speech recognition degradation and enabling transcript-free monitoring.

Original authors: Sandra Arcos-Holzinger, Sarah M. Erfani, James Bailey, Sanjeev Khudanpur

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sandra Arcos-Holzinger, Sarah M. Erfani, James Bailey, Sanjeev Khudanpur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that listens to human speech and turns it into text. This robot is built using "self-supervised" learning, meaning it taught itself by listening to thousands of hours of audio without anyone telling it what the words were. It's incredibly good at its job, but we don't fully understand how its brain works, especially when things go wrong.

This paper introduces a new way to peek inside that robot's brain to see if it's being tricked or confused. Here is the breakdown using simple analogies.

The Problem: The Robot's "Brain" is a Mystery

When this speech robot listens to a word, it doesn't just hear sound; it converts the sound into a complex map of numbers (called representations). Think of these maps as a high-dimensional landscape.

Clean Speech: When the robot hears a clear voice, the numbers form a neat, organized path, like a well-paved highway.
Noise or Attacks: When there is background noise (like a babble of voices) or a malicious "adversarial" attack (a specific sound designed to confuse the robot), that highway gets distorted. It might turn into a muddy field or a chaotic maze.

Previous studies tried to measure this by looking at the "big picture" (global dimensionality) or comparing how similar two maps look. But the authors argue this is like looking at a city from a satellite: you can see the general shape, but you miss the potholes and detours happening on specific streets.

The Solution: GRIDS (A Local GPS)

The authors created a framework called GRIDS (Geometric Robustness via Intrinsic Dimensionality in Speech).

The Analogy: The Neighborhood Density Test
Imagine you are standing in a crowd.

Normal Situation (Clean Speech): If you look around, you see people standing at a comfortable, predictable distance from you. The crowd is organized.
The "Local Intrinsic Dimensionality" (LID) Test: This tool measures how crowded the immediate neighborhood is around you specifically.
- If the crowd suddenly becomes chaotic, with people scattered in weird, unpredictable directions all around you, the "local dimensionality" goes up. It means the space feels "higher" and more complex because it's harder to predict where the next person is.
- If the crowd stays organized, the dimensionality stays low.

The authors use this "crowd density" test on every single layer of the robot's brain (from the ears to the thinking center) to see how perturbations (noise or attacks) distort the local neighborhood.

What They Found

1. The "Early Warning System"
They discovered that when the robot is attacked, the "crowd density" (LID) spikes, but only in the early layers of the brain.

Benign Noise (Real-world noise): If you just add some background chatter, the robot's brain gets a little messy at first, but as the signal gets clearer (higher volume), the brain cleans itself up. The "crowd" returns to normal.
Adversarial Attacks (Tricks): Even if the attack is quiet (low volume), the robot's brain stays messy in the early layers. It's like a ghost that haunts the front door; the robot never fully recovers its organized structure, even if the noise is faint.

2. The Connection to Mistakes
They found a direct link between this "messiness" and the robot making mistakes.

The Metaphor: Think of the robot's brain as a factory assembly line. If the raw materials (the sound waves) arrive in a chaotic, high-dimensional mess, the workers on the first few stations get confused. This confusion ripples down the line, and by the time the product (the text transcript) comes out, it's full of errors.
The Result: Whenever the "local dimensionality" (LID) went up, the Word Error Rate (WER) also went up. The messier the internal map, the more mistakes the robot made.

3. Catching the Tricksters (Anomaly Detection)
The most exciting part is that they used this "messiness" metric to build a security guard.

They took the LID measurements from all 12 layers of the robot's brain and fed them into a simple classifier.
The Result: This system could tell the difference between a robot hearing normal noise and a robot being attacked with 78% to 100% accuracy.
Why it matters: This is a "transcript-free" monitor. Usually, to know if a speech system is failing, you need to know the correct answer (the transcript) to compare it against. This new method can tell you "Something is wrong with the input" just by looking at the robot's internal geometry, without needing to know what the correct words were.

Summary

The paper shows that by measuring how "crowded" and chaotic the robot's internal map gets in its early layers, we can detect if it's being tricked or confused.

Real noise causes temporary chaos that the robot recovers from.
Adversarial attacks cause a persistent, deep-seated chaos that the robot cannot fix.
This "chaos meter" (LID) is a powerful tool to spot bad inputs before the robot even tries to write down the text.

The authors conclude that this geometric approach offers a new way to monitor these powerful speech models, ensuring they stay on the "paved highway" rather than wandering off into the "muddy fields" of error.

Technical Summary: Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

1. Problem Statement

Self-supervised speech models (S3Ms), such as WavLM and wav2vec 2.0, have achieved state-of-the-art performance in downstream tasks like Automatic Speech Recognition (ASR). However, their internal representations remain poorly understood when subjected to natural (benign) and adversarial perturbations. Prior research has largely relied on representation similarity metrics (e.g., CCA, CKA) or global dimensionality measures (e.g., effective rank) to analyze these models. These approaches offer limited visibility into local geometric changes within the learned representation space.

The authors pose a critical question: How do perturbations deform the local geometry of S3M representations, and do these geometric shifts correlate with downstream ASR degradation? Specifically, they investigate whether Local Intrinsic Dimensionality (LID) can serve as a diagnostic tool to detect anomalies and monitor robustness without requiring ground-truth transcripts.

2. Methodology: The GRIDS Framework

The authors propose GRIDS (Geometric Robustness via Intrinsic Dimensionality in Speech), a framework that utilizes LID to analyze layer-wise representations in transformer-based S3Ms.

Local Intrinsic Dimensionality (LID): LID quantifies the geometric properties of learned representations by measuring the rate at which the number of neighboring samples grows as the radius around a reference sample increases. It estimates the effective local dimensionality (degrees of freedom) required to describe the neighborhood of a sample.
Experimental Setup:
- Models: WavLM and wav2vec 2.0 (BASE variants, 12 layers).
- Data: 918 paired utterances from LibriSpeech test-clean, ensuring identical utterances are used across all perturbation types for fair comparison.
- Perturbations:
  - Benign: Gaussian noise, Noizeus babble noise, and competing speech (LibriSpeech), all rescaled to match target Signal-to-Noise Ratios (SNR) of 0, 10, 20, 30, and 40 dB.
  - Adversarial: Projected Gradient Descent (PGD) attacks under an $\ell_2$ -bounded budget calibrated to the same target SNR. Two objectives were used: Mean Squared Error (MSE) to distort final-layer representations, and Connectionist Temporal Classification (CTC) loss to disrupt transcriptions.
LID Computation:
- Frame-level embeddings are extracted from each of the 12 transformer layers.
- Local LID is estimated per frame using a $k$ -nearest neighbor (kNN) approach (Levina–Bickel MLE estimator).
- Aggregation: For geometric analysis and ASR monitoring, frame-level LIDs are aggregated via a harmonic mean to produce layer-level and condition-level summaries. For anomaly detection, utterance-level 12-dimensional feature vectors (one LID value per layer) are constructed.
- Hyperparameter Selection: The neighborhood size $k$ is not fixed globally but selected per condition to maximize the stability and discriminability of the perturbation-induced LID shift ( $\Delta$ LID).

3. Key Contributions

The paper makes three primary contributions:

LID–S3M Geometric Analysis: A layer-wise measurement of local geometric changes in learned representations across clean, benign, and adversarial conditions.
LID–ASR Monitoring: Empirical evidence demonstrating that increases in LID positively associate with Word Error Rate (WER) degradation, linking geometric shifts to downstream task performance.
LID–AD for Anomaly Detection: The development of a 12-dimensional LID-derived feature set that enables the classification of adversarial versus benign inputs, achieving high AUROC scores.

4. Results and Analysis

Geometric Behavior (LID-S3M):

Perturbation Effects: All perturbations increase LID at low SNR. However, a divergence occurs at high SNR (30–40 dB): benign noise converges toward the clean profile (LID decreases), while adversarial inputs retain elevated LID, particularly in early transformer layers.
Layer-wise Trajectories: Adversarial perturbations cause the most persistent LID elevation in early layers. While benign interference also raises LID at low SNR, it diminishes rapidly as SNR improves. This suggests that early-layer geometric distortion is a relative marker of adversarial shifts under matched SNR conditions.
Model Differences: The trend holds for both WavLM and wav2vec 2.0, though WavLM (trained with denoising objectives) shows slightly different robustness profiles. PGD-MSE attacks generally produce larger and more persistent LID elevations than PGD-CTC attacks.

Correlation with ASR (LID-ASR):

There is a consistent co-occurrence between geometric shifts ( $\Delta$ LID) and ASR degradation ( $\Delta$ WER). Both metrics decrease as SNR increases.
At low SNR (0–10 dB), adversarial attacks and severe benign interference (speech overlap) cause the largest deviations in both LID and WER.
Notably, at high SNR, benign corruptions result in near-zero WER increases, whereas adversarial attacks retain a non-trivial $\Delta$ LID footprint even when WER increases are small. This indicates that LID can detect geometric displacement even when decoding errors are minimal.

Anomaly Detection (LID-AD):

A lightweight logistic regression classifier trained on the 12-dimensional LID vectors effectively distinguishes adversarial from benign inputs.
Performance: The framework achieves AUROC scores ranging from 0.78 to 1.00 across different models and SNR levels.
- WavLM with MSE attacks showed the highest separability (AUROC 0.99 at 0–20 dB).
- Performance degrades at high SNR (40 dB) as both benign and adversarial inputs converge toward the clean manifold, reducing separability.
Robustness to Success Rate: The classifier retains discriminative power even when the Attack Success Rate (SR) is low (e.g., WavLM-CTC at 30 dB had an AUROC of 0.92 despite an SR of only 0.04), suggesting LID captures geometric anomalies that do not always manifest as immediate transcription failures.

5. Significance and Claims

The authors claim that GRIDS provides a transcript-free monitoring capability for S3Ms. By leveraging LID, the framework offers an interpretable geometric diagnostic that:

Detects adversarial inputs without needing ground-truth labels or transcripts.
Reveals that geometric distortion propagates through early representations before being partially absorbed by deeper layers.
Complements existing similarity-based analyses by capturing local distortions that global measures miss.

The paper concludes modestly, noting current limitations: the study is restricted to 12-layer S3M variants and untargeted attacks. Future work is proposed to scale to larger architectures, extend to security-critical tasks (e.g., speaker verification), and combine LID with global spectral measures for multi-scale geometric characterization.