⚡ electrical engineering

Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings

This paper demonstrates that CLAP audio embeddings reliably encode fundamental acoustic attributes like reverberation, loudness, and pitch through distinct linear and non-linear regimes, revealing geometric consistency across datasets and alignment with text embeddings of acoustic descriptors.

Original authors: Héctor Martel, Joe Hennessy-Priest, Taemin Cho

Published 2026-07-07✓ Author reviewed ⓘ

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Héctor Martel, Joe Hennessy-Priest, Taemin Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that listens to music and speech. When it hears a sound, it doesn't just store the audio file; it translates the sound into a unique "fingerprint" made of numbers. This robot is called CLAP, and it's famous for understanding both what sounds are (like "a guitar") and what they feel like (like "sad" or "loud").

But here's the mystery: How exactly does this robot organize those numbers inside its brain? Does it keep a neat list of how loud a song is? Does it have a specific slot for how echoey a room sounds? Or is all that information just jumbled up in a messy pile?

This paper is like a detective story where the authors try to "interrogate" the robot's brain to find out. They use a tool called a probing framework, which is basically a simple test to see if the robot's fingerprint contains specific, low-level details about the sound.

The Three Clues They Looked For

The researchers tested if the robot could reveal three specific things about any sound it heard:

The Echo (RT60): How long does the sound hang in the air before fading away? (Think of the difference between shouting in a small closet vs. a giant cathedral).
The Volume (LUFS): How loud is the sound to a human ear? (Not just the raw power, but how our ears perceive it).
The "Color" of the Sound (Spectral Content): Is the sound bright and sharp (like a whistle) or deep and muffled (like a bass drum)? They measured this in two ways: the raw "center of mass" of the sound (SC) and the musical pitch version of that (RP).

The Experiment: The "Frozen Brain" Test

To test the robot, the researchers froze its brain. They didn't let it learn anything new. They just took the numbers (embeddings) it had already created and tried to train a tiny, simple calculator (a "probe") to guess the Echo, Volume, or Color based only on those numbers.

They used different levels of calculators:

The Simple Calculator (Linear): Just a straight line. If the robot's brain is organized in a straight, logical way, this simple tool should work perfectly.
The Smart Calculator (Non-Linear): A slightly more complex tool that can handle curves and twists. If the simple tool fails but this one works, it means the information is there, but it's hidden in a complicated shape.

What They Found

1. The Echo and Volume are "Straight Lines"
The robot's brain is surprisingly organized when it comes to Echo and Volume.

The Analogy: Imagine the robot's brain is a giant library. For Echo and Volume, the books are stacked in a perfectly straight row. If you know where you are in the row, you know exactly how echoey or loud the sound is.
The Result: A simple, straight-line calculator could guess these values very accurately. Even better, this "straight row" was the same no matter if the sound was a human voice, a drum, or white noise. The robot uses the same "direction" in its brain to represent echo, regardless of the source.

2. The "Color" is a "Curved Path"
The Spectral Content (the brightness or pitch of the sound) was trickier.

The Analogy: If Echo is a straight row of books, the "Color" of the sound is like a winding, curved path through a forest. A straight line can't follow the path; you need a tool that can turn corners.
The Result: The simple calculator failed miserably here. It couldn't guess the color of the sound. But the "Smart Calculator" (the non-linear one) could do it easily. This means the robot does know the color of the sound, but it stores it in a complex, curved way that requires a more advanced tool to read.

3. The "Pitch" is a "Local Map"
The Relative Pitch (the musical note version of the color) was interesting.

The Analogy: The robot has a map for pitch, but it's a different map for every city. If you are in the "Speech City," the map looks one way. If you are in the "Music City," the map looks completely different.
The Result: The robot could guess the pitch well, but the "direction" it used to store that info changed depending on whether the sound was speech, music, or noise. It didn't have one universal "pitch direction" like it did for echo.

4. The "Volume" Surprise
The researchers checked other similar robots (different AI models). They found that some robots had been built to ignore volume entirely (like a robot that only cares about the shape of the sound, not how loud it is). For those robots, no amount of testing could find the volume information because it was never put in there to begin with.

The "Magic Text" Connection

Finally, they tested if the robot's understanding of sound matched its understanding of words.

They took the "Echo" direction they found in the audio brain and pointed it at text descriptions like "dry room" or "long reverb."
The Result: The text descriptions landed exactly where they should. "Dry" pointed to the low-echo end, and "Long reverb" pointed to the high-echo end. This proves the robot's "audio brain" and "language brain" are speaking the same language regarding echo.

The Bottom Line

The paper concludes that the CLAP robot is a very organized librarian.

It keeps Echo and Volume in neat, straight, universal rows that anyone can read.
It keeps Sound Color in a complex, curved maze that requires a smart tool to navigate.
It keeps Pitch in different maps for different types of sounds.

Most importantly, all this information is actually there. The robot isn't missing the data; it just stores it in different shapes depending on what the data is. This means we can use this single robot to figure out echo, volume, and sound color all at once, provided we use the right "key" (simple or complex) to unlock each piece of information.

Technical Summary: Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings

Problem Statement
Audio foundation models, particularly CLAP (Contrastive Language-Audio Pretraining), are widely utilized as general-purpose feature extractors for tasks ranging from retrieval to conditional generation. However, the internal structure of their learned representations remains insufficiently understood. While previous interpretability work has focused on high-level semantic attributes (e.g., emotion, speaker identity) or qualitative concept alignment, there is a lack of systematic quantitative analysis regarding how low-level physical acoustic descriptors are encoded. Specifically, it is unclear whether attributes such as reverberation time, loudness, and spectral content are explicitly represented in the embedding space, and if so, whether they are recoverable via linear or non-linear mappings.

Methodology
The authors employ a probing framework to evaluate the encoding of three fundamental perceptual dimensions within the CLAP embedding space:

Reverberation Time (RT60): A measure of room acoustic decay.
Loudness (LUFS): Integrated loudness per ITU-R BS.1770 standards.
Spectral Content: Measured via Spectral Centroid (SC) and its logarithmic transformation, Relative Pitch (RP).

The study utilizes the LAION-CLAP audio encoder (frozen) to generate 512-dimensional embeddings from five distinct datasets spanning white noise, speech, monophonic notes, and complex music mixtures (NSynth, VCTK-Corpus, MusDB18HQ, SonicMaster). To isolate acoustic attributes, the authors apply targeted data augmentations (e.g., synthetic room impulse responses for RT60, loudness normalization for LUFS, and pitch shifting or bandpass filtering for SC) to generate ground-truth labels.

Three probe architectures of increasing complexity are trained to predict these scalar attributes from the frozen embeddings:

Linear Probe: A single affine map to test for linear decodability.
MLP Probe: A shallow two-layer network to capture non-linear relationships.
Kernel Probe: Kernel Ridge Regression with an RBF kernel to serve as a flexible, non-parametric upper bound for non-linear performance.

Evaluation relies on Mean Absolute Error (MAE), Coefficient of Determination ( $R^2$ ), and Pearson Correlation ( $r$ ). The study further analyzes the geometric consistency of learned feature axes across datasets and generalizes findings to eight other audio foundation models (including MS-CLAP, MERT, Wav2Vec2, WavLM, Whisper, and VGGish).

Key Results

Universal Recoverability: All investigated attributes are reliably recoverable from CLAP embeddings using non-linear probes across all datasets.
Two Encoding Regimes:
- Linear Regime: RT60, LUFS, and RP are approximately linearly encoded. Linear probes achieve strong performance ( $R^2 \geq 0.67$ for RT60, $\geq 0.76$ for LUFS, and $\geq 0.73$ for RP on most datasets).
- Non-Linear Regime: Spectral Centroid (SC) requires non-linear probes. Linear probes perform at or below chance on most datasets ( $R^2 \leq 0.43$ , often negative), while MLP and Kernel probes achieve high accuracy ( $R^2 \geq 0.84$ ).
Geometric Consistency:
- The feature axes for RT60 and LUFS are geometrically consistent across datasets, suggesting a common, domain-invariant direction in the embedding space.
- The RP axis is highly domain-specific, with axes being nearly orthogonal across different audio domains (e.g., speech vs. music), despite being linearly recoverable within each domain.
Generalization: The linear encoding of RT60 and RP generalizes across eight additional audio foundation models. LUFS encoding varies by architecture; amplitude-invariant models (Wav2Vec2, WavLM-Large, MERT) discard loudness entirely by construction. SC remains non-linearly encoded in most models, with the exception of MS-CLAP checkpoints, which show partial linear recovery.
Cross-Modal Consistency: A qualitative demonstration shows that text embeddings of acoustic descriptors (e.g., "dry," "long reverb") align geometrically with the RT60 feature direction identified by the audio probe, confirming cross-modal consistency for this attribute.

Significance and Claims
The paper claims to provide the first systematic quantitative evidence that low-level acoustic attributes are explicitly encoded in CLAP embeddings. The primary contribution is the identification of distinct encoding regimes: while some physical descriptors (RT60, LUFS) map to linear, geometrically consistent directions in the embedding space, others (SC) reside on curved manifolds requiring non-linear decoding.

The authors suggest these findings have practical implications for audio practitioners, potentially enabling the use of a single frozen foundation model for simultaneous estimation of reverberation, loudness, and spectral content. The geometric consistency of RT60 and LUFS axes hints at their utility for workflows such as automatic mix analysis and text-driven effect control. However, the authors remain modest, noting that validating these specific applications remains future work. They also acknowledge limitations, including the exclusive focus on final-layer embeddings, the use of simplified "shoebox" geometry for RT60 augmentation, and the inability to guarantee fully dry source material in music mixture datasets.

The Three Clues They Looked For

The Experiment: The "Frozen Brain" Test

What They Found

The "Magic Text" Connection

The Bottom Line

More like this