💬 NLP

A framework for analyzing concept representations in neural models

This paper introduces a unified framework for analyzing neural concept representations along the axes of containment and disentanglement, revealing that the choice of estimation method significantly impacts these properties and highlighting specific challenges in generalizing concept erasure methods and isolating speaker information in speech models.

Original authors: Burin Naowarat, Hao Tang, Sharon Goldwater

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Burin Naowarat, Hao Tang, Sharon Goldwater

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical library where every book represents a thought or a piece of information. Inside this library, the books aren't organized on shelves; instead, they are floating in a 3D (or even 1,000-dimensional) cloud of space.

The paper by Burin Naowarat and colleagues asks a simple but tricky question: If we want to find a specific "idea" (like "gender" or "a specific phone sound") inside this cloud, where exactly is it hiding?

To answer this, the authors built a new "flashlight" framework to shine on these ideas and see how well they are hidden or separated from other ideas.

The Two Main Questions: "Is it in there?" and "Is it mixed up?"

The authors say that to understand an idea (a "concept"), we need to check two things:

Containment (The "All-in-One" Box):
- The Analogy: Imagine you have a box labeled "Gender." If you put all the "Gender" information inside this box, is everything about gender inside? And is nothing about gender left outside the box?
- The Paper's Terms: They call this Retention (how much is inside) and Leakage (how much is accidentally left outside). A good concept box should have high retention and low leakage.
Disentanglement (The "Clean Separation"):
- The Analogy: Imagine you have a box for "Gender" and a box for "Profession." If you open the "Gender" box, do you accidentally find "Profession" information mixed in? And if you throw away the "Gender" box, does the "Profession" box get ruined?
- The Paper's Terms: They call this Purity (is the box free of other stuff?) and Interference (does removing this box break other things?). A good concept box should be pure and shouldn't break other boxes when removed.

The Big Discovery: The "Map" Depends on How You Draw It

The most surprising thing the authors found is that there isn't just one correct way to draw the box.

They tested five different methods (like different cartographers) to draw the "Gender" box in a text model (BERT).

Method A drew a box that held all the gender info perfectly but leaked some out.
Method B drew a box that held all the gender info perfectly and leaked almost nothing out.

The Takeaway: If you only use Method A, you might conclude, "Oh, gender info is everywhere!" If you use Method B, you might conclude, "Oh, gender info is neatly packed!" The paper warns that your conclusion depends entirely on which "map-maker" (estimator) you choose. You can't just pick one and assume it's the only truth.

Testing on Speech: The "Voice" vs. The "Word"

They also tested this on a speech model (HuBERT) that listens to human voices. They looked at two concepts:

Phonemes: The actual sounds of words (like "ba" or "da").
Speaker Identity: Who is talking (like "John" or "Sarah").

What they found:

The Sounds (Phonemes): These were easy to find. The model had a very clear, tight "box" for sounds. You could isolate the sounds, and they stayed separate from the speaker's identity.
The Voices (Speakers): This was much harder. The model didn't seem to have a neat, compact box for "John's voice."
- When they tried to build a box for a specific speaker, it was "leaky" (it included sounds that weren't just about the voice).
- Even worse, if they trained the box on a small group of people, it failed completely when they tried to use it on a new person they hadn't seen before. It's like trying to make a "John" box based on 40 people, and then realizing it doesn't work for the 41st person.

The "Magic Eraser" Problem

The paper also looked at a popular tool called LEACE, which is like a "Magic Eraser" designed to remove a concept (like gender) from the model without hurting the rest.

The Good News: It works great on the data it was trained on. It erases the concept perfectly.
The Bad News: When they tried to use this eraser on new, unseen data, it started leaking. The concept wasn't fully gone; it was just hiding in the cracks. This suggests that while these tools are good for cleaning up what we already know, they struggle to generalize to new situations.

Summary in a Nutshell

The authors built a checklist to see how well AI models organize their thoughts. They found that:

How you look for a concept changes what you find. Different tools draw different boundaries.
Some concepts are easy to isolate (like the sounds of words), while others are messy and hard to pin down (like specific voices).
Current "eraser" tools are great at cleaning up known data but often fail to generalize to new data, meaning the "mess" might still be there when you look at something new.

The paper doesn't promise to fix these models or cure diseases; it simply provides a better ruler and a better flashlight so researchers can stop guessing and start measuring exactly how these AI models think.

Technical Summary: A Framework for Analyzing Concept Representations in Neural Models

Problem Statement
Understanding how neural models represent human-interpretable concepts remains a significant challenge in AI and cognitive science. While prior work has established that many concepts (e.g., gender, tense, phonetic classes) are represented as linear directions or low-dimensional subspaces within high-dimensional representation spaces, there is no unified framework for evaluating these claims. Different studies rely on disparate assumptions and methods: "probing" studies focus on whether concepts are decodable from a subspace, while "concept erasure" studies focus on whether concepts can be removed without damaging other information. Crucially, these approaches often use different methods to identify subspaces without systematic comparison, leading to potentially conflicting conclusions about the structure of concept representations.

Methodology
The authors propose a unified framework to characterize concept representations along two primary axes: containment and disentanglement.

Definitions:
- Concept Subspace ( $X_Y$ ): A linear subspace spanned by basis elements intended to represent a concept $Y$ .
- Containment: Measures if information about $Y$ $Y$ is fully contained within $X_Y$ $X_{Y}$ and absent from its complement ( $X^c_Y$ $X_{Y}^{c}$ ). This is operationalized via:
  - Retention: Accuracy of predicting $Y$ from $X_Y$ .
  - Leakage: Accuracy of predicting $Y$ from $X^c_Y$ .
- Disentanglement: Measures the isolation of $X_Y$ $X_{Y}$ from a second concept $Y'$ $Y^{'}$ . This is operationalized via:
  - Purity: Error rate of predicting $Y'$ from $X_Y$ (low error implies poor purity).
  - Interference: Error rate of predicting $Y'$ from $X^c_Y$ after erasing $X_Y$ (high error implies high interference).
Estimators:
The framework evaluates five supervised concept estimators to determine how the choice of method impacts these metrics:
- Multinomial Logistic Regression (MLR)
- Linear Discriminant Analysis (LDA)
- Centroid Principal Component Analysis (CPCA)
- Covariance of input representation and concepts (COV)
- LEACE (Linear Concept Erasure)
- Baseline: Random Projection (RAND)
Experimental Setup:
To ensure rigorous evaluation of generalization, the authors partition data into three disjoint sets: space-train (for estimating the subspace), probe-train (for training probing classifiers), and test (for evaluation). This contrasts with prior work that often uses overlapping sets, which can artificially guarantee zero leakage.

Experiments were conducted on:
- Text: BERT representations using gender and profession concepts (Short Biography dataset).
- Speech: HuBERT representations using phonetic categories (phones) and speaker identity (LibriSpeech).

Key Results

Non-Uniqueness of Subspaces: In BERT experiments, different estimators (MLR vs. LEACE) found distinct subspaces with high retention but vastly different leakage. This demonstrates that concept subspaces are not uniquely determined; high retention does not guarantee that a subspace is the only or best representation of a concept.
Estimator Impact: The choice of estimator significantly alters containment and disentanglement scores.
- LEACE: Performed best overall, finding subspaces with strong retention and low leakage on seen data. However, it exhibited significant leakage on unseen data, challenging its "no-leakage" guarantee when generalizing.
- MLR/LDA: Achieved optimal retention but suffered from high leakage, indicating substantial concept information exists outside the estimated subspace.
Speech Domain Findings (HuBERT):
- Phone Information: Phone subspaces were found to be both well-contained and disentangled from speaker information.
- Speaker Information: Speaker subspaces were difficult to contain in a compact subspace. While speaker information was disentangled from phones (erasing speaker subspace did not harm phone recognition), achieving high retention for speakers required large numbers of dimensions and speakers.
- Generalization: Speaker subspaces estimated on a small set of speakers failed to generalize to unseen speakers, showing poor containment. Increasing the number of training speakers and subspace dimensions improved containment but eventually degraded purity (increasing phone information leakage into speaker subspaces).

Significance and Claims
The paper claims to provide a necessary formalization for analyzing concept representations, moving beyond single-metric evaluations. Its primary contributions are:

A Unified Framework: It clarifies the assumptions behind different lines of research by defining four complementary metrics (retention, leakage, purity, interference) that capture both individual concept representation and inter-concept relationships.
Methodological Awareness: It highlights that conclusions about concept structure are heavily dependent on the choice of the subspace estimator. The authors argue that relying on a single estimator (e.g., one optimized only for retention) can lead to incomplete or misleading structural analyses.
Empirical Insights: The framework reveals that while some concepts (like gender in BERT or phones in HuBERT) have robust linear representations, others (like speaker identity) are difficult to capture in compact, generalizable linear subspaces.
Limitations of Current Erasure: The study demonstrates that even state-of-the-art erasure methods like LEACE struggle to generalize to unseen data, suggesting that "no-leakage" guarantees on training data do not extend to the broader representation space.

The authors conclude that their framework offers a more rigorous basis for future work in interpreting and controlling neural models, particularly by emphasizing the need to evaluate both what is inside a concept subspace and what remains outside it.