💬 NLP

Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms

This paper introduces a distribution-level unsupervised feature discovery method that clusters LLM continuations by jointly optimizing semantic coherence and mechanistic attribution signatures, thereby revealing diverse continuation modes and actionable internal mechanisms that single-view, target-conditioned analyses often miss.

Original authors: Hyunjin Cho, Youngji Roh, Jaehyung Kim

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyunjin Cho, Youngji Roh, Jaehyung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Answer" Trap

Imagine you are a detective trying to understand how a magic box (a Large Language Model) works. Usually, when you ask the box a question, it gives you one answer. To figure out how the box works, you might look at the gears and springs inside only for that specific answer.

The paper argues that this is a trap. If you only look at one answer, you might miss the fact that the box has many different "modes" of thinking. Sometimes, two answers might sound very similar (same meaning) but were built using completely different internal gears. Other times, two answers might sound totally different but were built using the exact same gears.

The authors call this the "Misalignment between semantic and mechanistic spaces." It's like two houses that look identical from the outside (same paint, same windows) but have completely different plumbing systems inside. Or, two houses that look totally different (one is a castle, one is a hut) but share the exact same plumbing.

The Solution: A New Way to Group Answers

Instead of picking one specific answer to study, the authors propose a new method to look at all the possible answers the box could give to a single question at once. They call this "Distribution-Level Unsupervised Feature Discovery."

Here is how their method works, broken down into three simple steps:

1. The "Sample the Crowd" Step

Instead of asking the box for one answer, they ask it for hundreds of different answers to the same question.

Analogy: Imagine asking a chef, "How do you make soup?" instead of getting one recipe, you get 500 different variations. Some are spicy, some are creamy, some use chicken, some use tofu.

2. The "Double-View" Step

For every single soup recipe, the team creates two different "ID cards":

The Semantic Card (What it says): This describes the meaning of the soup. Is it spicy? Is it a tomato soup?
The Mechanistic Card (How it's made): This describes the internal gears the model used to create it. Which specific neurons fired? Which internal "switches" were flipped?
Analogy: For every soup, you write down the flavor profile (Semantic) and the kitchen equipment list used (Mechanistic). You realize that two soups might taste the same (Semantic match) but one was made with a blender and the other with a mortar and pestle (Mechanistic mismatch).

3. The "Smart Sorting" Step

Now, they have a huge pile of soup recipes, each with two ID cards. They use a special mathematical sorting machine (called Rate-Distortion Clustering) to group these recipes.

The Goal: They want to find groups where the soups are similar in both flavor and equipment.
The Trade-off: The machine has a "knob" (called $\beta$ $β$ ) that controls how strict the sorting is.
- Loose setting: It groups soups that are just generally "soup-like."
- Strict setting: It separates them into tiny groups like "Spicy Tomato Soup made with a Blender" vs. "Spicy Tomato Soup made with a Mortar."
The Result: They find hidden "modes" of thinking that a human looking at just one answer would never see.

Why This Matters: The "Steering" Test

The paper doesn't just say, "Look, we found groups." They prove these groups are real by doing a "steering" test.

The Test: They take a group of soups that the machine identified as "Spicy Tomato made with a Blender." They then try to force the magic box to make more soups like that by turning up the "Blender" switch inside the machine.
The Result: When they turn up the switch, the box actually starts making more "Spicy Tomato" soups.
The Comparison: When they tried to do this using only the "flavor" (Semantic) or just picking one random soup, the steering didn't work as well.
Analogy: It's like realizing that to get a specific type of cake, you don't just need to say "I want cake"; you need to know exactly which mixer and oven settings produce that specific texture. The authors found the "mixer settings" for different types of thinking.

The Main Takeaway

The paper introduces a tool that helps us audit AI models by looking at the whole landscape of possible answers, not just one.

Old Way: Pick one answer, find the gears that made it. (Like studying one car to understand how all cars work).
New Way: Look at all the cars the factory could build, group them by how they are built and what they look like, and find the hidden patterns.

This helps researchers understand that an AI model isn't just a single path of logic; it's a complex landscape with many different "paths" (mechanisms) that can lead to similar or different results. By mapping these paths, we can better understand, audit, and control how these models think.

Technical Summary: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms

Problem Statement

As large language models (LLMs) are deployed in high-stakes domains, there is an increasing need to audit not just model outputs, but the internal computations that generate them. While mechanistic interpretability offers tools to explain specific behaviors, the dominant approach, circuit analysis, is typically target-conditioned. This means analysts must specify a single prompt and a chosen target completion to identify the causal circuit.

This target-conditioned setup presents two critical limitations:

Generalizability: It is unclear if a mechanism identified for one specific completion generalizes to semantically similar paraphrases or if it reflects a brittle dependence on a specific surface form.
Hidden Heterogeneity: Even for a fixed target, single-target analysis can obscure heterogeneity in internal computation. Semantically similar outputs may rely on distinct internal activation patterns that single-target analysis fails to capture.

Current behavioral evaluations are limited to input-output correlations, while existing mechanistic methods often require manual specification of targets, potentially missing the diverse "continuation modes" a model can produce for a single prompt.

Methodology

The authors propose a framework for distribution-level unsupervised feature discovery. Instead of analyzing a single target, the method clusters a model's sampled continuation distribution by jointly aligning semantic content and sequence-level mechanistic attributions.

The framework operates in three stages:

1. Continuation Sampling

Given an input prefix $x$ , the model samples $N$ distinct continuations $y^{(n)}$ using nucleus sampling. Each continuation is assigned a length-normalized probability $P_n$ , representing its likelihood under the model's distribution. These probabilities are used to weight the contributions of each continuation during clustering.

2. Dual Representation

Each sampled continuation is represented by two complementary views:

Semantic Embedding ( $e_n$ ): Captures the linguistic meaning of the continuation in context, typically derived from a pretrained embedding model (e.g., EmbeddingGemma).
Mechanistic Embedding ( $a_n$ ): Captures how prefix features affect the continuation's logits. This is computed using transcoders (sparse autoencoders trained to approximate MLP layers) to decompose MLP computations into interpretable sparse features. The attribution vector $a_n$ aggregates the direct linear attribution of prefix features to the demeaned logits over the entire continuation span.

3. Joint Clustering via Rate-Distortion

The core innovation is formulating latent discovery as a rate–distortion optimization problem. The goal is to find a clustering that minimizes distortion (reconstruction error) in both semantic and mechanistic views while minimizing the rate (entropy of cluster assignments).

The objective function is:
$L_{RD} = H(C) + \beta_e D(e) + \beta_a D(a)$
Where:

$H(C)$ is the entropy of the cluster prior (Rate).
$D(e)$ and $D(a)$ are probability-weighted distortions in semantic (L2) and attribution (L1) spaces, respectively.
$\beta$ controls the trade-off between compression (fewer clusters) and fidelity (low distortion).
$\gamma$ balances the weight between semantic and mechanistic views ( $\beta_e = \gamma\beta, \beta_a = (1-\gamma)\beta$ ).

Adaptive Operations: Unlike K-means, which fixes $K$ a priori, this framework uses Split and Junk operations to adaptively adjust the number of clusters:

Split: Divides a heterogeneous cluster if the reduction in distortion outweighs the cost of increased entropy (rate).
Junk: Removes spurious clusters that capture noise rather than coherent modes if reassigning their members reduces the total objective.

Key Contributions

Distribution-Level Analysis: The paper introduces the first framework to cluster a model's continuation distribution using both semantic structure and sequence-level mechanistic attribution, moving beyond single-target analysis.
Joint Optimization: It demonstrates that jointly optimizing for semantic coherence and mechanistic consistency reveals continuation modes that single-view baselines (semantic-only or attribution-only K-means) miss.
Causal Validation via Steering: The authors provide interventional evidence that the discovered cluster signatures correspond to actionable mechanistic factors. By steering along cluster-derived directions (using medoids), they show consistent, directional shifts in the model's preference for continuations within that cluster.
Granularity Control: The framework allows users to control the granularity of discovered modes by annealing the $\beta$ parameter, revealing hierarchical structures where finer clusters inherit mechanistic signatures from coarser ones.

Experimental Results

The method was evaluated on AmbigQA (using Qwen3-8B) and MMLU, with additional experiments on Gemma3 and reasoning tasks (GSM8K, MATH500).

Clustering Performance: In rate-matched comparisons, the proposed Rate-Distortion (RD) clustering achieved the lowest mixed distortion ( $D_\gamma$ ) in 94% of settings compared to K-means baselines (KM-S for semantic only, KM-A for attribution only). KM-S preserved semantic coherence but failed on mechanistic consistency, while KM-A showed the opposite failure mode.
Granularity Analysis: Varying $\beta$ revealed a hierarchical structure. For example, a coarse cluster defined by "formatting style" (e.g., solo artist vs. band) split into finer clusters based on specific content details (e.g., listing band members vs. genre descriptions) as $\beta$ increased.
Steering Validation: Steering experiments confirmed causal relationships. RD-derived medoid directions produced strong, monotonic correlations between steering strength ( $\epsilon$ ) and changes in target-cluster logits. In contrast, semantic-only baselines (KM-Sem) showed near-zero correlations, and single-continuation baselines were weaker and less consistent.
Reasoning Tasks: In reasoning benchmarks, the method successfully identified that specific prior reasoning steps causally influence specific target-step clusters, with the strongest effects observed between early source steps and later targets.

Significance and Claims

The paper claims that this approach complements conventional circuit analysis and behavioral evaluation by providing a scalable audit of the mechanisms underlying a model's continuation distribution.

Complementarity: It reduces reliance on hand-chosen targets, surfacing mechanistic hypotheses supported by cluster-level steering. Users can first discover clusters and then trace the specific circuits associated with a selected cluster.
Actionable Mechanisms: The discovered modes are not merely descriptive; they correspond to actionable mechanistic factors that can be manipulated to shift model behavior in predictable ways.
Limitations: The authors acknowledge limitations regarding computational overhead (due to the need for backward passes for attribution) and the lack of a principled framework for assigning human-interpretable labels to clusters. They note that validation under intervention does not guarantee the uniqueness of identified features, requiring further counterfactual inspection.

In summary, the work proposes a shift from analyzing isolated input-output pairs to analyzing the distribution of continuations, offering a more robust lens for understanding the heterogeneous internal computations of large language models.