💬 NLP

Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling

This paper introduces Supervised Multi-Dimensional Scaling (SMDS), a model-agnostic method that reveals how language models encode concepts like temporal reasoning into distinct, stable, and dynamic geometric manifolds (such as circles, lines, and clusters), thereby supporting a framework of entity-based reasoning through structured latent representations.

Original authors: Federico Tiblias, Irina Bigoulaeva, Jingcheng Niu, Simone Balloccu, Iryna Gurevych

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Federico Tiblias, Irina Bigoulaeva, Jingcheng Niu, Simone Balloccu, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like a giant, super-smart library. Inside this library, every concept the model knows—like "Monday," "summer," or "Paris"—is stored in a specific spot in its "brain" (its internal math space).

For a long time, scientists thought these concepts were just scattered randomly or arranged in simple straight lines. But this new paper, "Hypothesis-Driven Feature Manifold Analysis," suggests the library is actually organized like a giant, multi-dimensional playground with slides, merry-go-rounds, and clusters of swings.

Here is the breakdown of what the researchers did and found, using simple analogies.

1. The Problem: Trying to Map a Mystery City

Imagine you are dropped into a strange city with no map. You want to know how the streets are laid out.

Old Methods: Previous tools were like having a compass that only points North. They could tell you if things were "up" or "down" (linear), but they couldn't tell you if the streets formed a circle (like a roundabout) or a cluster (like a neighborhood). They were too rigid.
The New Tool (SMDS): The authors invented a new tool called Supervised Multi-Dimensional Scaling (SMDS). Think of this as a smart GPS that can test different map theories.
- You tell the GPS: "I think the streets form a circle."
- The GPS checks the data and says, "Yes, the streets actually do form a perfect circle, and here is the proof."
- You can then say, "Okay, what if they form a straight line?" and the GPS checks that too. It lets you compare different shapes to see which one fits the data best.

2. The Experiment: The Time Travel Test

To test their new GPS, the researchers asked the AI models simple questions about time, like:

"Alice was born in May. Bob was born in March. Who is older?"
"Kevin waters plants every day. Who waters them more often?"

They looked at the AI's "brain" while it was thinking about these answers to see how it organized the concepts of days, months, and years.

3. The Big Discoveries

A. The Playground Shapes (Finding the Geometry)

The researchers found that the AI doesn't just store time randomly; it builds specific shapes:

The Merry-Go-Round (Circular): Concepts like "Days of the Week" or "Months" are arranged in a perfect circle. In the AI's mind, December is right next to January, just like on a real calendar. If you walk far enough in one direction, you loop back to the start.
The Slide (Linear): Concepts like "Years" or "Durations" are arranged in a straight line. 2020 is further down the slide than 2010.
The Neighborhood (Clusters): Concepts like "Seasons" (Spring, Summer, etc.) are grouped into tight little clusters, like houses in a specific neighborhood.

The Cool Part: These shapes are the same whether you use a small AI or a giant AI. It's like how every human brain organizes the concept of "up" and "down" the same way.

B. The Shapeshifter (Adapting to the Task)

The most surprising finding is that the AI's brain is dynamic. It changes the shape of its playground depending on what question you ask.

Scenario: You ask, "Who was born in the summer?"
- The AI instantly reshapes its "Date" playground. Instead of a circle, it turns the dates into clusters (Summer dates group together, Winter dates group together).
Scenario: You ask, "Who was born first?"
- The AI reshapes the playground into a straight line so it can easily compare who is earlier.

It's like a Lego set: The AI has all the same blocks (the dates), but it builds a different structure depending on the job it needs to do.

C. The Proof: Breaking the Toy

To prove these shapes are actually used for thinking (and not just a coincidence), the researchers did a "stress test."

They took the AI's brain and added "static noise" (like radio interference) specifically to the circular part of the playground where "Months" live.
Result: The AI immediately got terrible at answering questions about months.
Control: When they added noise to a random, useless part of the brain, the AI didn't care.

This proves the AI is actively using these shapes to solve problems. It's like proving a car needs wheels by taking them off; the car stops working.

4. Why This Matters

This paper changes how we understand AI.

Before: We thought AI was a "black box" that just guessed answers based on patterns.
Now: We know AI builds structured, geometric maps of the world. It organizes knowledge into circles, lines, and clusters, and it physically moves these shapes around to solve logic puzzles.

The Takeaway

The authors have given us a new way to look inside the AI's brain. They found that AI doesn't just "know" facts; it architects them. It builds geometric playgrounds (manifolds) that are perfectly shaped for the job at hand.

If you want to make AI smarter or less biased, you don't just need to feed it more data; you might need to understand the geometry of its thoughts. If the "map" is broken, the AI gets lost. If the map is a perfect circle, the AI can navigate time effortlessly.

In short: The AI isn't just a calculator; it's an architect building 3D shapes out of math to help it think.

1. Problem Statement

Recent research in mechanistic interpretability suggests that Large Language Models (LLMs) encode concepts as structured directions or manifolds in their latent space (e.g., linear for numbers, circular for days of the week). However, existing methods for analyzing these structures face significant limitations:

Fixed Structural Assumptions: Common dimensionality reduction techniques like PCA, LDA, and PLS enforce specific geometric assumptions (e.g., maximizing variance or linear separability), making it difficult to compare competing geometric hypotheses (e.g., is a feature linear, circular, or clustered?).
Lack of Quantitative Comparison: There is no unified, quantitative metric to evaluate which geometric hypothesis best fits the model's internal representations.
Unclear Functional Role: It remains unclear how LLMs utilize these manifolds during reasoning, whether they adapt to task contexts, or if they are merely artifacts of representation.

The authors aim to address these gaps by introducing a method to systematically test and compare arbitrary geometric hypotheses about feature manifolds and to determine their causal role in reasoning.

2. Methodology: Supervised Multi-Dimensional Scaling (SMDS)

The core contribution is Supervised Multi-Dimensional Scaling (SMDS), a model-agnostic method that reframes manifold identification as a model selection problem.

Concept: Unlike unsupervised MDS, SMDS incorporates supervision. It assumes that model activations ( $X$ ) lie on a manifold parametrized by known labels ( $y$ ).
Mechanism:
1. Hypothesis Definition: The user defines a target geometry (e.g., circle, line, cluster) by specifying a distance function $d(y_i, y_j)$ between labels. For example, a circular manifold uses chord distance on a unit circle.
2. Projection Learning: SMDS learns a linear projection $W$ that maps high-dimensional hidden states $X$ to a low-dimensional space such that the Euclidean distances in the projected space best match the ideal distances defined by the hypothesis.
3. Optimization: It minimizes a stress loss function:
  $L = \sum_{i<j} (\|W(x_i - x_j)\|^2 - d(y_i, y_j)^2)^2$
4. Evaluation: The quality of the fit is measured using a normalized stress metric ( $S$ ) on a held-out test set. Lower stress indicates a better fit between the model's internal representation and the hypothesized geometry.
Distance Functions: The authors define a library of distance functions to detect various topologies:
- Linear: Monotonic progression.
- Cyclical: Ring structures (e.g., days of the week).
- Categorical: Discrete clusters (e.g., polytope vertices).
- Logarithmic: Compressed progressions (e.g., numbers).

3. Experimental Setup

Task Domain: Temporal reasoning (Short-form QA), chosen for its diversity (dates, durations, periodicity) and known difficulties for LLMs.
Datasets: Five synthetic datasets (Date, Duration, Notable, Periodic, Time of Day) and three variants (Season, Temperature, Phase) designed to isolate specific temporal reasoning skills.
Models: A diverse set of LLMs including Qwen2.5-3B, Gemma-2-2B, and Llama-3.2-3B (and larger variants 8B/70B for Llama), covering both base and instruction-tuned versions.
Analysis Sites: Activations were extracted from three token positions:
1. End of the temporal expression ($te$).
2. End of the prompt ($lp$).
3. The generated answer token ( $a$ ).
Validation: Extensive statistical testing (Friedman test, Nemenyi post-hoc, bootstrapping) was used to ensure the identified manifolds were statistically significant and not overfitting artifacts.

4. Key Findings

The study yields three major findings (F1–F3):

F1: Universal and Intuitive Manifold Structures

Temporal entities consistently form feature manifolds with intuitive geometries across different model architectures and sizes.
Circular: Dates and times of day form rings.
Linear/Log-Linear: Durations and intervals often show logarithmic compression (similar to human perception/Weber-Fechner law).
Clustered: Categorical distinctions (e.g., seasons) form distinct clusters.
Consistency: These structures are robust across model families (Llama, Qwen, Gemma) and scales (3B to 70B), suggesting a "Platonic" convergence in how LLMs encode structured knowledge.

F2: Dynamic Adaptation to Context

Feature manifolds are not static; they dynamically reshape based on the task and prompt.
Example: In a "Date" task, the manifold is circular. However, if the prompt changes to ask about "Seasons" (a classification task), the manifold transforms into a cluster structure.
Information Flow: The structure of the manifold at the answer token ( $a$ ) often mirrors the structure at the input token ($te$), but with increased stress (noise), suggesting that structured representations are propagated and preserved through the network layers during reasoning.

F3: Causal Role in Reasoning

Intervention: Perturbing the specific low-dimensional subspaces identified by SMDS (by adding noise) significantly degrades reasoning performance. In contrast, adding noise to random subspaces has negligible effect.
Correlation: There is a significant positive correlation between the "quality" of the manifold (low stress) and the model's downstream reasoning accuracy. Models that form better-organized manifolds perform better.
Conclusion: LLMs do not just have these structures; they actively use them to perform reasoning.

5. Generalization

The authors extended SMDS beyond temporal reasoning:

Spatial Reasoning: Applied to a geography task (identifying the closest city). The results showed that LLMs represent geographic locations on a spherical manifold, aligning with the true geometry of the Earth, outperforming planar or cylindrical hypotheses.
Multidimensional Features: Successfully identified 2D manifolds combining duration and start dates, proving the method works for higher-dimensional feature spaces.

6. Significance and Contributions

Methodological Innovation: SMDS provides a rigorous, hypothesis-driven framework for interpretability, moving beyond "discovery" to "testing" of specific geometric hypotheses. It bridges the gap between theoretical manifold hypotheses and empirical verification.
Mechanistic Insight: The paper provides strong evidence for an entity-based reasoning pipeline in LLMs:
1. Represent entity properties on a coherent manifold.
2. Transform the manifold based on task context.
3. Select outputs based on the transformed representation.
Implications for Control: Understanding that reasoning relies on specific geometric subspaces opens new avenues for model steering, alignment, and debugging. If reasoning fails, it may be due to the degradation of these manifolds, which can be targeted for intervention.
Robustness: The findings challenge the notion that LLM reasoning is purely statistical pattern matching, suggesting instead that they rely on structured, geometrically organized internal representations that are consistent across scales and architectures.

7. Limitations

Bias: The datasets rely on English-centric cultural norms (Gregorian calendar, Northern Hemisphere seasons), which may bias the manifold structures.
Fuzziness: The method requires precise, numerical labels. It cannot currently analyze fuzzy temporal expressions (e.g., "in the morning") where exact manifold positions are undefined.
Hypothesis Dependency: The effectiveness of SMDS relies on the user proposing the correct set of candidate geometries. If the true manifold structure is not in the hypothesis set, it will not be detected.