💬 NLP

Navigating the Concept Space of Language Models

This paper introduces Concept Explorer, a scalable interactive system that uses hierarchical neighborhood embeddings to organize and enable progressive, multi-resolution navigation of sparse autoencoder features for discovering and analyzing human-interpretable concepts in large language models.

Original authors: Wilson E. Marcílio-Jr, Danilo M. Eler

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wilson E. Marcílio-Jr, Danilo M. Eler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library containing millions of books. But instead of words, these books are written in a secret code that only a super-smart robot (a Large Language Model) understands. Recently, scientists built a special tool called a Sparse Autoencoder (SAE) that acts like a translator. It takes the robot's secret code and breaks it down into thousands of tiny, individual "ideas" or "concepts."

For example, one tiny idea might be "the word 'click' in a computer manual," while another might be "the feeling of sadness in a poem."

The Problem: The Library is Too Big
The trouble is, the translator found millions of these tiny ideas. Trying to read them one by one to understand what they mean is like trying to find a specific needle in a haystack by looking at every single piece of hay individually. It's impossible. The old way of doing this was to pick a few random ideas, ask a computer to guess what they mean, and hope you find something interesting. It's slow, boring, and you miss a lot of the big picture.

The Solution: Concept Explorer
This paper introduces a new tool called Concept Explorer. Think of it as a smart, interactive map for this giant library of ideas.

Here is how it works, using a few simple analogies:

1. The "Zoomable" Map (Hierarchical Navigation)

Imagine you are looking at a map of the world on your phone.

At first, you see the whole world: You see big continents like "Science," "Programming," and "Emotions." This is the coarse view.
Then, you zoom in: You click on "Programming," and suddenly you see countries like "Python," "Java," and "HTML."
Zoom in again: You click on "Java," and you see specific neighborhoods like "Error Messages," "Loops," and "Variables."

Concept Explorer does exactly this. It organizes the millions of ideas into a hierarchy. You don't have to look at every single idea at once. You start with the big groups and "drill down" until you find the specific, tiny idea you are looking for.

2. The "Neighborhood" System (Preserving Relationships)

In the old way, ideas were just a messy pile. In Concept Explorer, ideas that are similar are neighbors.

If an idea is about "apples," it will be physically close to ideas about "oranges" and "fruit."
If an idea is about "coding," it will be far away from ideas about "cooking."

This helps you see families of ideas. You can instantly see, "Oh, there's a whole cluster of ideas related to punctuation marks," or "Here is a tiny, isolated island of ideas about a very specific type of error message."

3. Finding the "Rare Gems"

Sometimes, the most interesting things are the weird, rare ones that don't fit into big groups.

Imagine you are looking for a specific, rare bird in a forest.
Concept Explorer highlights the "lonely" spots on the map. If an idea is far away from all the other groups, the tool flags it. This helps researchers find unique, strange, or very specific concepts that would have been missed in a messy pile of data.

Real-World Example from the Paper

The researchers tested this on a model called SmolLM2.

They found a big "neighborhood" dedicated to punctuation.
By zooming in, they discovered a tiny, specific "house" in that neighborhood that only lights up when the model sees dashes or list markers (like --- or *).
They also found a cluster of ideas related to programming symbols (like << or >>), separating them from regular words.

Why This Matters

Before this tool, exploring these millions of ideas was like trying to find your way through a dark maze with a flashlight that only shines on one spot at a time. Concept Explorer turns on the lights, draws a map, and lets you fly over the maze to see the whole structure, find the hidden paths, and discover the rare treasures hidden in the corners.

It doesn't teach the robot new things; it just gives us a much better way to understand what the robot has already learned.

1. Problem Statement

The paper addresses the scalability bottleneck in Mechanistic Interpretability, specifically regarding Sparse Autoencoders (SAEs) trained on Large Language Models (LLMs).

Context: SAEs decompose LLM activation spaces into hundreds of thousands (or millions) of sparse, human-interpretable features.
The Challenge: While SAE training has scaled up, the analysis and discovery of these features have not. Current workflows rely on manual inspection of top-activating examples, semantic search for specific concepts, or heuristic ranking (e.g., by activation frequency).
Limitations: These existing methods are:
- Local and Query-driven: They require the user to know what they are looking for beforehand.
- Inefficient for Global Reasoning: They fail to reveal relationships between large groups of features or identify rare, isolated concepts.
- Visually Cluttered: Single-level embeddings (like neuropedia.org) become unreadable at scale due to visual noise.
Goal: To create a scalable, interactive system that allows for open-ended discovery, hierarchical navigation, and relationship analysis across millions of SAE features.

2. Methodology: Concept Explorer

The authors introduce Concept Explorer, a system designed to organize and visualize SAE feature explanations using hierarchical neighborhood embeddings.

Core Components

Feature Explanation Generation:
- Instead of relying solely on raw activation vectors, the system generates textual explanations for each feature.
- For a given feature, the top- $k$ activating contexts are collected.
- An LLM (GPT-5-mini in the study) generates a concise natural language description (e.g., "This neuron captures...") for each feature.
Hierarchical Embedding (HUMAP):
- The system uses HUMAP (Hierarchical Uniform Manifold Approximation and Projection) to embed the textual explanations of the features.
- Mechanism: HUMAP constructs a hierarchy of subsets based on "landmarks." It applies UMAP repeatedly to create a multi-resolution manifold:
  - Coarse Levels: High-level clusters of related concepts.
  - Fine Levels: Granular neighborhoods of specific features.
- Structure Preservation: The hierarchy preserves both local structure (features within a cluster) and global structure (relationships between clusters).
Interactive Interface:
- Multi-level Canvas: Users can navigate from broad concept clusters down to specific feature neighborhoods.
- Drill-down Capability: Users can select a region of interest (a landmark) at a coarse level and "reproject" its region of influence to the next finer level to inspect sub-structures.
- Triage Signals: The interface highlights:
  - Outliers: Points far from dense groups (potential rare concepts).
  - Influence Region Size: Small regions indicate specialized/rare features.
  - Explanation Similarity: Filters for duplicated or highly overlapping concepts.

3. Key Contributions

Concept Explorer System: A novel interactive tool that shifts the paradigm from "query-driven" feature analysis to "exploratory" concept discovery.
Hierarchical Manifold Organization: The application of HUMAP to SAE feature embeddings, enabling a coarse-to-fine navigation strategy that scales to millions of items without visual clutter.
Integration of NLP and Visualization: A pipeline that combines LLM-generated textual explanations, text embeddings (Nomic Embed), and hierarchical dimensionality reduction to make high-dimensional feature spaces human-interpretable.
Shift in Focus: The work does not propose a new SAE architecture but solves the critical downstream problem of organizing and surveying existing SAE outputs.

4. Results and Use Case

The system was evaluated on 1.7 million SAE features extracted from SmolLM2 (a 135M parameter model) using a bottleneck layer (Layer 27).

Global Structure: The HUMAP projection revealed coherent, high-level concept families (e.g., "Science/Biology," "Negative Prefixes," "Programming/Glyphs") that were visually distinct.
Rare Concept Discovery:
- The system successfully identified rare or isolated features that are often missed by global clustering methods due to context sparsity.
- Example: A specific cluster of features related to dashes and list-boundary markers was discovered. While frequent enough to be distributed, their sparse context made them hard to find via standard heuristics.
Fine-Grained Substructure:
- By drilling down into a "Marker/Glyph" cluster, the system revealed distinct sub-clusters:
  - Programming tokens and glyphs.
  - Placeholder markers (e.g., <<>>).
  - Java-style annotations (e.g., @Override).
  - Enumeration separators.
Relationship Analysis: The hierarchical layout allowed researchers to visualize how different concept families interact and overlap, answering questions about feature influence and connectivity.

5. Significance

Scalability: Concept Explorer demonstrates that it is possible to manage and explore millions of learned features, a task previously infeasible with manual inspection or flat visualizations.
Open-Ended Discovery: It enables researchers to find "unknown unknowns"—concepts they did not know to search for—by exploring the topology of the feature space rather than querying specific keywords.
Interpretability Pipeline: It provides a principled mechanism for organizing the "concept space" of LLMs, facilitating better understanding of model internals, potential safety risks (rare features), and functional decomposition.
Future Work: The authors suggest that hierarchical manifold-based organization is a necessary abstraction for the next generation of interpretability tools as SAEs continue to scale.

In summary, Concept Explorer bridges the gap between the massive scale of modern SAEs and human cognitive limits, transforming feature analysis from a manual, hypothesis-testing task into a scalable, exploratory visual journey.