💻 computer science

Domain-Filtered Knowledge Graphs from Sparse Autoencoder Features

This paper proposes a method to transform sparse autoencoder features from language models into structured, domain-filtered knowledge graphs by constructing a strict concept universe and building aligned co-occurrence and mechanism graphs with automated labeling, thereby converting flat feature inventories into interpretable global maps of model knowledge for auditing reasoning faithfulness.

Original authors: John Winnicki, Abeynaya Gnanasekaran, Eric Darve

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: John Winnicki, Abeynaya Gnanasekaran, Eric Darve

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant library inside a computer brain (a Large Language Model). Inside this library, there are millions of tiny, glowing switches (features) that light up whenever the computer reads a word.

The Problem: A Messy Pile of Switches
Right now, if you ask researchers to look at these switches, they get handed a flat, chaotic list of millions of items. It's like being given a box of 10 million Legos dumped on the floor. Some Legos are red (biology concepts), some are blue (grammar rules), and some are just dust (punctuation).

The red Legos are scattered everywhere, mixed in with the blue ones.
There's no map showing which red Legos belong together to build a "heart" or a "cell."
You can't tell how one Lego connects to another.

The Solution: Building a "Knowledge Map"
This paper proposes a way to clean up that messy pile and build a structured map, specifically for a biology textbook. They do this in three main steps, using some clever tricks:

1. The "Domain Filter" (The Bouncer)

First, they need to get rid of the junk. They use a "bouncer" system to check every single Lego.

The Trick: They compare the biology textbook against other books (like history or geology).
The Logic: If a Lego lights up in every book (like a switch for the word "the" or a comma), it's generic noise. The bouncer kicks it out.
The Result: They keep only the Legos that light up specifically for biology. This creates a "Strict Concept Universe"—a clean box containing only the biology pieces.

2. Building Two Views of the Map

Once they have the clean box of biology Legos, they build two different maps to understand how they fit together.

Map A: The "Co-occurrence" Map (Who hangs out together?)

The Analogy: Imagine walking through a party. You notice that people who talk about "photosynthesis" also tend to talk about "leaves" and "sunlight."
The Paper's Method: They look at the whole textbook and draw lines between concepts that appear in the same sentences, paragraphs, or chapters.
The Result: This map shows the "neighborhoods." It proves that the computer organizes biology topics just like a human textbook does. It shows that "Respiratory System" is a distinct neighborhood, separate from "Immune System," and it even shows the "bridges" where topics overlap.

Map B: The "Transcoder" Map (How does one idea turn into another?)

The Analogy: Imagine a factory assembly line. You put a raw piece of wood (a source concept) on the belt, and it comes out the other side as a finished chair (a target concept).
The Paper's Method: They look at how the computer processes a sentence as it moves from one layer of its brain to the next. They trace the "wiring" to see how a concept in the first layer gets transformed into a concept in the next layer.
The Result: This isn't just a list of neighbors; it's a flow chart. It shows the mechanism of how the computer thinks. For example, it shows how the idea of "oxygen" in one layer gets processed and turned into "breathing" in the next.

3. Adding Labels (Turning a Diagram into a Story)

A map with just lines and dots is still hard to read. The final step is adding "Auto-Relate" labels.

The Analogy: Instead of a line just connecting "Node A" to "Node B," they write a sentence on the line: "Node A leads to Node B" or "Node A is part of Node B."
The Paper's Method: They use the actual sentences from the book to write these labels. If the evidence shows that "cells" and "energy" always appear together in a specific way, the line between them gets the label "provides energy for."
The Result: The messy pile of switches is now a readable, labeled "Knowledge Graph." It's no longer just a list of numbers; it's a story about how the computer understands biology.

The Big Picture

The authors tested this on a biology textbook. They found that:

Structure: The map they built naturally grouped chapters and subchapters together, just like the book's table of contents, without being told to do so.
Clarity: They could take a single messy sentence with thousands of active switches and compress it into a clean, readable diagram showing exactly which concepts were talking to each other.

In short: They took a chaotic, flat list of millions of computer features, filtered out the noise, and organized the rest into a structured, labeled map that shows how the computer internally organizes and processes knowledge.

1. Problem Statement

Sparse Autoencoders (SAEs) have enabled the extraction of millions of interpretable features from Large Language Models (LLMs), moving beyond opaque neurons to monosemantic features. However, current SAE inventories present significant challenges for interpretability:

Flat Inventories: Features are often presented as unstructured lists, making it difficult to understand relationships between them.
Noise and Generality: Inventories contain a mix of domain-specific concepts, generic formatting/punctuation features, and weakly grounded patterns.
Scattered Concepts: Semantically related ideas are often distributed across many neighboring features rather than being localized.
Lack of Structure: There is no inherent mechanism to understand how concepts cluster, separate, or bridge topics across a corpus, nor how they flow through the model's layers during local computation.

The core problem is transforming a flat, noisy SAE inventory into a structured, domain-specific internal knowledge graph that reveals both global organizational structure and local mechanistic pathways.

2. Methodology

The authors propose a pipeline that moves from a raw SAE inventory to a strict concept universe, which is then organized into two aligned graph views: a Co-occurrence Graph (corpus-level) and a Transcoder Mechanism Graph (local-level).

A. Constructing a Strict Concept Universe (Filtering)

Before graph construction, the authors apply a multi-stage filtering process to isolate a relevant domain-specific region ( $V^*$ ) from the raw inventory ( $V$ ):

Contrastive Shortlisting: Features are scored based on three axes by comparing the target corpus (e.g., a biology textbook) against contrast corpora (e.g., history/geology):
- Support: Is the feature meaningfully present in the target?
- Enrichment: Is it more characteristic of the target than the contrasts?
- Localization: Does it concentrate in coherent chapters/subchapters rather than diffusing everywhere?
Packet-Level Adjudication: Surviving features are packaged into "evidence packets" containing descriptions, representative windows, and contrast data. A staged adjudicator verifies if the concept is visible, belongs to the target domain, and is distinctive compared to contrasts.
Result: A strict universe $V^*$ containing only high-confidence, domain-relevant features.

B. Graph Construction

Two complementary graph views are built over $V^*$ :

Multi-Granular Co-occurrence Graph (Corpus Structure):
- Goal: Reveal how concepts cluster and bridge topics across the entire corpus.
- Mechanism: Features are scored for presence in sentences based on activation thresholds. This presence is deterministically lifted to paragraphs, subchapters, and chapters.
- Edges: Constructed using Jaccard-normalized co-occurrence counts ( $J_{ab}$ ) between features. This creates a same-layer graph showing which concepts recur together at different scales.
Transcoder Mechanism Graph (Local Cross-Layer Flow):
- Goal: Explain how source-layer concepts transform into target-layer concepts on specific inputs.
- Mechanism: Uses a Transcoder (a sparse autoencoder trained to map the residual stream of Layer $L$ to Layer $L+1$ ) alongside SAE dictionaries for both layers.
- Edges: Directed edges represent readable pathways where a source feature activates a sparse latent, which in turn writes to a target feature.
- Static vs. Dynamic:
  - Static: A library of possible pathways based on summary statistics.
  - Dynamic: A sentence-conditioned graph showing which pathways are actually active for a specific input.

C. Hierarchy-Aware Compression & Edge Labeling

Abstraction Hierarchy: To handle density, features are organized into a semantic tree (abstraction tree) using mutual-kNN clustering on feature descriptions.
Compression: Dense local mechanism graphs are compressed by collapsing coherent sub-trees into "supernodes" while preserving critical active edges (blocking edges) to maintain fidelity.
Edge Labeling (Auto-Relate): Instead of unlabeled weighted edges, the system generates readable relation phrases.
- Co-occurrence edges: Labeled using joint and contrastive evidence sentences (e.g., distinguishing "Yellowstone" from generic "yellow").
- Mechanism edges: Labeled using the strongest sentence-level evidence and latent hints to describe the transformation (e.g., "Source concept supports Target concept via Latent X").

3. Key Contributions

Multi-Stage Contrastive Filtering: A novel process to extract a strict, domain-specific concept universe from massive SAE inventories, removing generic noise.
Dual-View Knowledge Graphs: The construction of two aligned graphs:
- A Co-occurrence Graph for global, multi-granular corpus structure.
- A Transcoder Mechanism Graph for local, directed cross-layer concept flow.
Hierarchical Compression: A method to compress dense mechanism graphs into readable "mechanism atlases" using a semantic hierarchy, balancing detail with readability.
Evidence-Backed Edge Semantics: An automated labeling system that converts statistical edges into readable, evidence-supported knowledge graph relations.

4. Results

The framework was evaluated on a biology textbook corpus using Gemma Scope SAEs (Layers 20-21).

Structure Recovery: The graphs successfully recovered the textbook's human-authored organization.
- Global Scale: Distinct "basins" of activation were observed for different chapters (e.g., Inheritance vs. Biotechnology) on shared coordinates.
- Local Scale: Finer subchapter structures were preserved within chapter basins.
- Metrics: Quantitative analysis showed high alignment between graph partitions and textbook chapters/subchapters, with sentence and paragraph aggregation yielding the strongest structural recovery.
Readability: The pipeline successfully transformed diffuse sentence-level activity (thousands of features) into compact, readable mechanism atlases.
Bridge Concepts: The graphs revealed concepts that bridge neighboring topics, which are often missed in flat lists.
Mechanism Explanation: The transcoder graphs provided directed, interpretable views of how specific concepts (e.g., "respiratory system") are processed and transformed across layers, with edges labeled by natural language relations.

5. Significance

This work represents a paradigm shift in mechanistic interpretability:

From Inventory to Map: It reframes SAE features not as a flat list of "what the model knows," but as an internal knowledge graph representing "how the model organizes and uses that knowledge."
Faithfulness Audit: By creating a structured, evidence-backed internal map, the method enables future audits of whether a model's generated reasoning (Chain-of-Thought) faithfully reflects its internal activation structure.
Scalability: The approach demonstrates that even with millions of features, it is possible to construct coherent, domain-specific views of model internals without manual curation of every feature.
Interactive Exploration: The authors provide an interactive browser artifact, allowing researchers to navigate the learned graph structure, trace bridge regions, and inspect sentence-level mechanisms dynamically.

In summary, the paper provides a robust framework for converting raw, high-dimensional SAE activations into structured, human-readable knowledge graphs that capture both the global topology of model knowledge and the local mechanics of reasoning.