🤖 machine learning

CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs

This paper introduces CLT-Forge, an open-source library that enables scalable distributed training and comprehensive interpretability analysis of Cross-Layer Transcoders to address the redundancy and complexity challenges in mechanistic interpretability of Large Language Models.

Original authors: Florent Draye, Abir Harrasse, Vedant Palit, Tung-Yu Wu, Jiarui Liu, Punya Syon Pandey, Roderick Wu, Terry Jingchen Zhang, Zhijing Jin, Bernhard Schölkopf

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Florent Draye, Abir Harrasse, Vedant Palit, Tung-Yu Wu, Jiarui Liu, Punya Syon Pandey, Roderick Wu, Terry Jingchen Zhang, Zhijing Jin, Bernhard Schölkopf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot (a Large Language Model) that writes stories, answers questions, and chats with you. But here's the problem: the robot is a "black box." You can see what goes in (your question) and what comes out (the answer), but you have no idea what's happening inside its brain.

Mechanistic interpretability is the field trying to open that box and see the gears turning.

This paper introduces a new tool called CLT-Forge. Think of it as a universal translator and a high-definition microscope rolled into one, designed to help researchers understand how these AI brains actually work.

Here is the breakdown using simple analogies:

1. The Problem: The "Noisy Library"

Imagine the AI's brain is a massive library with 100 floors (layers). In the past, researchers tried to map out every single book (feature) on every floor.

The Issue: They found that the same book (concept) appeared on Floor 1, Floor 5, and Floor 90, but with slightly different titles. This created a map with thousands of duplicate entries. It was like trying to navigate a city where "Main Street" is named "Main St," "Main Road," and "Main Avenue" on every single block. The map was huge, messy, and impossible to read.

2. The Solution: Cross-Layer Transcoders (CLTs)

The authors used a smarter way to map the brain called Cross-Layer Transcoders.

The Analogy: Instead of listing every book on every floor separately, they created a centralized catalog. They realized that the concept of "Cat" is the same whether it's on Floor 1 or Floor 50. So, they linked them all together under one single "Cat" entry.
The Result: This shrinks the map from a messy 100-page document down to a neat, compact 10-page guide. It removes the redundancy and makes the AI's logic much clearer.

3. The New Tool: CLT-Forge

While this "smart catalog" idea existed, it was incredibly hard to build. It required supercomputers, massive amounts of memory, and a lot of manual coding. It was like trying to build a skyscraper with a hammer and a toothbrush.

CLT-Forge is the construction kit that makes building these maps easy and fast. It's an open-source software library (a toolbox) that does three main things:

A. The "Smart Warehouse" (Scalable Training)

Training these models requires remembering billions of tiny details (activations) from the AI's brain.

The Old Way: You'd need a warehouse the size of a city to store all this data.
The CLT-Forge Way: It uses compression and quantization. Imagine taking a 4K movie and shrinking it down to a high-quality MP3 file without losing the plot. CLT-Forge shrinks the data storage needs by 7 to 12 times, allowing researchers to train these models on standard computer clusters instead of needing a supercomputer the size of a building.

B. The "Auto-Translator" (Automated Interpretability)

Once the model is trained, you have a list of abstract "features" (like a code for "sadness" or "math"). You need to know what they actually mean.

The Old Way: A researcher would have to manually look at thousands of examples, guess what a feature means, and write a description.
The CLT-Forge Way: It runs an automated pipeline. It scans the data, finds the best examples for each feature, and even uses another AI to write a plain-English explanation for you. It's like having a team of interns who instantly summarize what every single gear in the machine does.

C. The "Interactive Map" (Visualization)

Finally, you need to see how these features talk to each other.

The Old Way: The data was just a spreadsheet or a static image.
The CLT-Forge Way: It provides a live, interactive dashboard (like a video game map). You can click on a feature, see which other features it influences, group them into "teams" (clusters), and even run experiments (interventions) to see what happens if you change one part of the brain.

Why Does This Matter?

Before this, only a few giant tech companies (like Anthropic) had the resources to do this kind of deep AI analysis. They kept their tools secret.

CLT-Forge is like releasing the blueprints and the construction tools to the public. It allows any researcher, student, or curious mind to:

Train these complex models efficiently.
Understand exactly how the AI thinks.
Fix or improve the AI by seeing its internal logic.

In short, CLT-Forge turns the messy, overwhelming brain of an AI into a clean, readable, and interactive map, making it possible for everyone to understand the "magic" behind the machine.

1. Problem Statement

Mechanistic interpretability aims to understand how Large Language Models (LLMs) process information, often relying on the linear representation hypothesis. Recent methods use Transcoders to approximate MLP layers with sparse, interpretable features, generating Feature Attribution Graphs that map causal influences between features.

However, two major bottlenecks limit the scalability and practicality of this approach:

Redundancy and Size: Standard transcoders often produce massive, redundant attribution graphs because similar features emerge independently across different layers, creating duplicated nodes.
Computational and Infrastructure Barriers: While Cross-Layer Transcoders (CLTs) solve the redundancy issue by sharing features across layers, they introduce a quadratic increase in parameters ( $O(L^2)$ ), making them computationally prohibitive to train at scale.
Fragmented Tooling: Existing open-source frameworks lack end-to-end support for CLTs. They often miss critical components like efficient distributed training (model sharding), memory-efficient activation caching (quantization/compression), and unified pipelines for automated interpretability (AutoInterp) and visualization. Most existing tools are fragmented, requiring practitioners to stitch together disparate libraries.

2. Methodology: The CLT-Forge Framework

The authors introduce CLT-Forge, a unified, open-source library designed to enable scalable training and post-hoc analysis of CLTs. The framework integrates four core technical components:

A. Scalable Training Infrastructure

Feature-Wise Sharding: To handle the massive parameter counts of CLTs (e.g., ~27B parameters for a 1B model), the library implements feature-wise sharding across GPUs. Unlike standard Data Parallelism (DDP) or Fully Sharded Data Parallel (FSDP), this splits the feature dimension, allowing each worker to handle a subset of features while aggregating outputs. This significantly improves memory efficiency.
Compressed Activation Caching: Training requires storing activations for hundreds of millions of tokens. CLT-Forge implements symmetric per-layer quantization (supporting int8, int4, int2) combined with zstd entropy coding. This reduces storage requirements by 7–12× (e.g., from 20TB to ~4TB for a LLaMA 1B model) with negligible impact on reconstruction quality (<3% drop).
Low-Rank Fine-Tuning: To adapt pretrained CLTs without expensive retraining, the library supports low-rank fine-tuning (LoRA), enabling efficient adaptation to new datasets.

B. Automated Interpretability (AutoInterp) Pipeline

The library provides a parallelized workflow to interpret trained features:

Top-K Tracking: It processes large token streams on the GPU, incrementally tracking the top- $K$ activating sequences for each feature.
Unified Generation: It aggregates statistics (top tokens, sequences) and automatically generates natural language explanations using an LLM.
Efficient Storage: Instead of writing millions of individual files, results are stored in a per-worker database and merged into a single unified store, drastically reducing I/O overhead.

C. Attribution Graph Computation

Circuit-Tracer Integration: The framework natively integrates with the Circuit-Tracer library to compute attribution graphs.
Graph Pruning: It calculates attribution scores based on the Jacobian of the model's forward pass (with frozen nonlinearities) to identify causal pathways between features, allowing for efficient pruning of irrelevant edges to create compact graphs.

D. Interactive Visualization Interface

Dash-Based UI: A Python-based Dash interface allows researchers to visualize attribution graphs, explore feature clusters, and perform interventions (e.g., replacing feature activations).
Extensibility: Unlike existing tools (e.g., Neuronpedia) which can be difficult to extend, CLT-Forge is designed as a flexible research tool for rapid iteration and hypothesis testing.

3. Key Contributions

First Unified Open-Source Library: CLT-Forge is the first framework to provide an end-to-end pipeline for CLTs, covering training, activation caching, AutoInterp, attribution analysis, and visualization.
Scalable Training Architecture: It introduces efficient GPU sharding strategies and quantized activation caching, enabling the training of CLTs on models like LLaMA 1B (with ~1.5M features) that were previously infeasible for open-source researchers.
Integrated Interpretability Workflow: It unifies fragmented tools into a single workflow, automating the retrieval of activating data, statistical analysis, and explanation generation.
Practical Visualization: It offers a simplified, extensible interface for exploring feature interactions and performing interventions, bridging the gap between complex graph data and human understanding.

4. Results and Evaluation

Performance Metrics: On GPT-2, the implementation matches prior state-of-the-art results, achieving:
- ~0.8 Explained Variance at total $L_0 = 100$ .
- ~0.8 Replacement Score.
- ~0.95 Graph Completeness on standard prompts (e.g., antonym completion).
Efficiency: Feature-wise sharding demonstrated significantly better memory efficiency compared to DDP, allowing for larger expansion factors (48x) and micro-batch sizes (512) on 8x80GB GPUs.
Storage Reduction: The quantization strategy successfully reduced storage footprints by up to 12× with minimal degradation in model performance.
Qualitative Validation: The pipeline successfully generated interpretable graphs (e.g., for the sentence "The opposite of 'large' is") that align with known mechanistic circuits, as verified in the paper and associated studies.

5. Significance

CLT-Forge represents a critical step toward democratizing mechanistic interpretability at scale.

Lowering Barriers: By providing a unified, open-source alternative to proprietary frameworks (like those used by Anthropic), it enables academic researchers to study CLTs without needing massive proprietary infrastructure.
Reproducibility: The library standardizes the training and analysis pipeline, facilitating reproducible research in understanding LLM internals.
Future Directions: The framework sets the stage for future work in attention attribution (currently frozen in the graph computation), more efficient architectures, and optimizing objectives directly for replacement scores rather than just reconstruction error.

In summary, CLT-Forge transforms Cross-Layer Transcoders from a theoretically promising but computationally prohibitive concept into a practical, scalable tool for dissecting the internal mechanisms of large language models.