💬 NLP

Inside the LLM Word Factory

This paper uses activation patching to reveal that LLMs perform a two-stage detokenization process—where attention transmits token-specific signals and MLPs compose them with local embeddings—across varying depths depending on positional encoding, enabling a highly accurate probe for detokenization success based on early-layer activations.

Original authors: Benzi Busigin, Yuval Pinter

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benzi Busigin, Yuval Pinter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, super-smart factory called the LLM Word Factory. Its job is to take sentences and understand them so it can write new ones. But there's a catch: the factory doesn't speak "human words" like "table" or "error." Instead, it only understands subword fragments, like little puzzle pieces: _ta and ble.

When you type the word "table," the factory might see it as one piece (_table) or split it into two (_ta + ble). If it's split, the factory has a problem: it needs to glue those pieces back together in its brain to understand the whole concept of a table before it can do anything useful. This gluing process is called detokenization.

If the factory fails to glue the pieces, it gets confused. It might think "table" is just "ta" and "ble" separately, leading to weird mistakes.

This paper is like a team of detectives using a special "time-travel microscope" (called activation patching) to peek inside the factory and see exactly how and where this gluing happens. Here is what they found:

1. The Two-Step Assembly Line

The detectives discovered that the factory doesn't glue the pieces together all at once. It happens in a very specific two-step dance right at the beginning of the assembly line (in the first few layers of the computer model):

Step 1: The Messenger (Attention)
Imagine the first puzzle piece (_ta) has a tiny note attached to it. A special messenger (called the Attention mechanism) picks up this note and flies it over to the second piece (_ble).
- The Catch: The note isn't a full picture of the word. It's just a tiny, weak "nudge" or a hint saying, "Hey, I'm part of a specific word." It's like a postal worker dropping off a single letter that says "Package Incoming."
Step 2: The Builder (MLP)
Once the second piece (_ble) receives that tiny note, a builder (called the MLP) steps in. The builder takes the local piece (_ble) and combines it with the tiny note from the messenger.
- The Magic: The builder doesn't just snap them together; it smoothly blends them to create the full, unified concept of "table."

2. How Long Does the Assembly Line Need to Be?

The paper found that this two-step dance happens almost immediately, usually within the first 1 to 10 layers of the factory.

The "RoPE" Factory: Some factories use a specific type of address system (called RoPE). In these, the gluing happens super fast, often in just the first 1 or 2 layers. It's like a factory with a super-efficient conveyor belt where the pieces meet almost instantly.
The "Learned" Factory: Other factories use a different address system (Learned Absolute). Here, the gluing takes longer, stretching out over 5 to 10 layers. It's like a slower conveyor belt where the pieces have to travel a bit further before they get glued.

3. What About Longer Words?

What if the word is split into three or four pieces (like _an + ti + ci + pa + tion)?
The paper found that the factory uses a relay race strategy.

The first piece passes a baton to the second.
The second passes it to the third.
The third passes it to the final piece.
Each intermediate piece acts as a relay station, passing the "nudge" down the line until it reaches the end, where the final gluing happens. The longer the word, the more relay stations are needed, but the process remains the same.

4. Can We Predict Failure?

The most exciting discovery is that the factory leaves a "smoke signal" very early in the process.

If the gluing is going to work, the early layers look a certain way.
If the gluing is going to fail, they look different.

The researchers built a simple detector (a probe) that can look at just the very first few layers of the factory and predict with 94% to 97% accuracy whether the word will be understood correctly or not. It's like a quality control inspector who can tell if a car will break down just by looking at the engine before the car is even finished being built.

Summary

In simple terms, this paper explains that when AI models try to understand split-up words, they use a two-step process right at the start:

Attention passes a tiny hint from the first piece to the last.
The MLP uses that hint to finish building the full word.

This process is fast, happens early, and the model leaves a clear "signature" in the early layers that tells us if it's going to succeed or fail. The speed of this process depends entirely on how the factory is built (specifically, how it handles positions), not on how big or smart the factory is.

Technical Summary: Inside the LLM Word Factory

Problem Statement

Transformer language models rely on subword tokenization (e.g., splitting "table" into _ta and ble), creating a systematic mismatch between input representation and the model's need to operate on unified semantic concepts. This necessitates detokenization: the internal process where the model reconstructs a coherent word-level representation from fragmented subword tokens. While prior work established that detokenization occurs in early-to-middle layers and involves both attention and MLP components, the exact mechanics, signal transmission pathways, and the factors determining the depth of this process remained unclear. Furthermore, it was unknown whether the success or failure of this reconstruction could be predicted from early-layer activations.

Methodology

The authors employ activation patching combined with a controlled paired-dataset design to isolate causal contributions of model components.

Controlled Pairs: The study utilizes words that can be tokenized as a single canonical token (e.g., _folder) or artificially split into subwords (e.g., _fo + lder).
- FST (First-Shared-Token) Pairs: Two split words share the first token but differ in the second (e.g., _fo+lder vs. _fo+od). This isolates the contribution of the MLP, as the attention input from the first token is constant.
- LST (Last-Shared-Token) Pairs: Two split words share the last token but differ in the first (e.g., _po+or vs. _er+or). This isolates the contribution of Attention, as the local embedding at the final position is constant.
Success Metric (Canonicity): Defined as the cosine similarity between the residual stream at the last subword position of a split input and the residual stream of the canonical single-token representation at layer $n-2$ (layer 30 in Llama2-7B). High canonicity indicates successful reconstruction.
Activation Patching: The authors patch activations from a "successful" run (high canonicity) into a "failed" run (low canonicity) at specific layers and positions. The degree to which the patch closes the "success–failure gap" identifies the causal role of that component.
Scope: Primary experiments use Llama2-7B, with cross-architecture generalization tested across 12 models from 8 families, varying in width, depth, and positional encoding schemes (RoPE, ALiBi, Learned Absolute).

Key Contributions and Results

1. A Two-Stage Mechanism at Layer 1

The paper identifies a specific, two-stage process for detokenization that initiates in Layer 1:

Stage 1 (Attention): A small subset of attention heads (specifically heads 24, 27, and 28 in Llama2-7B) transmits a weak, token-specific signal from the preceding subword to the final position. This signal is a "directional nudge" weakly aligned with the canonical embedding but is not a generic success flag; it encodes the identity of the first token.
Stage 2 (MLP): The Layer 1 MLP composes this transmitted signal with the local subword embedding. Patching the MLP output accounts for the majority of the reconstruction effect in FST pairs. The operation is continuous rather than discrete; scaling the MLP output linearly scales the canonicity score.

2. Scaling with Token Count

For words split into more than two tokens (up to 6), the mechanism generalizes via sequential relays:

Intermediate positions actively relay information from the first token to the last.
Each intermediate position contributes a "relay window" of 2–3 layers, shifting deeper as the token index increases.
The total depth required grows sublinearly with token count. The transition from 2 to 3 tokens adds ~3 layers, but subsequent tokens add less than one layer each, as intermediate positions provide parallel relay paths.

3. Depth is Governed by Positional Encoding

While the two-stage mechanism (Attention transmits, MLP composes) is universal across 12 models, the depth over which it occurs depends strictly on the positional encoding scheme:

RoPE-based models: Detokenize rapidly, reaching 80% of the success gap within 1–5 layers.
Learned-absolute models: Require 5–10 layers to achieve the same effect, accumulating the signal more gradually.
ALiBi models: Exhibit an intermediate profile, transitioning from concentrated to distributed as token count increases.
The authors hypothesize this is because relative-position encodings (RoPE, ALiBi) allow Layer 1 attention to attend sharply from position 2 to 1 in a single step, whereas learned absolute encodings require multiple layers to recover relative position from absolute coordinates.

4. Early-Layer Predictability

The paper demonstrates that the outcome of detokenization (success vs. failure) is linearly readable from early-layer activations:

A simple class-mean-difference probe fitted on Layer 2 activations achieves 0.94 AUROC on isolated sequences and 0.97 AUROC on natural text (WikiText).
Crucially, a probe direction learned on isolated, contextless sequences transfers effectively to natural text (0.91 AUROC), suggesting the early-layer geometry of the "composition" signal is robust to context accumulation.

Significance and Claims

The paper claims to provide the first mechanistic localization of detokenization, moving beyond correlational probing to causal intervention. The key significance lies in:

Mechanistic Clarity: Defining detokenization not as a vague layer-level phenomenon but as a specific two-stage computation (Attention signal + MLP composition) occurring in the first few layers.
Architectural Insight: Establishing that positional encoding, rather than model scale or width, dictates the computational depth of this fundamental process.
Predictive Utility: Demonstrating that the success of word reconstruction can be predicted with high accuracy using only early-layer activations, even in natural text. This suggests potential for inference-time interventions to flag tokenization-induced failures before generation concludes and offers a new, mechanistic metric for tokenizer evaluation independent of downstream behavioral metrics.

The authors remain modest regarding limitations, noting that their metric (canonicity) relies on an internal model reference rather than external ground truth, and their methodology is restricted to words that can be represented as single tokens, leaving naturally multi-token words (rare words, proper names) as an open question.