💬 NLP

A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability

This paper proposes Tensor Product Representations (TPRs) as a unifying framework that mathematically and empirically demonstrates how diverse language model interpretability methods, such as linear probing and sparse autoencoders, can be derived from a single underlying structure of filler-role bindings.

Original authors: Zhang Enyan, R. Thomas McCoy

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhang Enyan, R. Thomas McCoy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet, humming world of artificial intelligence, researchers are trying to understand how massive computer programs, known as language models, actually think. These systems can write poetry, solve math problems, and hold conversations, but inside their digital brains, information is stored as long lists of numbers. For years, scientists have used various tools to peek inside these black boxes, finding that the models seem to organize information in surprisingly orderly ways. Some tools show that the models can solve puzzles by adding and subtracting these number lists, while others reveal that specific parts of the model light up when they encounter a particular word or idea. Yet, these discoveries have remained isolated, like separate clues found in different rooms of a dark house. The big question has been whether there is a single, underlying structure that explains all these different behaviors, or if the models are just a chaotic jumble of unrelated tricks.

A team of researchers at Yale University has proposed a unifying answer to this puzzle. They suggest that these complex language models are built on a specific architectural principle called Tensor Product Representations. In plain terms, this means the models store information by tightly binding two things together: the content of an idea (like a specific word) and its role in a sentence (like whether it is the subject or the object). Imagine a filing system where every piece of information is stored not just by what it is, but by exactly where it belongs in a structure. The researchers found that this simple, structured way of organizing data can explain a wide range of previously unrelated discoveries. It accounts for why the models can perform mathematical analogies, why simple tests can reveal hidden concepts, and why changing a single part of the model's internal state can alter its behavior in predictable ways.

To test this idea, the team did not just rely on theory; they built a new kind of tool to act as a translator. They created a system that takes the known structure of a sentence—identifying the subject, verb, and object—and converts it into the specific number patterns the models use. They then compared this translated version against the actual internal workings of several different computer models, ranging from small, simple networks to massive, state-of-the-art language systems. The results were strikingly consistent. In tests involving sequences of numbers and structured English sentences, the researchers' structural translation matched the models' internal states with extremely high precision. For the simpler models, the match was nearly perfect, capturing almost all the variation in how the models processed information. Even for the largest, most complex language models, the translation captured the vast majority of the information, suggesting that these giant systems, despite their size, rely on this same fundamental method of binding content to position.

The power of this finding lies in how it connects different methods of investigation. The researchers showed that the mathematical logic behind their structural translation could be used to recreate the results of four major interpretability techniques that scientists use today. First, they explained why language models can perform "additive analogies," a phenomenon where subtracting one concept from another and adding a third yields a fourth related concept. Their work demonstrated that this happens because the models are essentially calculating the difference between specific content-and-role pairings. Second, they showed how "linear probes," which are simple tests used to detect if a model knows a certain fact, are actually just a way of unbinding the role from the content to retrieve the original idea. Third, they explained how "sparse autoencoders," which break down complex signals into simpler parts, are effectively identifying these same content-and-role bindings. Finally, they demonstrated that "activation patching," a technique where researchers swap parts of a model's brain to see how it changes behavior, works because they are directly swapping these specific structural bindings.

In a series of experiments, the team proved that they could use their structural translation to build these four different testing tools from scratch, without needing to train them on the models first. When they used these constructed tools to analyze the models, they performed just as well as the standard, heavily trained tools used by the field. For instance, when they used their method to predict which word would appear next in a sentence, or to identify the subject of a sentence, the accuracy was nearly identical to the best existing methods. In one specific test involving a large language model, the difference between their structural prediction and the standard method was so small it was almost invisible, with a correlation score of nearly one. This suggests that the complex, learned behaviors of these models are not mysterious or accidental, but are direct consequences of this underlying structural rule.

The researchers also explored how well this structure holds up when the models face new situations. They trained their structural translator on a set of sentences and then tested it on sentences containing words and roles it had never seen before. The translator successfully generalized, accurately reconstructing the model's internal state for these novel combinations. This indicates that the models are not simply memorizing specific examples but are using a flexible system to combine new content with known roles. The study did not claim to solve every mystery of artificial intelligence, nor did it suggest that all models are identical. However, it provided strong evidence that a single, coherent framework can explain how these systems represent the world. By showing that diverse interpretability methods all point to the same structural reality, the work moves the field closer to a unified understanding of how neural networks function, turning a collection of isolated observations into a single, coherent picture of the machine's mind.

Technical Summary: A Unifying Perspective on Language Model Representations

Problem Statement

The field of mechanistic interpretability has produced a diverse array of methods for understanding neural network representations, including additive analogies, linear probing, sparse autoencoders (SAEs), and activation patching. While each method yields valuable insights, they currently exist in relative isolation. There is no overarching theoretical framework that explains why these disparate methods work or how they relate to one another. Specifically, the field lacks a precise, closed-form account of the underlying structure of neural representations that can simultaneously explain their compositional nature, their linear decodability, and their susceptibility to causal intervention.

Methodology

The authors propose Tensor Product Representations (TPRs) as a unifying hypothesis for the structure of language model representations. TPRs, originally developed in Vector Symbolic Architectures, encode information by binding "fillers" (content elements) to "roles" (structural positions) via tensor products (outer products).

The core methodology involves three steps:

Approximation: The authors operationalize TPRs using a Tensor Product Encoder (TPE). A TPE is a neural architecture trained to approximate the hidden states of frozen target models (ranging from toy RNNs to large LLMs) given an input encoded as a set of filler-role pairs.
Mathematical Derivation: The authors derive four major interpretability methods directly from the mathematical properties of TPRs (specifically the binding and unbinding operations).
Empirical Construction: Using the trained TPEs, the authors construct instances of additive analogies, linear probes, SAEs, and activation patching interventions. They compare these "constructed" variants against standard, empirically trained versions of these methods.

The experiments utilize two primary domains to ensure ground-truth knowledge of the encoded concepts:

Synthetic Sequences: Tasks involving copying and reversing fixed-length digit sequences (using RNNs, GRUs, and LSTMs).
Structured Sentences: Subject-Verb-Object (SVO) sentences (e.g., "the [subject] will [verb] the [object]") processed by various embedding models and Large Language Models (LLMs) like Qwen3, OLMo, and GPT-OSS.

Key Contributions

1. TPRs as a Precise Approximation

The authors demonstrate that TPRs can approximate the hidden states of diverse neural networks with high fidelity.

Results: On synthetic sequence tasks, TPEs achieved an average substitution accuracy of 99.97% and an $R^2$ of 0.9385. On SVO sentence tasks, TPEs achieved $R^2 > 0.90$ for embedding models and $R^2 > 0.60$ for LLMs (specifically period-token representations).
Significance: This confirms that the representations being analyzed are structurally consistent with linearly transformed TPRs.

2. Unification of Interpretability Methods

The paper mathematically derives four distinct interpretability techniques from the TPR formalism, showing they are not independent phenomena but different views of the same underlying structure:

Additive Analogies: The authors show that the vector difference used in analogies (e.g., $A - B + C \approx D$ ) corresponds exactly to the difference in filler-role bindings. If two sequences differ only in the filler at a specific role, the vector difference isolates that filler's embedding.
Linear Probing: A linear probe is shown to be equivalent to the TPR unbinding operation. Probing for a specific concept (e.g., the subject) is mathematically identical to unbinding the "subject" role from the TPR to recover the filler.
Sparse Autoencoders (SAEs): The authors derive SAE feature detectors directly from TPR unbinding vectors. They show that SAE features can be constructed to detect specific filler-role bindings, explaining why SAEs often find semantically meaningful features.
Activation Patching: The authors propose that activation patching works by modifying the filler-role binding structure. They derive a formula to construct the destination activation ( $h_{dest}$ ) by subtracting the binding difference from the source activation ( $h_{src}$ ), rather than running a second forward pass.

3. Empirical Validation of Derived Methods

The authors constructed instances of the four methods using only the TPE approximation and compared them to standard implementations.

Analogies: TPE-constructed analogies matched or exceeded the accuracy of standard vector analogies (e.g., 99.2% top-1 accuracy on sequence tasks) while requiring fewer forward passes.
Probes: Analytically constructed probes achieved performance comparable to trained probes across all models and positions.
SAEs: TPE-constructed SAEs achieved high reconstruction scores ( $R^2 \approx 0.96$ ) and feature quality, comparable to trained SAEs, though with slightly lower reconstruction fidelity due to the continuous vs. discrete nature of the representations.
Patching: TPE-constructed patching interventions produced restoration scores nearly identical to standard patching (Pearson correlation $r > 0.999$ across models), correctly reproducing the magnitude of effects and the shift of information across layers.

Significance and Claims

The paper claims to move the field toward a "unified account of the nature of neural networks." By positing TPRs as the underlying structure, the authors argue that:

Precision: The TPR hypothesis provides a closed-form equation for representation structure, allowing representations to be constructed independently of the network being analyzed.
Explanatory Power: It explains how representations can be analyzed and edited. For instance, it clarifies why linear probes work (unbinding), why analogies hold (binding differences), and why patching is effective (modifying bindings).
Beyond Linear Representation: While the "Linear Representation Hypothesis" posits that concepts are encoded linearly, TPRs offer a more expressive structure. They explain how position-content combinations are stored compositionally (filler-role bindings) rather than as a "bag of concepts," addressing limitations of simple linear models in capturing structural relationships (e.g., sequence order).

The authors conclude that this work serves as a step toward a "science of language models" that seeks unifying structural regularities across different models, tasks, and interpretability techniques, similar to how biology unified the study of individual species into general laws.