Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability
This paper introduces "mechanistic tomography," a unified framework that models interpretability techniques as designed measurements to systematically recover internal model mechanisms and intervention effects, demonstrating through various experiments that control-oriented validation and tailored measurement families can efficiently reconstruct causal structures and interactions in large language models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern artificial intelligence systems, often called large language models, are built from billions of tiny mathematical switches arranged in layers. These systems can write stories, solve problems, and answer questions, but they operate as black boxes: we can see the input and the output, but the internal steps that connect them remain hidden. For years, researchers have tried to peek inside these boxes by turning parts of the system on or off, or by watching how the internal numbers change when the input shifts slightly. The goal is to understand which specific parts of the machine are responsible for specific thoughts or words. However, these different ways of looking inside often seem to tell different stories. One method might suggest a component is vital, while another suggests it is irrelevant, leaving scientists unsure if they are measuring the same thing or just different shadows of the same object.
A new approach called mechanistic tomography offers a way to unify these conflicting views. The core idea is to treat the search for internal mechanisms not as a mystery to be solved by intuition, but as a measurement problem to be solved by design. Just as a doctor uses different types of scans to build a clear picture of a patient's health, researchers can design specific, controlled experiments to measure how the AI's internal parts work together. This paper proposes a structured workflow for these experiments, starting with the simplest, cheapest measurements and only adding complexity if the simple ones fail to predict what the machine will actually do when pushed. The researchers found that while simple measurements often work well, they sometimes miss crucial interactions between parts, and that the right way to measure depends entirely on what the machine is being asked to do.
The journey begins with a fundamental realization: the way we measure a machine's internal state changes the answer we get. If you nudge one part of the system slightly, you get a different result than if you nudge it hard, or if you nudge several parts at once. The author realized that all the different methods researchers use—whether they are changing one part at a time, using gradients to estimate effects, or testing groups of parts together—can be described as a single type of measurement problem. They framed it as a puzzle where the goal is to recover a hidden map of effects from a series of partial observations. Each observation is a specific intervention, like changing a few words in a prompt or altering a specific internal signal, and the result is a data point that helps fill in the map. The challenge is that the machine is not perfectly linear; its parts interact in complex ways, and the "noise" in the measurement depends on how big the nudge is.
To test this framework, the researchers started in a controlled environment where they knew the exact answer beforehand. They built a simulation using a simple probabilistic system, like a hidden Markov model, which acts as a perfect reference point. In this setting, they could create a "wind tunnel" for the AI, where they knew exactly what the machine's internal belief state should be. They then tested how well different measurement methods could predict the machine's behavior when used to control it. They found a direct link between the accuracy of the internal measurement and the success of the control. If the measurement was slightly off, the control action failed. More importantly, they discovered that a measurement could look successful by improving the main goal while accidentally messing up a different, unrelated part of the system. This proved that simply seeing a machine get better at a task is not enough; you must verify that the internal explanation actually matches the action being taken.
Moving from simulations to real, pre-trained models, the researchers applied their step-by-step procedure to see how far simple measurements could go. In one experiment with a model trained to identify indirect objects in sentences, they started by assuming the machine's behavior was a simple sum of individual parts. They measured the effect of removing specific groups of internal components. The simple model worked well at first, but when they tested it on new, unseen combinations of parts, it failed to predict the outcome accurately. The error wasn't random; it was structured. It turned out that the machine relied on interactions between parts. For example, one group of components would only become important if another group was already weakened. By adding measurements specifically designed to catch these interactions, the researchers could fix the prediction errors. They found that a specific pair of component groups was responsible for the largest remaining error, and adding just that one interaction term made the model's predictions nearly perfect.
In a second experiment with a much larger, modern model designed to follow instructions, the researchers asked if they needed to look for these complex interactions. They started with the same simple, additive approach and calibrated it with a few extra measurements to account for the size of the nudge. This time, the simple model worked almost perfectly. It predicted the model's behavior on new, unseen prompts with extremely high accuracy. When they tried to add the complex interaction measurements, the results did not improve. The extra complexity was unnecessary. This result was just as important as the first one: it showed that you should not assume a machine is complex just because it is large. Sometimes, a simple explanation is sufficient, and adding more layers of measurement only adds cost without adding value.
The paper concludes by offering a practical guide for anyone trying to understand these machines. The advice is to start with the simplest, cheapest measurement that could possibly explain the behavior. If that works, stop. If it fails, look at the pattern of the failure. If the error is just a matter of scale, a simple correction might fix it. If the error suggests that parts are working together in a way the simple model can't see, then design new measurements to catch those interactions. Finally, always test the explanation on a task it was never trained on, or in a control loop where it guides an action. The goal is not just to reconstruct a picture of the machine, but to build an observer that can reliably predict what the machine will do when the real world pushes it. This approach shifts the focus from finding the "true" internal mechanism to finding the most useful and reliable way to measure it for a specific purpose.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.