Attribution Under a Floor: Why Single-Component Ablation Overcounts Safety-Critical Layers and Neurons
This paper demonstrates that the standard method for identifying safety-critical layers and neurons in fine-tuned language models is statistically flawed due to a "floor effect" caused by small prompt batteries and unsafe baselines, which leads to the erroneous overcounting of critical components and invalidates the published evidence for a dual-mechanism safety account.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When we ask a large language model to refuse a harmful request, we are asking it to perform a complex act of self-restraint. These models are trained on vast amounts of human text, learning to predict the next word in a sentence, but they also undergo a special training phase to learn what not to say. Researchers in the field of mechanistic interpretability treat these models like intricate machines that can be taken apart to see how they work. They believe that specific parts of the model's internal structure act as gatekeepers, deciding whether to answer a question or refuse it. To find these gatekeepers, scientists often use a method called ablation. This involves temporarily disabling one small part of the model, such as a single layer of processing or a specific neuron, and then watching to see if the model suddenly starts producing harmful answers when it previously would not. If the model fails to refuse after a part is disabled, that part is labeled as critical to safety. The goal is to map the exact machinery of refusal so that we can understand, and perhaps improve, how these systems stay safe.
A recent study by Yangyi Zhang at Georgetown University examines a specific instance of this research and finds that the conclusions drawn from it are built on a statistical illusion. The study audits an archived experiment where researchers claimed to have identified six specific layers and 205 specific neurons that act as the primary safety gatekeepers for a language model. Zhang's analysis reveals that these numbers do not reflect the true mechanics of the model. Instead, they are the result of a design flaw in the experiment itself, specifically a problem known as a floor effect. In this context, the floor effect occurs because the test used to measure safety was too small and too weak to detect real changes. The experiment relied on a battery of only four harmful prompts to test the model. Before any parts of the model were disabled, the model had already failed to refuse three of those four prompts. It was already unsafe on the majority of the test cases.
Because the model was already failing on three out of four prompts, disabling any single part of the model could not make it fail any more often than that. The count of unsafe responses was stuck at a minimum of three, regardless of which layer was turned off. The only way a layer could be marked as "critical" was if it happened to be the one that caused the single remaining safe prompt to flip and become unsafe. The study shows that the six layers reported as critical were simply the ones that happened to flip that one specific prompt. The other 26 layers, which did not flip that prompt, were reported as safe, even though they performed almost identically to the critical ones. The distinction between a "critical" layer and a "safe" layer was not a difference in how the model worked, but a difference in how the tiny test battery happened to behave. The reported set of six critical layers was not a discovery of the model's safety mechanism; it was a property of the four prompts used in the test.
The investigation also uncovered similar issues with the claims regarding individual neurons. The original report stated that researchers had scanned hundreds of thousands of neurons and found 205 that were critical, using a specific threshold to select them. However, when Zhang examined the actual data files stored from that experiment, the numbers did not match. The stored data contained records for only 500 neurons, not the full set, and the impact scores recorded for them were far lower than the threshold used in the report. In the stored data, a threshold that would select 205 neurons did not exist; instead, the data showed that only one neuron exceeded the reported threshold. The claim of 205 critical neurons was not supported by the evidence that was actually saved. Furthermore, other specific measurements mentioned in the original report, such as the speed of a calculation or the overlap between different testing methods, were not found in any of the stored files. The numbers appeared in the text of the paper but could not be traced back to the raw data.
The study also challenges a claim that the importance of the model's layers follows a smooth, predictable pattern that fades away as you move deeper into the network. The original report suggested that the impact of disabling layers dropped off in a neat, exponential curve, with the most important work happening in the very first layers. The re-analysis of the stored data shows that the actual pattern is much messier. The impact of the layers does not fade smoothly; instead, it is dominated by a few specific layers, including the very last layer of the network, which showed a large impact when disabled. The data does not support the idea of a clean, fading curve. Instead, it suggests a more complex reality where safety is influenced by a sparse set of layers scattered throughout the model, rather than a single, orderly decay.
This work does not prove that the model lacks safety mechanisms or that the dual-mechanism theory is false. It simply demonstrates that the specific evidence presented in the archived run is insufficient to support those claims. The findings highlight a fundamental requirement for this type of research: the test battery must be large enough and composed of enough prompts that the model actually refuses, so that disabling a part can be seen to change the outcome. When the baseline is already unsafe on most prompts, the test cannot measure the contribution of individual parts. To truly identify which layers or neurons are responsible for safety, researchers need a much larger set of baseline-safe prompts, and they must report the full distribution of results rather than just a list of layers that crossed an arbitrary threshold. The study concludes that attribution claims are only as strong as the data they can be recomputed from, and in this case, the data did not support the bold conclusions that were drawn.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.