💬 NLP

Recovering Lesion Parameters from Aphasic Picture Naming Error Profiles in Large Language Models

This paper demonstrates that while specific transformer layer indices cannot be uniquely recovered from aphasic picture-naming error profiles due to functional redundancy, perturbation intensity and noise parameters can be successfully inverted to generate counterfactual models that reproduce target behaviors and discriminate clinical syndromes, offering a robust framework for causal LLM interpretability.

Original authors: Yong Yang, Roger Newman-Norlund, Xiang Guan, Saeed Ahmadi, Regan Willis, Nadra Salman, Kalil Warren, Sophie Arheix-Parras, Srihari Nelakuditi, Leonardo Bonilha, Christopher Rorden, Rutvik H. Desai, Ju

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yong Yang, Roger Newman-Norlund, Xiang Guan, Saeed Ahmadi, Regan Willis, Nadra Salman, Kalil Warren, Sophie Arheix-Parras, Srihari Nelakuditi, Leonardo Bonilha, Christopher Rorden, Rutvik H. Desai, Julius Fridriksson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how a giant, magical library works. You can't see the books or the librarians inside, but you can watch what happens when you whisper a secret to the building. Sometimes the library whispers back a perfect answer; other times, it stammers, mixes up words, or says something completely nonsense. Scientists call these "Large Language Models" (LLMs), and they are the digital brains behind many of the smart tools we use today. But here's the tricky part: we don't really know which part of the brain is doing the talking. It's like trying to guess which specific gear in a clock made the hands move just by watching the clock face.

To understand this, scientists often use a method called "lesioning." Imagine taking a tiny, precise hammer and tapping a specific gear in that clock to see if it breaks the timekeeping. If the clock starts running slow, you know that gear was important. In the world of AI, researchers can "break" parts of the digital brain by adding a little bit of digital noise to it, just like a stroke can damage parts of a human brain and cause speech problems. This paper asks a fascinating "reverse" question: If we see the clock running slow, can we look at the time and figure out exactly which gear we tapped? And if we guess the gear, can we tap that same gear on a brand-new clock and make it run slow in the exact same way? This isn't just about fixing clocks; it's about proving we truly understand how these digital brains think, rather than just guessing what they are doing.

The Great AI Detective Game

In this study, a team of researchers decided to play a high-stakes detective game with a digital brain called LLaVA-Vicuna 13B. Think of this AI as a super-smart robot that can look at a picture and tell you what it is. The researchers wanted to see if they could figure out exactly how they "broke" the robot just by looking at the mistakes it made.

First, they set up a massive experiment. They took the robot and applied tiny "lesions" (digital damage) to it in 4,840 different ways. They changed three things for each attempt:

Where they hit it (which of the robot's 40 layers of "thinking" they touched).
How much of that layer they damaged (from 0% to 100%).
How loud the noise was (how chaotic the damage was).

After breaking the robot in each of these 4,840 ways, they showed it 175 pictures of everyday objects and asked it to name them. They recorded every mistake the robot made—whether it called a "dog" a "cat" (a semantic error), made up a nonsense word (a neologism), or just stayed silent. This collection of mistakes is called an "error profile."

The Reverse Engineering Challenge

Now came the hard part. The researchers built a new, special computer program (an "inverse model") to act as a detective. They fed this detective the error profiles (the list of mistakes) and asked: "Based on these mistakes, what exactly did we break?"

The detective had to guess three things:

Which layer was hit?
How much of it was hit?
How noisy was the damage?

The Big Discovery: Intensity vs. Location
The results were surprising and revealed a secret about how these AI brains are built.

The "How Much" was easy to find: The detective was very good at guessing how much damage was done. Whether the robot was slightly confused or completely scrambled, the detective could tell the difference with high accuracy. It's like being able to tell if a car engine is "a little rough" or "totally dead" just by listening to the sputter.
The "Where" was fuzzy: The detective was not very good at guessing the exact layer. It could only guess the right layer about 14.8% of the time (which is much better than random guessing, but still far from perfect). However, it could usually guess the right neighborhood of layers (within 5 layers) about 77.6% of the time.

The "Magic Trick" Test (Counterfactual Validation)
Here is where the paper gets really cool. Just because the detective guessed the "how much" and "where" doesn't mean it was right. To prove it, the researchers did a "magic trick." They took the detective's guess, went to a brand new, unbroken robot, and applied that exact guess to it.

The Result: In 81.4% of the cases, the new robot started making the exact same mistakes as the original broken robot!
The Twist: Even when the detective guessed the wrong layer, if it guessed the right amount of damage, the new robot still made the same mistakes.

This proves that for these AI brains, how much you break them matters way more than where you break them. It suggests that the middle parts of the robot's brain are "redundant"—they all do similar jobs. If you damage one middle layer, another middle layer can kind of take over, so the robot's behavior only changes based on the severity of the hit, not the specific location.

Connecting to Real People

To make sure this wasn't just a game with robots, the researchers tested their detective on 278 real people who had suffered strokes and had trouble naming pictures (a condition called aphasia). The researchers didn't teach the detective anything about human brains; it only knew about the broken robots.

When they fed the human patients' error profiles into the detective, something amazing happened:

The detective's guesses about the "damage intensity" (how much was broken) perfectly matched the severity of the patients' conditions. Patients with severe speech problems were predicted to have "high damage," while those with mild problems had "low damage."
The detective also grouped the patients by their specific type of aphasia (like Broca's or Wernicke's) based on the "location" it guessed, even though it had never seen a human before.

What This Means

This paper doesn't claim to have solved the mystery of the human brain or fixed all AI problems. Instead, it offers a new way to test if we truly understand AI.

What it rules out: It shows that simply looking at which layer an AI uses isn't enough to understand it. You can't just say, "Layer 10 is for nouns," because damaging Layer 10 might look exactly like damaging Layer 15 if the damage is the same size.
What it suggests: The "middle" of these AI brains is full of backup systems. They are flexible and redundant.
The Confidence: The researchers are very confident in the "intensity" finding because they tested it on a fresh robot and it worked 81.4% of the time. They are confident that the "location" is hard to pin down because the robot behaves the same way even when you guess the wrong layer.

In short, this study teaches us that to understand these giant digital brains, we need to stop looking for a single "broken gear" and start looking at how hard the whole machine is being shaken. And the best part? The method they invented—guessing the damage, then testing it on a fresh machine—could be used to check if we really understand any part of how AI thinks in the future.

Technical Summary: Recovering Lesion Parameters from Aphasic Picture Naming Error Profiles

Problem Statement
Current interpretability methods for Large Language Models (LLMs)—including probing classifiers, attention visualization, causal tracing, and activation patching—primarily describe internal model states and their correlations with behavior. However, these methods do not directly test whether an inferred internal state is causally sufficient to produce the observed behavior. They identify what correlates with behavior, not what, if intervened upon, would produce it. This paper addresses this gap by posing an inverse question: given a specific behavioral output (an aphasic picture naming error profile), can the specific lesion parameters (layer index, modification percentage, and noise magnitude) that produced it be recovered? Furthermore, does the structure of this inverse problem reveal fundamental properties of transformer computation?

Methodology
The study utilizes the multimodal model LLaVA-Vicuna 1.6 13B (combining a CLIP vision encoder and a 40-layer Vicuna language decoder). The methodology involves a forward perturbation phase followed by an inverse mapping phase.

Forward Perturbation (Data Generation):
- Parameter Space: The authors defined a grid of 4,840 configurations varying three parameters:
  - Layer Index ( $L$ ): 0–39 (language decoder layers only).
  - Modification Percentage ( $M$ ): 0–100% (fraction of hidden units affected).
  - Noise Sigma ( $\sigma$ ): 1.0–2.0 (intensity of Gaussian noise).
- Lesion Mechanism: For a given configuration $(L, M, \sigma)$ , weights in layer $L$ associated with a fraction $M$ of hidden units were modified by multiplying them with $(1 + \varepsilon)$ , where $\varepsilon \sim N(0, \sigma^2)$ .
- Behavioral Output: Each perturbed model was evaluated on the 175-item Philadelphia Naming Test (PNT). Responses were classified into a clinical taxonomy of seven error types: correct, semantic, unrelated, formal, mixed, neologism, and no-response.
- Feature Extraction: The inverse model input was a 166-dimensional vector comprising:
  - Tier 1: 7 BERT-classified error rates.
  - Tier 2: 5 derived statistics (e.g., total error rate, entropy).
  - Tier 3: 12 text-level statistics (length, character classes).
  - Tier 4: 10 probability moments (uncertainty metrics).
  - Tier 5: 8 lexical diversity features.
  - Tier 6: 116 sentence-transformer embedding statistics.
- Datasets: The 4,840 configurations were split 70/15/15 for training, validation, and testing. An independent validation set was generated using a distinct random seed to prevent data leakage. Additionally, behavioral data from 278 stroke survivors (CSTAR dataset) were used for out-of-distribution testing.
Inverse Model Architecture:
- A multi-task neural network was trained to map the 166-dimensional behavioral profile back to the three perturbation parameters.
- Encoder: Three fully connected blocks (256 $\to$ 128 $\to$ 64 units) with batch normalization, ReLU, and dropout.
- Heads:
  - Layer Head: A linear layer for 40-way classification (Cross-Entropy loss).
  - Intensity Heads: Two parallel regression heads for Modification Percentage and Noise Sigma, using heteroscedastic Gaussian negative log-likelihood loss to predict mean and variance.
- Training: Optimized with AdamW over 500 epochs. The total loss was a weighted sum of the three task losses.
Validation Framework:
- Counterfactual Validation: To verify causal sufficiency, the predicted parameters from the inverse model were applied to a fresh, unperturbed instance of the LLaVA-Vicuna model. The resulting error profile was compared to the original target profile using cosine similarity and Pearson correlation. High fidelity was defined as similarity > 0.85 and correlation > 0.90.
- Clinical Application: The trained inverse model (without retraining) was applied to the 278 stroke survivors to test if recovered parameters could discriminate between clinical aphasia syndromes (e.g., Broca's, Anomic, Global).

Key Results

Partial Recovery of Parameters:
- Intensity is Recoverable: The model successfully recovered perturbation intensity. Modification percentage prediction achieved $R^2 = 0.570$ and Pearson $r = 0.771$ . Noise sigma prediction achieved $R^2 = 0.315$ and $r = 0.581$ .
- Location is Ambiguous: Layer index prediction was significantly less accurate. Top-1 accuracy was 14.8% (chance is 2.5%), though within-5 accuracy was 77.6% (chance is 27.5%). This suggests the model can identify the neighborhood of the lesion but not the exact layer.
Counterfactual Fidelity:
- Despite the difficulty in pinpointing the exact layer, the recovered parameters were causally sufficient to reproduce the target behavior. On the independent full-grid validation set (4,840 configurations), 81.4% of cases achieved high-fidelity reproduction.
- Crucially, predictions with incorrect layer indices but correct intensity often yielded high-fidelity reconstructions. Conversely, incorrect intensity led to low fidelity even if the layer was correct. This indicates that perturbation intensity is the primary determinant of behavioral fidelity, while location is secondary.
Functional Redundancy:
- The dissociation between low layer-recovery accuracy and high counterfactual fidelity suggests functional redundancy across transformer layers. Many mid-stack layers produce functionally equivalent behavioral footprints when perturbed at matched intensities.
- Per-layer analysis showed an asymmetric U-shaped pattern: layers at the embedding interface (0–2) and output interface (36–39) were more recoverable than the mid-stack (3–35), consistent with information localization at the network boundaries.
Clinical Generalization (Out-of-Distribution):
- When applied to 278 stroke survivors, the inverse model's recovered parameters were syndrome-discriminative.
- Modification Percentage: Showed the strongest discrimination ( $\eta^2 = 0.288$ ), with severity gradients aligning with clinical syndromes (e.g., Global and Broca's aphasia mapped to higher modification percentages than Anomic or controls).
- Layer Index: Showed moderate discrimination ( $\eta^2 = 0.130$ ), with severe non-fluent syndromes mapping to earlier layers.
- Noise Sigma: Did not significantly discriminate between syndromes.
- Sensitivity analyses across five different grouping schemes confirmed these findings were robust and not driven solely by control group contrasts or small sample sizes.

Significance and Claims

The paper claims to establish a general methodological framework for LLM interpretability based on counterfactual validation. Its primary contributions are:

Causal Sufficiency Test: It moves beyond correlational interpretability by demonstrating that inferred internal states (lesion parameters) can be recovered and re-applied to reproduce specific behaviors.
Architectural Insight: The results reveal a property of transformer architecture not captured by standard interpretability methods: functional redundancy. The finding that behavior is determined more by how much a layer is disrupted than which layer is disrupted challenges the assumption that specific behaviors map uniquely to specific layers.
Bridging Synthetic and Clinical Data: The study demonstrates that an inverse model trained entirely on synthetic LLM perturbations can generalize to real-world clinical aphasia data, recovering parameters that align with established clinical syndromes. This suggests the learned behavior-to-perturbation mapping captures structural properties of language processing that transcend the specific training distribution of the LLM.

The authors modestly note that these findings are an "existence proof" for one specific multimodal model (LLaVA-Vicuna 13B) and that the degree of recoverability may vary across different transformer architectures. They do not claim to have mapped specific transformer layers to specific brain regions, but rather that the structure of the inverse problem reveals computational properties of the model itself.

The Great AI Detective Game

The Reverse Engineering Challenge

Connecting to Real People

What This Means

Technical Summary: Recovering Lesion Parameters from Aphasic Picture Naming Error Profiles

More like this