💬 NLP

Artificial Aphasias in Lesioned Language Models

This paper introduces an aphasia-inspired technique to lesion language model parameters, revealing that while these models exhibit a full spectrum of language symptoms, their resulting impairment profiles differ qualitatively from human aphasias and vary systematically across architectural components and network depth, suggesting that language breakdown patterns are shaped by specific learning and processing mechanisms rather than being domain-invariant.

Original authors: Nathan Roll, Jill Kries, Laura Gwilliams, Cory Shain

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nathan Roll, Jill Kries, Laura Gwilliams, Cory Shain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (like the ones powering chatbots) as a massive, intricate factory. Inside this factory, there are different teams of workers (the "components") who handle different parts of the job. Some workers are the Attention Team, who decide which pieces of information to focus on. Others are the Feed-Forward Team, who actually process and transform that information into words.

This paper is like a team of "digital doctors" who decided to simulate a stroke in this factory. Instead of using real brains, they "lesioned" (or zeroed out) parts of the factory's machinery to see what kind of mistakes the factory would start making. They then compared these machine mistakes to the speech errors made by real humans who have aphasia (a language disorder caused by brain damage).

Here is what they found, using simple analogies:

1. The "Symptom Battery" Test

Just as doctors use specific tests to diagnose human speech problems, the researchers created a test called the Text Aphasia Battery (TAB). They fed the damaged factory thousands of prompts (like "Describe this picture" or "Repeat this sentence") and graded the output for specific types of errors, such as:

Semantic errors: Saying the wrong thing or being vague.
Syntactic errors: Breaking the rules of grammar.
Phonological errors: Making up nonsense words.
Fluency errors: Stuttering, repeating words, or stopping mid-sentence.

2. Different Teams Make Different Mistakes

The biggest discovery was that damaging the Attention Team caused different types of errors than damaging the Feed-Forward Team.

If you hurt the Attention Team: The factory started making "sound" errors. It struggled with the flow of words, stuttering, repeating phrases, and producing nonsense sounds (phonological and fluency errors). It was like a person who knows what they want to say but can't get the words out smoothly.
If you hurt the Feed-Forward Team: The factory started making "meaning" errors. The output became vague, short, repetitive, or completely off-topic. It was like a person who is confused about the topic entirely, producing short, empty sentences or getting lost in the conversation.

The Analogy: Imagine a car. If you break the steering wheel (Attention), the car might swerve wildly and hit the same curb over and over (repetition/fluency issues). If you break the engine (Feed-Forward), the car might just stop moving or move in a straight line without any purpose (vague/off-topic issues).

3. Where You Break It Matters (Depth)

The factory has many layers, like floors in a skyscraper. The researchers found that where they broke the machine mattered:

Breaking the early floors: Caused errors in meaning and grammar (the "big picture" stuff).
Breaking the middle-to-late floors: Caused errors in how the words sounded and flowed (the "delivery" stuff).

This was surprising because, in human brains, we often think of "sound" processing as happening early and "meaning" later. In these AI factories, it's almost the opposite: the early layers handle the heavy lifting of meaning, while the later layers handle the fine-tuning of how the words are delivered.

4. Machines vs. Humans: Similar Symptoms, Different Patterns

The researchers compared the broken machines to real humans with aphasia.

The Good News: Some broken machines did look a bit like specific human conditions. For example, a machine with a damaged "Gate" component made errors that looked a little like a human with Broca's aphasia.
The Bad News: The patterns weren't a perfect match. Humans with aphasia tend to make a lot of meaningful errors (forgetting words, mixing up grammar). Broken machines, however, tended to make a lot of "Other" errors—like repeating the same phrase over and over, or going off on a tangent.

The Takeaway: The researchers concluded that while we can use human language disorders as a map to understand how AI works, AI is not a human brain. The way a machine breaks down is unique to its own architecture and how it was trained. You can't simply say, "This AI has Broca's aphasia," because the underlying causes and the mix of symptoms are fundamentally different.

Summary

This paper is a "stress test" for AI. By intentionally breaking parts of the machine, the authors learned that:

Different parts of the AI handle different types of language tasks (meaning vs. flow).
The location of the damage changes the type of error.
While AI can mimic human speech errors, the "symptoms" it produces are a unique mix that doesn't perfectly copy human brain disorders.

It's a way of reverse-engineering the AI's "brain" to understand how it thinks, using the tools we use to understand human brains, but with the important realization that the two are built very differently.

Technical Summary: Artificial Aphasias in Lesioned Language Models

Problem and Motivation
Neuropsychologists have long utilized naturally occurring brain damage (aphasias) to map the functional organization of human language, linking specific brain regions to distinct symptom profiles. Concurrently, researchers in mechanistic interpretability intervene on language models (LMs)—often by silencing weights or activations—to study emergent structure-function correspondences. However, these two communities have largely operated independently. While recent work has attempted to induce human aphasia subtypes (e.g., Broca's, Wernicke's) in LMs via lesioning, these studies often rely on coarse categorical labels that may obscure substantial heterogeneity in symptom patterns.

This paper introduces an aphasia-inspired technique to characterize the emergent functional organization of LMs more granularly. The authors argue that rather than mapping LMs to broad clinical categories, researchers should profile detailed symptom distributions to understand how specific model components contribute to language processing failures. The study aims to determine if lesioned LMs exhibit functional organizations similar to humans or if their failure modes are distinct, thereby constraining hypotheses about why human aphasia symptoms pattern as they do.

Methodology
The authors induce "artificial aphasias" by "lesioning" (zeroing out) specific parameter matrices in decoder-only Transformer architectures. The experimental setup involves:

Models: Five ~1B-parameter model variants across three families: Llama 3.2, Gemma 3, and OLMo 2.
Lesion Targets: Seven specific components are targeted across layers: the Query (Q), Key (K), Value (V), and Output (O) matrices of the attention mechanism, and the Gate, Up, and Down matrices of the feed-forward network (FFN).
Severity: Lesions are applied at five severity levels ( $s \in \{0, 0.25, 0.50, 0.75, 1.0\}$ ), where $s$ represents the expected fraction of weights zeroed.
Assessment: Outputs are scored using the Text Aphasia Battery (TAB), a benchmark specifically engineered for text-only LMs. The TAB assigns 21 binary symptoms across five categories: Semantic, Syntactic, Fluency, Phonological, and Other (including repetition loops, stereotypies, and off-topic responses).
Comparison: The study compares 112,426 scored outputs from lesioned LMs against 6,000 productions from the AphasiaBank corpus (covering various human aphasia subtypes and controls).
Statistical Analysis: The authors employ nonparametric methods, including condition-level bootstrapping, paired sign-flip tests, and permutation tests, to assess differences in symptom profiles between components and against human baselines.

Key Results

Distinct Functional Profiles for Attention vs. FFN: Lesions to attention mechanisms and feed-forward networks produce systematically different symptom profiles.
- FFN Lesions: Shift outputs toward vague content ("Meaning unclear"), short or formulaic strings, off-topic responses, and repetition loops. They disproportionately increase symptoms in the Semantic and "Other" categories.
- Attention Lesions: Disproportionately impair phonological and fluency aspects (e.g., phonemic paraphasias, halting speech).
- Sub-structure: Within attention, Q/K and V/O form distinct cohorts with similar effects. Within FFN, Down lesions cause the highest symptom burden, while Gate lesions often reduce outputs to punctuation-only or repeated function words.
Depth-Dependent Symptom Localization: The layer at which a lesion occurs significantly shapes the symptom profile.
- Early Layers: Lesions in early layers disproportionately cause semantic and syntactic deficits.
- Late-Middle Layers: Lesions in later layers yield higher rates of phonological and fluency deficits.
- Necessity vs. Content: This contrasts with brain decoding studies that often find alignment in mid-to-late layers; here, early layers appear more causally necessary for core function, while later layers handle finer-grained refinements.
Divergence from Human Aphasia: While some LM lesions produce symptom profiles that are descriptively closer to certain human aphasia subtypes than others, there are fundamental qualitative differences:
- Symptom Composition: Humans exhibit higher rates of semantic and syntactic symptoms, whereas lesioned LMs overproduce "Other" category symptoms (repetition loops, off-topic text).
- Symptom Burden: Humans generally have higher overall symptom burdens than lesioned LMs.
- Co-occurrence: Human aphasia shows dense symptom co-occurrence patterns (e.g., specific syndromes), whereas LMs show less clear co-occurrence structures.
- Conclusion: The paper finds that while there is some shared structure, the emergent functional organization of Transformer LMs is qualitatively distinct from the human brain. Aphasia symptoms appear to be contingent properties of the human brain's specific learning and processing details rather than a domain-invariant consequence of disrupted language processing.

Significance and Claims
The paper makes four primary contributions:

Proof of Concept: It establishes a "neuropsychology" for LMs using the TAB to profile detailed failure modes rather than relying on coarse diagnostic labels.
Functional Differentiation: It identifies reliable functional differences between attention and FFN mechanisms across multiple model architectures.
Depth Effects: It reveals that semantic and syntactic deficits are reliably localized to earlier layers, while phonological and fluency deficits emerge in later layers.
Nuance on Convergence: It refines prior claims of convergence between LM and human aphasias. While some superficial similarities exist, the paper argues that the detailed symptom patterns suggest human aphasia syndromes are not inherent to the structure of language itself but are specific to the biological implementation of language processing.

The authors explicitly state that they do not claim LMs possess clinical aphasia. Instead, they use the clinical framework as a coordinate system to study emergent function. The results suggest that while LM lesions can produce clinically legible failures, the underlying organization differs significantly from the human brain, cautioning against direct equating of transformer components with human brain regions.

1. The "Symptom Battery" Test

2. Different Teams Make Different Mistakes

3. Where You Break It Matters (Depth)

4. Machines vs. Humans: Similar Symptoms, Different Patterns

Summary

More like this