Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

Technical Summary: Lesioned Multimodal Language Models Reproduce Aphasic Picture-Naming Patterns

Problem Statement
Aphasia following stroke produces systematic, non-random error patterns in language production, characterized by specific profiles such as semantic errors, formal paraphasias, and neologisms. While previous computational efforts have modeled these errors using purpose-built connectionist architectures with task-specific parameter fitting, it remains untested whether general-purpose, large-scale multimodal language models (LLMs) can reproduce these clinical error profiles without being designed for clinical simulation. A central question is whether the "perturbation space" of a general-purpose model contains trajectories that can replicate the joint error distributions of individual persons with aphasia (PWAs), thereby serving as potential cognitive analogs for human language processing.

Methodology
The study utilizes the LLaVA 1.6 multimodal language model, selected for its open weights, transparent architecture (a CLIP visual encoder feeding a Vicuna-13B language backbone), and lack of intervening architectural devices like gated experts. The experimental design involves a controlled perturbation protocol applied to the model's internal representations during a standardized picture-naming task, the Philadelphia Naming Test (PNT).

• Perturbation Protocol: The researchers applied multiplicative Gaussian noise to specific "units" (hidden dimensions) within the model's 40 transformer decoder layers. The perturbation is defined by a triplet of parameters:

  1. Target Layer ($l$): One of the 40 decoder layers (0–39).
  2. Modification Proportion ($p$): The fraction of units in the target layer randomly selected for perturbation (10% to 100%).
  3. Noise Level ($\sigma$): The standard deviation of the Gaussian noise (1.1 to 2.0).
     This creates a 4,000-condition parameter space ($40 \text{ layers} \times 10 \text{ proportions} \times 10 \text{ noise levels}$).

• Data and Classification: The study evaluated 278 PWAs from the C-STAR dataset. Model outputs were classified into seven categories mirroring clinical taxonomies: Correct, Semantic, Formal, Mixed, Unrelated, Neologism, and No Response. To ensure methodological consistency between model outputs and clinical data, a feature-augmented neural classifier (based on DeBERTa-v3-small) was fine-tuned on 3,160 expert-annotated responses. This classifier achieved "Almost Perfect" agreement (Cohen's Kappa = 0.87) with human experts, notably outperforming rule-based methods in detecting "No Response" errors.

• Matching Analysis: For each PWA, the researchers searched the perturbation space to find the configuration $(l, p, \sigma)$ that best reproduced the patient's joint error distribution across the seven categories. Matching was evaluated against a retest-derived tolerance threshold (2 SD) and validated against Monte Carlo baselines to ensure results reflected joint inter-category structure rather than marginal overlap.

Key Results

1. Reproduction of Error Categories: Controlled perturbations successfully generated six of the seven clinical error categories (Correct, Semantic, Unrelated, Mixed, Neologism, No Response) at clinically comparable proportions. Distinct regions in the parameter space were associated with specific error types (e.g., lower layers with high noise produced "No Response" errors, while upper layers produced "Neologisms").

   • Exception: Formal paraphasias (phonologically similar real words) were systematically under-produced in the model (peaking at ~3.5% of responses) compared to clinical profiles where they can dominate. The authors attribute this to the model's token-level vocabulary access, which favors the generation of phonologically related non-lexical strings (neologisms) over the retrieval of specific real-word neighbors.

2. Individual Profile Matching: The perturbation framework successfully reproduced the complete error profiles of individual PWAs.

   • 97.8% of PWAs had a configuration that matched at least six of seven error categories within tolerance.
   • 79.5% of PWAs had a configuration that matched all seven categories.
   • Monte Carlo baselines confirmed that these matches were not due to chance; the "Real Selection" baseline (preserving joint structure) outperformed the "Generated" baseline (preserving only marginals) by a factor of 1.35 at the high-quality matching threshold, indicating the model's perturbation manifold captures the complex joint structure of aphasic errors.

3. Parameter Sensitivity: The "layer" parameter acted as a primary axis of differentiation. Perturbing early layers (0–9) led to rapid collapse into no-response outputs, while perturbing later layers (30–39) allowed for a graded transition from correct responses to semantic errors and neologisms.

Significance and Claims
The paper establishes a quantitative framework for reproducing individual aphasic error patterns in picture naming using a general-purpose multimodal language model. The authors claim that:

• General-purpose models, when subjected to controlled, focal perturbations, can exhibit output-space error distributions that match those of individual stroke survivors without task-specific design or per-patient parameter fitting.
• This capability suggests that such models can serve as "digital twins" for individuals with post-stroke aphasia.
• The work demonstrates the potential of language models not merely as computational tools, but as cognitive models of language processing in the human brain.
• The framework offers a foundation for future translational studies, such as testing and developing therapies, by characterizing individual error profiles in a compact behavioral space defined by the perturbation parameters.

The authors explicitly limit their claims to output-space correspondence. They do not assert that the specific perturbations applied to the model are biologically equivalent to the neural lesions in the patients, nor do they claim the model replicates the underlying mechanisms of recovery or other language domains (e.g., comprehension, repetition) beyond the scope of the picture-naming task. The residual failure to model formal paraphasias is identified as an architectural limitation of the specific model's lexicon access, not a fundamental failure of the approach.