💬 NLP

Dementia-Agents: A Multi-Modal Multi-Agent System for Dementia Staging and Phenotyping

This paper introduces Dementia-Agents, a clinically aligned multi-modal multi-agent framework that outperforms existing models in real-world dementia staging and phenotyping by integrating structured clinical records through specialized expert agents and probabilistic aggregation to address the syndrome-level complexity of dementia beyond Alzheimer's-centric approaches.

Original authors: Yaling Shen, Maja Christensen, Yiwen Jiang, Jenna Dennison, David Darby, Amy Brodtmann, Zongyuan Ge

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Yaling Shen, Maja Christensen, Yiwen Jiang, Jenna Dennison, David Darby, Amy Brodtmann, Zongyuan Ge

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to diagnose a complex medical condition like dementia. In the real world, this isn't a job for one person looking at one piece of paper. It's a team effort. A doctor needs to talk to the patient, interview the family, look at memory test scores, check handwriting samples, and review medical history. Often, some of this information is missing, messy, or comes from different sources.

For a long time, AI systems trying to help with this have been like a single, overworked detective who is only trained to solve one specific type of crime (Alzheimer's disease). They often fail when the case is messy, the data is incomplete, or the problem is broader than just that one disease.

This paper introduces Dementia-Agents, a new AI system designed to act more like a specialized medical board working together in a real clinic.

The "Medical Board" Analogy

Instead of one giant brain trying to do everything at once, Dementia-Agents breaks the job down into a team of five specialized experts, plus a manager. Here is how they work:

1. The Translator (The Data Agent)
Think of this agent as a clerk who takes a messy pile of hospital forms, numbers, and notes and turns them into a clear, organized story. Crucially, this clerk is smart enough to say, "Hey, we don't have the patient's blood pressure reading," and marks that spot as "missing" instead of guessing. They then pass the right parts of the story to the right expert.

2. The Five Specialists (The Expert Agents)
These are five different "doctors," each trained to look at a specific type of clue:

The Risk Profile Expert: Looks at the patient's background (age, gender) and lifestyle risks (like high blood pressure or diabetes).
The Cognitive Profile Expert: Analyzes the patient's test scores (like memory games or math problems) to see how their brain is functioning.
The Collateral Expert: Reads reports from family members or caregivers about how the patient is behaving at home.
The Linguistic Expert: Listens to how the patient speaks and what words they use.
The Visuographic Expert: Looks at drawings or handwriting samples to spot physical signs of decline.

3. The Manager (The Coordinator Agent)
This agent is the team leader. It listens to the predictions from all five specialists. It doesn't just take a simple average; it learns how much to trust each specialist based on the situation. If the family report is very detailed but the handwriting sample is blurry, the manager might weigh the family report more heavily. It then combines all these opinions to make the final diagnosis.

What Did They Actually Do?

The researchers tested this system on 1,066 real patients from actual neurology clinics. They didn't use perfect, made-up data; they used the messy, incomplete data that real doctors deal with every day.

They compared their "Team of Experts" against:

The "Super-Brain" (Monolithic Models): Single, massive AI models that try to do everything alone.
Other AI Teams: Previous attempts at using multiple AI agents for medical tasks.

The Results

The paper claims that Dementia-Agents won the race.

Better Accuracy: It was better at figuring out the stage of dementia (is it mild, moderate, or severe?) and the specific type (phenotyping) than the single "Super-Brain" models.
Handling Missing Data: Because the system was designed to handle missing information (like a real doctor would), it didn't get confused when data was incomplete.
Interpretability: Because the system uses specific experts, doctors can see why a decision was made. They can see that the "Cognitive Expert" was the main reason for a certain conclusion, rather than the AI just giving a black-box answer.

The Bottom Line

The paper argues that dementia is too complex for a single AI model to solve alone, especially when data is imperfect. By mimicking how human doctors collaborate—splitting the work among specialists and having a leader synthesize the findings—this new system provides a more accurate and trustworthy way to diagnose dementia in the real world.

Important Note: The paper focuses strictly on diagnosing and categorizing dementia based on existing clinical data. It does not claim to cure dementia, predict the future beyond the current assessment, or replace human doctors in the clinic. It is a tool to help organize and interpret the complex information doctors already collect.

Technical Summary: Dementia-Agents

Problem Statement

Current AI-driven approaches to dementia diagnosis are predominantly pathology-centric, focusing on Alzheimer's disease (AD) detection or progression modeling within well-curated research cohorts. These methods typically frame the problem as binary AD detection or three-stage AD progression using complete, fixed-modality datasets. However, real-world dementia diagnosis is a heterogeneous, syndrome-level process involving multiple phenotypes and etiologies beyond AD. Clinical assessments in routine practice are characterized by incomplete, heterogeneous data collected incrementally from diverse informants (patients, caregivers, clinicians) and varying modalities (demographics, cognitive tests, questionnaires, linguistic samples, and visuographic tasks). Existing multi-modal large language models (MLLMs) and medical multi-agent systems often fail to address these specific challenges, lacking the ability to handle missing data signals, route multi-party inputs effectively, or model the syndrome-level complexity required for real-world staging and phenotyping.

Methodology: Dementia-Agents Framework

The authors propose Dementia-Agents, a clinically aligned multi-agent framework designed to simulate a multi-disciplinary diagnostic workflow. The system operates on a real-world cohort of 1,066 patients and follows a three-step pipeline:

1. Clinical Data Translation and Routing

A Data Agent processes raw, structured clinical records (tabular data) and translates them into semantically faithful textual representations. Crucially, this agent preserves missing-data signals (e.g., marking items as "unknown" rather than imputing values) and routes specific information subsets to five domain-aligned expert agents:

Risk Profile Expert: Analyzes demographics and nine modifiable risk factors (vascular and non-vascular).
Cognitive Profile Expert: Processes serialized cognitive assessments (e.g., ACE-R), including orientation, memory, language, fluency, and visuospatial tasks, capturing the full task-response-score triad.
Collateral Expert: Evaluates caregiver-reported questionnaires (e.g., CBI-R, BEHAVE-AD), converting numeric ratings into severity descriptors while preserving missing-data semantics.
Linguistic Expert: Analyzes speech-derived fluency tasks (category and letter fluency) including word sequences and annotations.
Visuographic Expert: Processes image-based tasks (figure copying, clock drawing, sentence writing) including the digitized patient output.

2. Expert-Level Assessment

Each of the five expert agents employs a fine-tuned encoder-classifier pipeline.

Architecture: Experts use pretrained encoder backbones (Qwen3-VL-Embedding-8B for multi-modal, Qwen3-Embedding-8B for text-only) adapted via LoRA. Outputs are mean-pooled and projected through a two-layer MLP to generate logits.
Tasks: Each expert independently performs two prediction tasks:
- Staging: Multi-class classification (Healthy Control, Mild Cognitive Impairment, Dementia) optimized via class-balanced cross-entropy.
- Phenotyping: Multi-label classification across 12 syndromic categories (e.g., Vascular, Behavioral, LATE, specific PPA variants) optimized via weighted binary cross-entropy.

3. Probabilistic Diagnostic Aggregation

A Coordinator Agent aggregates the probabilistic outputs from the five experts.

Mechanism: The final decision is a weighted sum of expert predictions ( $p = \sum \alpha_s p_s$ ).
Optimization: The weights $\{\alpha_s\}$ are optimized on a validation set using diverse strategies (Bayesian optimization, derivative-free search, evolutionary algorithms, and gradient-based methods) to reflect the relative diagnostic contribution of each expert for the specific task.

Key Contributions

Clinically Aligned Framework: The paper introduces Dementia-Agents, the first multi-agent system explicitly modeling syndrome-level dementia diagnosis as a distributed process that mirrors real-world clinical workflows, moving beyond pathology-driven AD detection.
Structured Expert Decomposition: The system proposes a novel decomposition strategy that enables domain-specific evidence routing and coordinated prediction under conditions of missing data, while maintaining domain-level interpretability.
Empirical Validation: The framework is developed and validated on a large, real-world clinical cohort (1,066 patients) from two cognitive neurology services, demonstrating consistent performance improvements over existing baselines.

Results

The system was evaluated on a held-out test set using stratified sampling to preserve label distributions.

Performance Gains: Dementia-Agents achieved state-of-the-art results compared to monolithic MLLMs (e.g., Qwen3-VL, InternVL, Hulu-Med, LLaVA-Med) and prior medical multi-agent systems (MedAgents, MedAgentPro).
- Staging: Achieved an F1 score of 0.745 and Macro-AUC of 0.874, outperforming the best supervised monolithic MLLM baseline (F1 0.680) by 8.7%.
- Phenotyping: Achieved a Micro-F1 of 0.364, representing a 28.6% relative gain over the best baseline. It also showed significant improvements in Hit@3 (0.754), indicating better alignment with clinical differential diagnosis.
Ablation Studies: Leave-one-out experiments confirmed that each expert contributes non-redundant clinical signals.
- Removing the Cognitive Expert caused the largest drop in staging performance, highlighting the centrality of direct cognitive testing.
- Removing the Collateral Expert most significantly impacted phenotyping, underscoring the importance of caregiver-reported behavioral patterns for syndrome differentiation.
- Replacing the Coordinator with uniform weighting degraded performance, proving that expert contributions are task-dependent and not equally informative.
Weight Analysis: The optimized weights revealed that staging relies heavily on cognitive evidence, whereas phenotyping emphasizes collateral information, a pattern consistent with routine clinical diagnostic processes.

Significance

The paper claims that Dementia-Agents addresses a critical gap in AI-assisted dementia care by shifting the focus from pathology-specific detection to syndrome-level staging and phenotyping. By explicitly modeling the heterogeneity of real-world data and the multi-party nature of clinical assessments, the framework offers a more robust and interpretable solution for routine clinical settings. The results suggest that structured expert decomposition provides a superior inductive structure for modeling multi-source dementia assessments compared to monolithic models or generic multi-agent designs, offering a pathway toward more reliable, deployable AI tools for complex neurological syndromes.