Vision-language framework for multi-sequence brain magnetic resonance imaging

The paper introduces ReMIND, a vision-language framework trained on over 73,000 patient visits to enable robust, generalizable, and clinically coherent automated interpretation of multi-sequence brain MRI through large-scale instruction tuning and specialized report generation strategies.

The Gist

Imagine you are a detective trying to solve a complex medical mystery. You have a stack of different types of evidence: some are black-and-white photos, some are color scans, some show blood flow, and others show tissue density. To solve the case, you need to look at all these clues together, not just one, and then write a clear report explaining what's wrong and what to do next.

For a long time, computers have been good at looking at one picture at a time. But human brains are complex, and doctors (radiologists) need to look at a whole "movie" of different brain scans to make a diagnosis. This is where a new AI system called ReMIND comes in.

Here is a simple breakdown of how ReMIND works, using everyday analogies:

1. The Problem: The "Specialist Shortage"

Imagine a world where there are only a few expert detectives for every million people. These detectives are radiologists who look at brain scans. They are overworked, and because there aren't enough of them, people have to wait a long time for answers, or sometimes get a diagnosis that isn't quite right.

We need a way to give every doctor a "super-assistant" that can look at all the brain scan clues as quickly and accurately as a top expert.

2. The Solution: ReMIND (The Super-Assistant)

The researchers built ReMIND (Radiology–encoded Multimodal Interpretation for Neurological Disorders). Think of ReMIND not just as a calculator, but as a student detective who has been trained in two very specific ways:

• The "Library of Questions" (Instruction Tuning): First, the AI was fed over one million examples of questions and answers based on real brain scans. Imagine a student reading a million flashcards where a teacher asks, "What does this white spot on the scan mean?" and the student learns the correct answer. This taught the AI the language of medicine and how to think like a radiologist.
• The "Report Writing Class" (Supervised Fine-Tuning): Next, the AI practiced writing actual medical reports. It learned that a medical report isn't just a list of facts; it needs to be structured, clear, and follow a specific format (like "Findings" and "Conclusion"). This step made the AI sound professional and reliable.

3. The Secret Sauce: The "Safety Net" (MARC)

Here is the tricky part. Sometimes, an AI gets too confident and starts "hallucinating."

• The Hallucination: Imagine the AI looks at a scan that only has a T1 sequence (a type of photo). But because it's so smart, it might accidentally say, "I see a problem on the DWI scan," even though the DWI scan wasn't even in the folder! This is dangerous because it could lead a doctor to make a wrong decision.

To fix this, the researchers added a Safety Net called MARC (Modality-Aware Candidate Reranking and Correction).

• How it works: Before the AI hands you the final report, the Safety Net acts like a strict editor. It checks the report against the actual files in the folder. If the report says, "Look at the DWI scan," but the DWI file is missing, the Safety Net says, "Stop! You can't talk about that file. Delete that sentence."
• It does this without making the report sound robotic or confusing. It just ensures the AI only talks about what it actually saw.

4. The Results: A Global Test Drive

The researchers didn't just test this in one hospital. They tested ReMIND on data from:

• Boston (USA)
• Chennai (India)
• Various research centers

It was like taking a new car model to drive on snowy roads, sandy deserts, and busy city streets to see if it handles well everywhere. ReMIND performed consistently well across all these different "roads," proving it can understand brain scans from different machines and different types of patients.

Why This Matters

• Democratizing Expertise: It helps doctors who aren't brain specialists feel like they have a brain specialist right next to them.
• Consistency: It helps ensure that a patient in a small town gets the same quality of report analysis as a patient in a big city.
• Safety: By using the "Safety Net," it reduces the risk of the AI making things up, which is crucial when dealing with human health.

In short: ReMIND is a highly trained AI assistant that reads brain scans like a detective, writes reports like a professional doctor, and has a built-in editor to make sure it never lies about what it saw. It's a step toward making brain healthcare faster, fairer, and more accurate for everyone.

Technical Summary

1. Problem Statement

The interpretation of multi-sequence brain MRI is critical for diagnosing neurological disorders (e.g., neurodegenerative, vascular, neoplastic) but remains a bottleneck due to:

• Expert Scarcity: High-quality interpretation requires specialized neuroradiologists, whose availability is strained by rising disease prevalence.
• Cross-Sequence Reasoning: Radiologists must synthesize information across multiple imaging sequences (e.g., T1, T2, FLAIR, DWI, SWI) to form a coherent diagnosis. Existing AI models often struggle with this multi-modal integration.
• Protocol Variability & Hallucination: Clinical protocols vary widely. Current Vision-Language Models (VLMs) often suffer from "modality hallucinations," where they generate findings attributed to specific sequences (e.g., "restricted diffusion on DWI") even when those sequences are missing from the input, leading to clinically unsafe reports.
• Data Limitations: There is a lack of large-scale, diverse datasets pairing multi-sequence MRIs with structured radiology reports and question-answer (QA) annotations for training.

2. Methodology: The ReMIND Framework

The authors introduce ReMIND (Radiology–encoded Multimodal Interpretation for Neurological Disorders), a vision-language modeling framework designed for comprehensive multi-sequence brain MRI analysis.

A. Data Curation and Preprocessing

• Dataset: Trained on 73,000+ de-identified patient visits (850,000+ MRI sequences) from diverse cohorts (Gradient Health, Boston Medical Center, Lahey Hospital, Brain and Spine Hospital India, and NACC).
• Sequence Labeling: A deterministic rule-based pipeline standardized heterogeneous DICOM metadata into canonical labels (T1, T1CE, T2, FLAIR, SWI, DWI/ADC, MRA/MRV).
• Video Packing: 3D MRI volumes were converted into fixed-length 2D frame sequences (up to 256 slices) to leverage the model's native video understanding capabilities, concatenating all available sequences for a visit into a single visual block.
• Report Parsing: Radiology reports were segmented into seven canonical sections (Examination, Technique, Comparison, History, Indication, Findings, Impression) using an LLM-based multi-agent pipeline with rigorous quality control.

B. Model Architecture and Training Pipeline

ReMIND is built upon the Qwen3-VL-8B-Instruct (Q8B) foundation model, adapted via a dual-stage tuning pipeline:

1. Large-Scale Instruction Tuning: The model was trained on 1.04 million clinically grounded QA pairs generated from radiology reports. A multi-agent system (generator + reviewer) ensured questions were clinically meaningful, sequence-specific, and strictly grounded in the source text. This stage taught the model to interpret complex neuroimaging patterns and reason across sequences.
2. Supervised Fine-Tuning (SFT): The model underwent targeted SFT on aligned multi-sequence MRI–report pairs to optimize for the structured, concise, and factually constrained language required for radiology reporting.

C. Inference-Time Strategy: Modality-Aware Reranking and Correction (MARC)

To address modality hallucinations without retraining, the authors introduced MARC:

• Detection: A lexicon of modality-specific cues (e.g., "DWI," "SWI," "blooming artifact") is used to detect unsupported claims in generated reports.
• Reranking: For each input, $N=4$ candidate reports are generated. Each candidate is scored based on a violation weight (higher weights for high-risk modalities like Post-Contrast T1 or Diffusion).
• Correction: The candidate with the lowest violation score is selected. If it still contains violations, specific sentences are removed or neutralized, ensuring the final output is strictly bound to the available input modalities.

3. Key Contributions

• ReMIND Framework: A specialized VLM tailored for multi-sequence brain MRI, demonstrating that domain-specific tuning can outperform general-purpose scale in high-stakes medical imaging.
• Massive Instruction Dataset: Creation of a 1M+ QA dataset derived from real radiology reports, enabling robust cross-sequence reasoning.
• MARC Decoding Strategy: A novel inference-time mechanism that suppresses modality hallucinations while preserving linguistic fluency, solving a critical safety issue in medical AI.
• Comprehensive Evaluation: Rigorous testing across internal and external cohorts (including international data from India) covering diverse neurological conditions.

4. Results

• Visual Question Answering (VQA):
  • Instruction tuning significantly improved performance on contradiction and binary verification tasks (accuracy gains of 49–104%).
  • The full pipeline (Q8B+IT+SFT) achieved the highest average accuracy across all datasets (e.g., +31.61% on GH, +30.24% on BSH).
  • The model demonstrated strong ability to integrate complementary contrasts (e.g., using FLAIR for white matter changes and SWI for microbleeds) in multi-choice benchmarks.
• Radiology Report Generation (RRG):
  • Instruction tuning alone degraded report quality (lower GREEN and RaTEScore).
  • SFT was critical: The Q8B+IT+SFT model significantly improved lexical overlap (ROUGE-L: 0.16 $\to$ 0.37), semantic similarity (BERT-F1: 0.85 $\to$ 0.89), and clinical factuality (RadGraph-F1: 0.07 $\to$ 0.30).
• Hallucination Mitigation:
  • MARC successfully reduced violation rates (unsupported modality claims) across natural-missing and counterfactual settings.
  • There was a trade-off between hallucination suppression and language quality, but the Pareto frontier showed that MARC applied to the fine-tuned model (Q8B+IT+SFT) achieved the best balance, maintaining high clinical factuality while minimizing hallucinations.
• Generalization: The model maintained strong performance on external datasets (BMC, LHMC, BSH, NACC), indicating robustness to different institutions, scanner vendors, and patient demographics.

5. Significance

• Clinical Utility: ReMIND offers a scalable solution to democratize expert-level MRI interpretation, potentially reducing diagnostic disparities and cognitive burden on radiologists.
• Safety First: The MARC mechanism addresses the "hallucination" problem, a major barrier to deploying VLMs in clinical settings, by ensuring reports are faithful to the actual input data.
• Resource Efficiency: The study demonstrates that targeted, domain-specific engineering (instruction tuning + SFT + decoding constraints) can achieve high reliability without requiring "frontier-scale" (trillion-parameter) resources, making deployment feasible in resource-limited settings.
• Future Impact: This work sets a precedent for trustworthy, equitable, and structurally informed AI in neurological care, paving the way for prospective clinical trials.

Limitations: The study notes a lack of pediatric data and underrepresentation of certain racial/ethnic groups. Additionally, while MARC reduces hallucinations, residual errors in edge cases remain, and the framework has not yet been tested across the full spectrum of scanner vendors and acquisition protocols.