MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis

The paper introduces MM-NeuroOnco, a large-scale multimodal instruction dataset and benchmark for MRI-based brain tumor diagnosis that addresses annotation scarcity through an automated pipeline, demonstrating significant improvements in clinically grounded diagnostic reasoning via the proposed NeuroOnco-GPT model.

The Gist

🧠 The Big Picture: Teaching AI to Be a Brain Surgeon's Assistant

Imagine you have a brilliant but inexperienced medical student. They have memorized every textbook in the library, but they've never actually looked at a real MRI scan of a brain tumor. If you show them a picture and ask, "What is this?" they might guess correctly by luck, or they might confidently make up a story that sounds medical but is completely wrong (a "hallucination").

This paper introduces a new tool called MM-NeuroOnco to fix that problem. It's a massive "training school" and a "final exam" designed to teach Artificial Intelligence (AI) how to actually see and reason about brain tumors, rather than just guessing.

---

1. The Problem: The "Magic 8-Ball" of Medicine

Currently, most AI models for brain scans are like Magic 8-Balls.

• How they work: You shake the ball (feed it an image), and it gives you an answer ("It's a Glioma!").
• The flaw: They often get the right answer for the wrong reasons. They might have memorized that "round shapes usually mean tumors" without actually understanding why. They can't explain their reasoning, and they often fail when the picture is tricky.
• The missing piece: Real doctors don't just guess; they look at specific clues: Is the edge sharp or fuzzy? Is the swelling big? Is the signal bright or dark? Existing datasets didn't teach AI these specific "clues."

2. The Solution: A Massive "Medical Detective" Training Camp

The authors built MM-NeuroOnco, which is like a giant, high-tech training camp for AI.

• The Data (The Classroom): They gathered over 24,000 MRI slices from 20 different sources. Think of this as a library of 24,000 different brain cases.
• The Labels (The Teacher's Notes): Usually, these images only have a simple label like "Tumor." But here, they added 200,000 detailed instructions.
  • Old way: "This is a tumor."
  • New way: "This is a tumor. It has an irregular shape, fuzzy edges, and lots of swelling around it. Because of these specific clues, it is likely a Glioma."
• The Secret Sauce (The "Silver" Labeling): Getting human doctors to write 200,000 detailed notes would take forever and cost a fortune. So, the authors built a Robot Council.
  • They used three different powerful AI models to look at the same image.
  • If two robots agreed on a detail (e.g., "The edges are fuzzy"), they kept it.
  • If they disagreed, they threw the detail out to be safe.
  • This created a "Silver Standard" dataset—high-quality enough to train the AI, but built without needing a human to write every single word.

3. The Exam: The "I Don't Know" Option

One of the most clever parts of this paper is how they test the AI.

• The Old Exam: "Is this a Glioma, Meningioma, or Metastasis?"
  • The Trap: Even if the AI is confused, it must pick one. It might just guess the most popular answer.
• The New Exam (Rejection-Aware): "Is this a Glioma, Meningioma, Metastasis, or None of the above?"
  • The Goal: This forces the AI to admit, "I don't have enough evidence to decide." In real medicine, saying "I'm not sure, let's get more tests" is often the correct answer. This new test stops the AI from bluffing.

4. The Results: From Guessing to Reasoning

When they tested their new AI model (NeuroOnco-GPT) against the best commercial models (like the ones from Google or OpenAI):

• The Commercial Models: Even the smartest general AI models only got about 42% of the diagnosis questions right. They were essentially flipping a coin with a slight bias.
• The New Model: After training on MM-NeuroOnco, their model jumped to 51%.
• The "Chain of Thought" Boost: When they forced the AI to write out its reasoning step-by-step (like a detective writing a report: "I see a fuzzy edge, therefore..."), the accuracy jumped even higher.

🌟 The Takeaway Metaphor

Think of previous AI models as parrots. They repeat what they've heard in textbooks but don't understand the meaning.

MM-NeuroOnco turns the AI into a detective.

1. It gives the detective a magnifying glass (the detailed attributes).
2. It teaches them to look for specific clues (shape, edges, swelling).
3. It gives them a rulebook that says, "If you aren't sure, don't guess; say 'I need more info'."

This paper proves that if you teach AI to reason like a doctor rather than just recognize patterns like a photo editor, it becomes much safer and more reliable for real-world medical use.

🚀 Why This Matters

This isn't just about getting a higher score on a test. It's about trust. If an AI is going to help diagnose a brain tumor, we need to know why it made that decision. MM-NeuroOnco provides the blueprint for building AI that doesn't just "know" the answer, but can explain it, just like a human doctor would.

Technical Summary

1. Problem Statement

Current AI approaches for brain tumor diagnosis face three critical limitations:

• Semantic Gap: Existing datasets (e.g., BraTS) focus primarily on pixel-level segmentation (masking) but lack rich, clinically interpretable diagnostic semantics (e.g., edema severity, enhancement patterns, margin characteristics) required for reasoning.
• Evaluation Bias: Standard medical benchmarks often rely on closed-ended multiple-choice questions (MCQs) with obvious distractors. This encourages "shortcut learning," where models guess based on option distributions rather than genuine visual reasoning. Furthermore, the "forced-choice" paradigm fails to simulate real clinical scenarios where a physician must withhold judgment if evidence is insufficient.
• Annotation Cost: Generating high-quality, fine-grained radiological descriptions manually is prohibitively expensive and time-consuming, limiting the scale of instruction-tuning datasets.

2. Methodology

The authors propose a comprehensive framework consisting of a large-scale dataset, an automated construction pipeline, and a robust evaluation benchmark.

A. Dataset Construction (MM-NeuroOnco)

• Data Aggregation: The dataset aggregates 24,726 curated MRI slices from 20 diverse public sources (including BraTS, TCIA, and Kaggle), covering four modalities (T1, T1ce, T2, FLAIR) and eight tumor subtypes plus healthy controls.
• Automated Semantic Completion Pipeline: To overcome the lack of radiological text annotations, the authors developed a Multi-Model Collaborative Pipeline:
  1. Dual-Path Extraction: Two heterogeneous Vision-Language Models (VLMs) independently extract medical attributes (shape, margins, texture, edema, signal intensity) from MRI slices and masks.
  2. Consensus Fusion: A fusion strategy merges outputs. Conflicts in degree (e.g., "mild" vs. "moderate") are downgraded to a broader label (e.g., "Present"), while significant semantic conflicts are nullified to prevent hallucinations.
  3. Visual Verification (Subtraction-Only): A third high-performance VLM acts as a verifier. It is strictly prohibited from adding information; it can only reject or nullify untrustworthy fields. This ensures high-confidence "silver labels."
• Instruction Generation: The pipeline converts pixel masks and silver labels into structured Chain-of-Thought (CoT) reasoning and Visual Question Answering (VQA) pairs.
  • Closed-Ended: Adversarial distractors are generated to prevent simple elimination strategies.
  • Open-Ended: Questions require detailed diagnostic reasoning grounded in visual evidence.
  • Scale: The final dataset contains approximately 200,000 instruction samples.

B. Evaluation Benchmark (MM-NeuroOnco-Bench)

• Human-Annotated Ground Truth: A subset of 1,000 images is manually annotated by experts to ensure zero model-inference bias in the ground truth.
• Rejection-Aware Strategy: To address the "forced-choice" bias, every closed-ended question includes a fifth option: "None of the above." This forces models to verify if the provided options are actually supported by the image, simulating clinical uncertainty.
• Evaluation Metrics:
  • Closed-Ended: Accuracy.
  • Open-Ended: An LLM-as-a-Judge protocol (using Qwen3-80B) with a hierarchical scoring rubric that penalizes critical safety errors (e.g., laterality confusion) while tolerating minor semantic variations.

C. Proposed Model (NeuroOnco-GPT)

• The authors fine-tuned a Qwen3-VL-8B base model using the MM-NeuroOnco instruction dataset via Supervised Fine-Tuning (SFT) with LoRA.
• The model is trained to follow a structured clinical workflow: Observation (Modality/Location) $\rightarrow$ Reasoning (Morphology/Attributes) $\rightarrow$ Decision (Diagnosis).

3. Key Contributions

1. MM-NeuroOnco Dataset: A large-scale, multimodal instruction dataset with ~200k samples, bridging the gap between raw MRI pixels and high-level diagnostic reasoning.
2. Automated Semantic Pipeline: A novel "Dual-Path Extraction + Consensus Fusion + Subtraction-Only Verification" framework that generates reliable medical attributes without massive manual labeling.
3. Rejection-Aware Benchmark: A rigorous evaluation protocol that introduces a "None of the above" option to mitigate shortcut learning and better assess model reliability in high-stakes diagnostic settings.
4. NeuroOnco-GPT: A specialized domain-adapted model demonstrating that fine-tuning on this specific dataset significantly improves diagnostic reasoning capabilities.

4. Experimental Results

• Baseline Performance: Even the strongest general-purpose model, Gemini-3-Flash, achieved only 41.88% accuracy on diagnosis-related questions, highlighting the difficulty of the task. Medical-specialized models (e.g., HuLuMed, Lingshu) did not consistently outperform general models, suggesting that generic biomedical pre-training is insufficient without structured reasoning data.
• Impact of Rejection Mechanism: Introducing the "None of the above" option caused a ~10% drop in accuracy for all models compared to standard 4-option MCQs. This confirms that previous benchmarks overestimated model capabilities by allowing models to guess via elimination.
• Performance of NeuroOnco-GPT:
  • The fine-tuned model achieved a 27% absolute improvement in diagnostic accuracy over the base model.
  • With Chain-of-Thought (CoT) supervision, the model saw an additional 11.4% overall gain, proving that structured reasoning paths are more effective than shallow image-label associations.
  • It outperformed all other baselines (both general and medical-specialized) across dimensions like Diagnosis, Shape, and Spread.

5. Significance

• Clinical Relevance: The work shifts the focus from pure segmentation to diagnostic reasoning, aligning AI development with the actual workflow of radiologists who rely on specific imaging features (margins, edema, enhancement) rather than just tumor boundaries.
• Methodological Advancement: The "Subtraction-Only" verification pipeline offers a new paradigm for generating high-quality medical data at scale, reducing the reliance on expensive expert annotation while maintaining factual rigor.
• Evaluation Standard: The rejection-aware benchmark sets a new standard for evaluating medical AI, moving away from inflated scores derived from multiple-choice shortcuts toward a more conservative and realistic assessment of model reliability and uncertainty.
• Open Science: The dataset, benchmark, and code are publicly available, fostering further research in neuro-oncology and multimodal medical AI.