Technical Summary: CORTEX – A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) in medical imaging predominantly rely on a "direct response" paradigm, mapping inputs to answers in a single step without explicit intermediate reasoning. While Chain-of-Thought (CoT) reasoning has been adopted to generate intermediate steps, existing implementations in medicine are largely free-form (unconstrained natural language). This poses critical risks:

• Lack of Verifiability: Free-form traces vary widely in structure and depth, making it difficult to audit whether a correct answer was reached via systematic clinical logic or coincidental hallucination.
• Absence of Clinical Context: Existing 3D chest CT Question-Answering (QA) datasets (e.g., CT-RATE) often reduce dense radiology reports to answer-only pairs, discarding the patient history and examination indications that clinicians rely on for differential diagnosis.
• Evaluation Gaps: Current benchmarks evaluate models holistically based on final answer accuracy, lacking protocols to verify the validity of individual diagnostic steps.
• Structural Deficit: No existing 3D chest CT MLLM utilizes a structured, radiologist-inspired reasoning framework, leaving a gap in both the supervised data needed to train such models and the protocols needed to evaluate them.

2. Methodology

The authors introduce CORTEX (Clinically Organized Reasoning and sTructured EXplanation), a benchmark constructed from the CT-RATE dataset (1,304 patients, 3,039 examinations, 76,177 QA pairs). The methodology follows a three-stage pipeline:

A. Dataset Preprocessing and Context Integration

• Restructuring: Native CT-RATE categories are relabeled into three reasoning types: Open-Ended (oe), Closed-Ended (ce), and Report Generation (rg).
• Context Grounding: The pipeline reattaches clinical context (patient history or reason for examination) to the questions. In the base corpus, 48.1% of cases contained such context; these are concatenated with the query to ground the diagnosis in the interaction between findings and presentation.

B. Structured Reasoning Generation

Instead of free-form generation, the system synthesizes reasoning traces using frontier Large Language Models (LLMs) guided by clinician-designed prompts. Each trace is forced into a strict four-stage decomposition mirroring the radiologist's hypothetico-deductive workflow:

1. Task Understanding (tu): Restates the question, study type, and clinical context to anchor the task.
2. Visual Observation (vo): Narrates findings from the report (as a proxy for visual volume evidence) organized by anatomy. Crucially, it restricts observations to question-relevant anatomies and avoids fabrication for absent features.
3. Diagnostic Reasoning (dr): Forms candidate hypotheses and accepts/rejects them using evidence cited from the observation stage, creating an auditable decision path.
4. Answer Synthesis (as): Provides a concise rationale and the final answer consistent with the reference.

C. Quality Verification and Trace Selection

To ensure high-quality supervision data, a rigorous filtering process is applied:

• Rule-Based Gate: Traces must contain the four stage-tags (<task>, <observation>, <reason>, <answer>) in the correct order.
• Rubric-Based Scoring: An LLM judge scores surviving traces on five clinician-designed rubrics:
  1. Task Understanding
  2. Observation Fidelity
  3. Hypothesis Evaluation
  4. Reasoning Logic
  5. Answer Correctness
• Selection Strategy: "Task Understanding" and "Answer Correctness" serve as a hard gate (requiring scores of 8–10). The remaining traces are ranked by the mean score of the other three rubrics. For each question, the single highest-scoring trace across all model/temperature configurations is retained.
• Expert Validation: A random subset (1,000 questions) was independently reviewed by three board-certified radiologists, achieving 93% inter-rater agreement on correctness, validating the automated rubric system.

3. Key Contributions

1. Radiologist-Inspired Reasoning Framework: CORTEX structures 3D chest CT interpretation into four clinically grounded stages, constraining how evidence and hypotheses are handled to mirror clinical workflows.
2. Clinical Context Grounding: The dataset integrates patient history and examination indications into both questions and reasoning traces, reducing hallucination by grounding diagnosis in clinical presentation.
3. Clinician-Designed Evaluation Protocol: A five-rubric protocol scores each diagnostic step individually rather than the final answer alone. This protocol is used here to verify the benchmark but is designed to be reusable for evaluating future reasoning models.
4. Large-Scale Structured Dataset: The authors constructed a dataset of 76,177 validated reasoning traces (64,224 open-ended, 8,914 closed-ended, 3,039 report generation) selected from over 1.9 million generated candidates.

4. Results and Dataset Statistics

• Generation Scale: The pipeline utilized five frontier LLMs (MedGemma 27B, Qwen-3 32B, Baichuan-M2 32B, Gemini-3-Flash, Gemini-3.1-Flash) sampled at five temperatures, generating over 1.9 million candidate traces.
• Final Composition: The final dataset contains 76,177 traces.
• Validation: Automated rubric scoring showed that "Task Understanding" and "Answer Correctness" consistently saturated at high scores (8–10), while "Observation Fidelity," "Hypothesis Evaluation," and "Reasoning Logic" provided the variance necessary for ranking.
• Human Verification: Expert radiologist review confirmed the high quality of the selected traces, with 93% agreement on correctness.

5. Significance and Scope

The paper positions CORTEX not as a trained model, but as a foundational resource to enable the development of trustworthy 3D chest CT MLLMs.

• Bridging the Gap: It addresses the two primary obstacles preventing progress in reasoning-capable 3D CT MLLMs: the absence of structured reasoning supervision and the lack of a reasoning-specific evaluation protocol.
• Shift in Paradigm: By moving from holistic answer evaluation to stage-level verification, CORTEX aims to make radiology AI interpretable and auditable, ensuring that diagnoses are traceable to specific evidence in the scan.
• Future Work: The authors explicitly state that the current work focuses on the construction and validation of the benchmark. The next steps involve using this dataset to train the first reasoning-capable 3D chest CT MLLMs and evaluating them using the proposed stage-level protocol. The dataset and evaluation code are made publicly available to facilitate this future research.