Technical Summary: OPENMEDREASON

Problem Statement

The high-stakes application of Large Vision-Language Models (LVLMs) in clinical settings demands more than correct final answers; it requires reasoning that is transparent, grounded in visual evidence, and anchored in clinical knowledge. Current medical LVLMs often lack this transparency, producing outputs without coherent rationales that are difficult to audit or trust. While recent advances in reasoning models utilize Reinforcement Learning (RL) and supervised fine-tuning (SFT), existing open resources for medical reasoning supervision are limited. Most public corpora provide only image–question–answer triples without reasoning traces, or they rely heavily on end-to-end LLM rollouts to generate chains of thought. These synthetic traces inherit the priors and stylistic regularities of the generating model, resulting in supervision that lacks diversity and fidelity to actual clinical evidence. Consequently, SFT may learn brittle reasoning patterns, and RL lacks a principled structure for refinement.

Methodology

The authors introduce OPENMEDREASON, a large-scale, open multimodal medical reasoning corpus, and OPENMEDREASON-Bench, a diagnostic benchmark.

1. OPENMEDREASON Curation Pipeline

The dataset comprises approximately 450,000 image–question–answer instances. The core subset (196,000 examples) is derived from the Open-PMC-18M corpus, consisting of biomedical figure-caption pairs. The curation process involves a rigorous multi-stage pipeline:

• Multi-Level Quality Filtering:
  • Visual Usability: Pixel-level checks filter out images with low resolution, poor layout, lack of clarity, or compression artifacts to ensure reliable perceptual grounding.
  • Text and Context: An LLM-based filter assesses textual adequacy, clinical relevance (excluding schematics, animal models, or non-human cases), image-text alignment, and reasoning readiness.
• Question Construction:
  • Task Assignment: Image-context pairs are assigned to one of 19 clinical task categories (e.g., Diagnosis, Management, Risk) organized into five families (Perception, Diagnosis, Management, Risk, Workup).
  • Constrained Generation: Questions are generated in a USMLE-style format. Strict constraints ensure the question cannot be solved from text alone (visual necessity), prevents answer leakage in the stem, and ensures the rationale is derivable from the source text.
• Reasoning Trace Supervision:
  • Generation: A two-stage process produces scientifically grounded reasoning traces. First, a medical LVLM drafts an explanation linking visual findings to the answer. Second, a refinement model (gpt-5-mini) aligns the draft with the source evidence, removes unsupported claims, and structures the output into a fixed format: Modality/Visual Target → Visual Evidence → Clinical Interpretation → Justified Answer.
  • Verification: A final verification step using an LLM ensures source consistency, answer justification, and reasoning utility. Traces that are fluent but weakly grounded are discarded.

2. OPENMEDREASON-Bench

To evaluate model capabilities beyond final-answer accuracy, the authors introduce a held-out benchmark of 1,500 samples. This benchmark decomposes reasoning into three axes:

• Perception: Image-grounded observations.
• Medical Knowledge: General clinical facts.
• Rationale: Case-specific inferential links.
  Evaluation uses a checklist of atomic unit questions to score models on presence (whether a claim is mentioned) and correctness (whether the claim is accurate), producing a joint trace score.

3. Training and Evaluation

The authors train a Qwen2.5-VL-7B backbone using a two-stage post-training pipeline:

1. Supervised Fine-Tuning (SFT): Trained on OPENMEDREASON tuples (image, question, answer, reasoning trace).
2. Reinforcement Learning (RL): Uses Group Relative Policy Optimization (GRPO) with verifiable rewards on a held-out split, optimizing for format compliance and answer correctness.

Key Contributions

1. OPENMEDREASON (Dataset): A large-scale, open multimodal medical reasoning corpus where reasoning traces are derived from curated, human-authored scientific articles rather than end-to-end LLM rollouts. It covers 19 clinical taxonomies and 8+ imaging modalities.
2. OPENMEDREASON-Bench (Benchmark): A capability-level evaluation suite that decomposes model performance into perception, medical knowledge, and reasoning, enabling fine-grained diagnosis of failure modes.
3. Empirical Evidence for Grounded Supervision: Demonstrates that source-grounded reasoning supervision significantly improves the SFT→RL pipeline for medical LVLMs, enhancing the model's ability to surface relevant visual and contextual evidence.

Results

• Performance Gains: Training with OPENMEDREASON yields a 20% average improvement in VQA accuracy over the base Qwen2.5-VL-7B model.
• Competitiveness: The resulting model achieves performance within 4.2% of the strongest comparable-scale medical LVLMs (e.g., Lingshu 7B) and surpasses larger models like MedGemma 27B on classification benchmarks.
• Capability Decomposition: On OPENMEDREASON-Bench, the model achieves the highest Reasoning Trace Score, outperforming leading closed-source baselines (e.g., gpt-5-mini) by 6.9 points. The largest gains are observed in presence, indicating the model successfully surfaces the required visual and clinical evidence.
• Reasoning Quality: In pairwise comparisons, the model's reasoning traces are preferred over the base model in 86.1% of cases. Pass@k analysis shows that SFT on OPENMEDREASON expands the model's reachable solution capacity, particularly on out-of-distribution problems.
• Expert Validation: A board-certified internist reviewed 100 random cases, confirming 100% answer correctness, 100% faithfulness to image context, and 100% clinical meaningfulness.

Significance

The paper argues that the effectiveness of SFT→RL pipelines in improving medical reasoning is heavily dependent on the availability of high-quality, source-grounded supervision. By providing a dataset where reasoning traces are anchored in scientific evidence rather than model priors, OPENMEDREASON enables the training of models that are not only more accurate but also more transparent and auditable. The work emphasizes that for clinical utility, models must be able to explicitly demonstrate how they arrived at an answer through a valid inferential chain. The open nature of the dataset, benchmark, and trained 7B model allows academic and clinical groups to study model behavior and failure modes without reliance on proprietary, closed-source systems, fostering reproducible research in medical AI.

Note: The authors explicitly state that models trained on this dataset are not clinical tools and should not be used for diagnosis or treatment. The dataset inherits biases from the source literature, including over-representation of rare or pedagogically salient cases.