Technical Summary: OncoTriad-QA

Problem Statement

Cancer diagnosis and characterization rely on integrating complementary evidence from radiology (macroscopic anatomy), pathology (microscopic architecture), and genomics (molecular alterations). However, existing medical Large Language Model (LLM) and Vision-Language Model (VLM) benchmarks predominantly focus on isolated modalities or narrow image-text tasks. This creates a significant evaluation gap: while models may recognize findings within a single modality, they remain untested on their ability to integrate heterogeneous evidence streams into a unified, patient-level oncology interpretation. Current oncology resources typically lack the alignment of imaging, histology, and molecular data required to assess cross-modal concordance, missing-modality robustness, or integrated clinical reasoning.

Methodology

1. Dataset Construction: OncoTriad-QA

The authors introduce OncoTriad-QA, a patient-level benchmark spanning approximately 9,281 TCGA patient cases across 32 cancer cohorts. The dataset aligns four primary evidence streams:

• Radiology: CT and MRI data from The Cancer Imaging Archive (TCIA).
• Pathology: Whole-slide histopathology (WSI) from TCGA.
• Genomics: Somatic mutations, copy-number alterations, DNA methylation, and bulk RNA-seq from the Genomic Data Commons (GDC).
• Clinical Metadata: Structured clinical labels and diagnostic reports.

Data Processing Pipeline:

• Radiology: Tumor regions are segmented using Medical SAM3. Representative slices are selected based on segmentation consistency and tumor area. Geometric, intensity, and shape biomarkers are extracted and summarized.
• Pathology: Representative Regions of Interest (ROIs) are extracted from gigapixel slides (using TCGA-UniformTumor-8K) to supplement pathology reports, as current LLMs cannot process full WSIs directly.
• Genomics: Raw molecular data is converted into structured text summaries covering gene expression, methylation, mutation status, pathway activity, and molecular subtypes.
• QA Generation: A source-grounded, LLM-assisted pipeline (utilizing GPT-5.4) generates questions based on the structured evidence.
  • Universal MCQs: Procedurally generated from curated ground-truth labels (e.g., tumor stage, grade, mutation status) to ensure controlled evaluation.
  • Case-Specific MCQs: Generated by creating contrastive "fingerprints" for a target case against three same-cohort cases to prevent language-pattern shortcuts.
  • Open-Ended Questions: Covering modality-specific findings, integrated diagnosis, cross-modal concordance, and uncertainty.
• Quality Control: The pipeline includes automated consistency checks and a subset of cases reviewed by clinical experts to refine guidelines regarding evidence-grounded wording and uncertainty handling.

The benchmark supports three evaluation settings: All-Available (full multimodal), Radiology-Only, and Pathology-Only, simulating real-world scenarios where modalities may be missing.

2. Model Architecture: OncoVLM

To demonstrate the benchmark's utility for training, the authors introduce OncoVLM, a reference multimodal model designed to ingest modality-native data rather than text summaries or low-resolution proxies.

• Encoders: The model utilizes frozen, modality-specific foundation models:
  • Radiology: MedSigLIP-448.
  • Pathology: UNI (for WSI patch sets).
  • Genomics: BulkFormer (RNA-seq) and CpGPT (DNA methylation).
• Projectors: Each encoder is paired with a learned projector to map outputs into the LLM embedding space. The pathology branch uses a Perceiver-style resampler to compress variable-length slide evidence into a fixed token budget.
• Backbone: A Qwen3.5-9B LLM.
• Training Strategy:
  1. Modality Alignment: Projectors are trained independently on modality-specific captions while encoders and the LLM remain frozen.
  2. Supervised Fine-Tuning (SFT): The LLM is adapted using Low-Rank Adaptation (LoRA) on the OncoTriad-QA training split, while encoders and projectors remain frozen.

Key Contributions

1. OncoTriad-QA Benchmark: A clinician-guided, patient-level benchmark aligning radiology, pathology, genomics, and clinical metadata across 32 cancer cohorts. It features 86.1k semantic questions (expanding to 182.6k materialized rows) covering modality-specific recognition, molecular grounding, staging, and cross-modal concordance.
2. OncoVLM Reference Model: A multimodal VLM that processes native clinical data (gigapixel pathology, multi-slice radiology, genomic sequences) through specialized encoders and projectors, capable of handling arbitrary modality combinations and missing modalities.
3. Universal Evaluation Framework: A protocol that evaluates multimodal cancer QA across arbitrary modality combinations, providing fallback mechanisms for models unable to consume full-scale imaging or molecular profiles, thereby mimicking real clinical settings.

Experimental Results

Experiments were conducted on the held-out test split, comparing OncoVLM against general-purpose models (GPT-5.4), medical VLMs (MedGemma-4B), and open-weight models (Gemma-4-E4B).

• Performance Gains: Fine-tuned OncoVLM significantly outperformed baselines. On the All-Available setting, it exceeded MedGemma-4B by an average of 10.7 points (combining MCQ accuracy and BERTScore-F1).
• Structured Tasks: OncoVLM showed the largest gains on Universal MCQs (e.g., +21.8 points over the projector-only base model), demonstrating effective instruction tuning for mapping multimodal evidence to standardized oncology labels.
• Open-Ended QA: While BERTScore-F1 was clustered across models, LLM-as-judge evaluations (using Claude Sonnet 4.6) showed OncoVLM preferred over GPT-5.4 in 70.2% of cases, particularly in tasks requiring cross-modal synthesis and handling uncertainty. OncoVLM was noted for adhering closer to available patient evidence, whereas GPT-5.4 occasionally introduced findings absent from the case summary.
• Modality Ablation: Results confirmed that different modalities provide complementary signals. The All-Available setting performed best on integrative tasks. Pathology-only inputs preserved performance on grade and morphology tasks, while radiology-only inputs remained competitive for radiology findings.
• Cohort Variability: Improvements were more pronounced in larger cancer cohorts (e.g., Breast, Kidney), while smaller cohorts showed higher variance, suggesting that cohort-specific supervision scale is a critical factor.

Significance and Claims

The paper claims that OncoTriad-QA addresses a critical gap in medical AI by moving beyond single-modality recognition to integrated, patient-level oncology reasoning. The results demonstrate that:

1. Existing general-purpose and medical LLMs are limited in their ability to integrate imaging, morphology, and molecular evidence without specific training on such data.
2. Instruction tuning on a multimodal, evidence-grounded benchmark enables models to replicate integrated clinical interpretations.
3. The benchmark and reference model provide a standardized framework for evaluating modality-specific, cross-modal, and missing-modality performance, which is essential for developing AI systems that reflect the complexity of real-world clinical oncology.

The authors emphasize that the benchmark is a research tool intended to advance multimodal oncology AI, noting limitations regarding the retrospective nature of the data and the need for prospective validation before clinical deployment.