Technical Summary: Med-CRAFT

Problem Statement

The construction of large-scale, high-quality, explainable, and reproducible datasets for data-intensive AI applications remains a significant bottleneck, particularly in multimodal medical scenarios. Current approaches often rely on fragmented scripts, manual annotation, and ad-hoc quality checks, resulting in datasets that are weakly traceable, difficult to configure, and disconnected from their source evidence. This is especially critical in multimodal medical question answering (QA) involving instructional videos. Unlike static image-based QA, video-based QA requires samples to be semantically consistent, grounded in temporal and visual evidence, and capable of supporting complex reasoning chains. Existing benchmarks often lack systematic mechanisms to control reasoning difficulty, trace the derivation of samples back to source videos, or audit the construction process.

Methodology: Med-CRAFT System

Med-CRAFT is proposed as an information system designed to transform raw medical instructional videos into evidence-grounded, configurable, and reproducible multimodal QA datasets. Rather than treating dataset construction as a one-time generation task, Med-CRAFT models it as a provenance-aware pipeline consisting of six major layers:

1. Multimodal Video Ingestion: Raw videos are segmented and processed using Optical Character Recognition (OCR), Automatic Speech Recognition (ASR), subtitle parsing, and frame sampling to create temporally aligned multimodal segments containing textual, visual, and temporal cues.
2. Medical Operation Knowledge Graph Construction: Large Language Models (LLMs) extract medical entities (e.g., instruments, actions, anatomical structures), procedural events, and relations from the extracted cues. These are normalized into a structured Medical Operation Knowledge Graph (KG) where nodes represent entities/steps and edges represent temporal, causal, functional, or evidential relations.
3. Evidence Retrieval and Binding: The system performs "backward binding," linking graph nodes and edges to concrete evidence units in the source video (e.g., specific time intervals, key frames, OCR spans, or ASR transcripts). This ensures every graph element is grounded in observable video content.
4. Configurable Graph Traversal: Users define traversal strategies (e.g., hop count, relation types, starting/ending node types) to generate reasoning chains. This allows for the controlled generation of questions ranging from simple one-hop recognition to complex multi-hop cross-modal reasoning.
5. QA Generation and Quality Control: Reasoning chains are verbalized into natural language question-answer pairs using LLMs, constrained by the graph semantics and evidence. Automatic filters check for consistency, medical plausibility, and evidence relevance before human review.
6. Human-in-the-Loop Data Management: A user interface allows experts to inspect extraction results, visualize knowledge graphs, verify evidence bindings, review QA samples, and manage configurations. All actions are logged as provenance records.

Core Data Models: The system relies on explicit data models for Video Segments, Knowledge Graphs (with provenance links), Evidence Objects (treated as first-class citizens), Reasoning Chains, QA Samples (enriched with metadata), Configuration Profiles, and Provenance Records. This structure enables bidirectional traceability: from a final QA sample back to the source video segment, and from a video segment forward to all derived artifacts.

Key Contributions

1. Med-CRAFT Information System: A comprehensive system for the explainable, configurable, and reproducible construction of multimodal medical QA datasets from instructional videos, integrating extraction, graph construction, evidence binding, and human review.
2. Evidence-Enriched Medical Operation Knowledge Graph: A novel intermediate representation that connects procedural semantics with temporal, spatial, textual, and visual evidence, enabling controllable reasoning generation.
3. Configurable Graph Traversal Strategies: Mechanisms to generate reasoning chains and QA samples with explicit control over reasoning hops, relation patterns, evidence requirements, and visual-dependency levels.
4. Human-in-the-Loop Data Management Interface: A unified interface supporting video ingestion, graph inspection, evidence verification, and process monitoring, backed by logging-based provenance tracking.
5. Comprehensive Evaluation Framework: An evaluation strategy covering system utility (efficiency, traceability, reproducibility), dataset quality (semantic correctness, evidence relevance), and benchmark difficulty (performance of multimodal LLMs across reasoning depths).

Experimental Results

The authors evaluated Med-CRAFT against manual workflows, prompt-only LLM workflows, and sequential AI pipelines using source videos from the M3-Med dataset.

• Efficiency and Cost: Med-CRAFT achieved the highest Grounded Valid QA (GVQ) Yield (5.2 accepted pairs per expert hour) compared to manual (1.2), prompt-only LLM (2.7), and sequential AI (4.1) workflows. It reduced expert effort per accepted QA pair from 49.3 minutes (manual) to 11.5 minutes and lowered the cost per pair significantly.
• Quality: Med-CRAFT approached the quality of manual construction, achieving an All-Pass Rate of 0.79 (vs. 0.87 for manual) and high scores in medical plausibility (4.67/5.0) and evidence entailment (0.89). In contrast, prompt-only LLMs struggled with evidence grounding (All-Pass Rate: 0.14).
• Traceability and Reproducibility: Med-CRAFT demonstrated provenance completeness of 0.98, significantly outperforming baselines (0.23–0.67). It maintained high rerun agreement (1.00) and lineage success (0.99) under normal conditions.
• Configuration Fidelity: The system successfully realized user-specified distributions for reasoning depth and question types with low absolute error (0.02–0.04) and Jensen-Shannon divergence.
• Downstream Diagnostic Utility: The generated dataset effectively differentiated between text-only, vision-language, and domain-specific medical models. It revealed that performance declines as reasoning depth increases and highlighted gaps in grounding answers to specific evidence, even for strong models.

Significance and Claims

The paper claims that Med-CRAFT demonstrates how information system design principles—specifically provenance modeling, configurable workflow management, process monitoring, and human-in-the-loop verification—can be integrated into data-centric AI dataset construction.

The authors argue that Med-CRAFT addresses the "black box" nature of current dataset generation by making the construction process observable, auditable, and controllable. By treating dataset construction as a managed data production process rather than a simple annotation activity, Med-CRAFT enables:

• Traceability: Every sample can be traced back to its source video, evidence, and reasoning path.
• Configurability: Researchers can systematically control the difficulty and evidence requirements of generated samples.
• Reproducibility: Dataset construction can be exactly reproduced under the same configuration and input conditions.
• Diagnostic Value: The resulting datasets serve as effective benchmarks for exposing limitations in current multimodal models regarding cross-modal reasoning and evidence grounding.

The paper concludes that while Med-CRAFT improves the controllability and traceability of dataset construction, it remains dependent on the reliability of underlying extraction models (OCR, ASR, LLMs) and requires human review for medical safety and plausibility. Future work aims to improve terminology normalization, temporal localization, and the integration of active learning loops.