Automated Fracture Image Captioning Using Multimodal Vision-Language Models: A Comprehensive Comparative Study on a Clinically Curated Dataset

Technical Summary: Automated Fracture Image Captioning Using Multimodal Vision-Language Models

Problem Statement
While deep learning has achieved expert-level performance in fracture detection and classification, the automated generation of natural-language radiology reports (image captioning) for musculoskeletal radiographs remains underexplored. Existing literature predominantly focuses on chest X-rays (e.g., MIMIC-CXR), leaving a significant gap in fracture-specific captioning. Fracture radiographs present unique challenges, including subtle visual features distinguishing fracture types, locations, and severities, as well as a specialized linguistic register that differs from pulmonary imaging. Furthermore, manual reporting is time-intensive and prone to error, particularly in resource-constrained settings. There is a lack of systematic comparative studies evaluating diverse vision-language architectures on fracture-specific datasets, and the conditions under which certain models fail to converge in low-data medical scenarios are not well-characterized.

Methodology
The study presents a comprehensive benchmark of nine encoder-decoder architectures trained on a novel, clinically curated dataset from Ibn Sina Medical College, Dhaka, Bangladesh. The dataset comprises 710 fracture X-ray images paired with expert-annotated captions generated by board-certified orthopaedic radiologists.

• Data Processing: The dataset was split into training (75%), validation (10%), and test (15%) sets using stratified random splitting. To mitigate overfitting on the limited dataset, a stochastic on-the-fly augmentation pipeline was applied during training, including horizontal flipping, rotation, brightness/contrast adjustment, Gaussian noise, color jitter, and coarse dropout.
• Model Architectures: The study evaluated ten model combinations (grouped into nine distinct configurations) spanning three paradigms:
  1. Modular Encoder-Decoder Systems: Six visual encoders (ViT-Base, Swin-Tiny, DeiT-Base, BEiT-Base, ConvNeXt-Tiny, CLIP-ViT) paired with GPT-2 decoders of varying capacities (Small, Medium, Large).
  2. End-to-End Vision-Language Pre-training (VLP) Models: BLIP-Base and GIT-Base.
  3. Pre-trained Baseline: A ViT-GPT2 checkpoint pre-trained on general image captioning.
• Training Protocol: All models were trained under a unified protocol using AdamW optimization, a cosine annealing learning rate schedule with linear warm-up, mixed-precision (FP16) training, and gradient clipping. Evaluation was conducted using standard natural language generation metrics: BLEU-1 through BLEU-4, METEOR, ROUGE-L, and a CIDEr-like consensus score.

Key Contributions

1. Dataset Curation: Introduction of the first expert-annotated fracture radiograph captioning dataset from a South Asian tertiary medical institution, consisting of 710 image-caption pairs.
2. Systematic Benchmarking: The first comprehensive comparative evaluation of nine encoder-decoder architectures for fracture captioning, conducted under identical experimental conditions to ensure fair cross-architecture comparison.
3. Failure Mode Analysis: A systematic characterization of three distinct failure modes observed in vision-language architectures applied to small medical datasets:
   • Trivial Solution Collapse: Observed in CLIP-based models where contrastive pre-training objectives led to degenerate token distributions.
   • Optimization Divergence: Observed when coupling compact encoders with excessively large decoders (e.g., ViT-Base + GPT2-Large), leading to training instability.
   • Pre-training Objective Misalignment: Observed in models like BEiT and GIT where masked image modeling or unified auto-regressive objectives were less transferable to dense clinical caption generation than cross-attention architectures.
4. Efficiency and Performance Insights: Empirical evidence demonstrating that vision-language pre-training with unified objectives (BLIP) substantially outperforms independently coupled encoder-decoder systems, while identifying Swin-Tiny as the most parameter-efficient solution.

Results

• Top Performance: BLIP-Base achieved the highest performance across all metrics (BLEU-4 = 0.3529, METEOR = 0.5297, ROUGE-L = 0.4640, CIDEr-like = 0.5051). Its success is attributed to its unified pre-training strategy that jointly optimizes image-text contrastive alignment, matching, and language modeling.
• Best Modular Configuration: DeiT-Base + GPT2-Medium emerged as the best-performing custom-assembled model (BLEU-4 = 0.3058), demonstrating the efficacy of knowledge-distilled transformers in the medical domain.
• Best Efficiency: Swin-Tiny + GPT2-Small offered the highest BLEU-4 per parameter efficiency (1.493 × 10⁻³) and the shortest training time, making it the preferred choice for resource-constrained deployment.
• Degenerate Models: Several configurations failed to converge meaningfully. CLIP-ViT + GPT2-Small collapsed to a trivial solution (BLEU-4 = 0.0030). ViT-Base + GPT2-Large suffered severe optimization divergence (validation loss = 7.3868). GIT-Base and BEiT-Base also underperformed significantly, suggesting their pre-training objectives are less aligned with the specific demands of fracture captioning on small datasets.
• Caption Length Calibration: Well-performing models generated captions with lengths closely matching the ground truth mean (25.9 words), whereas degenerate models exhibited systematic over-generation, producing captions 20–87% longer than the reference.

Significance and Claims
The paper claims that this work provides the first rigorous, controlled comparative study of vision-language models specifically for fracture radiograph captioning. The primary significance lies in demonstrating that pre-training objective alignment is more critical than raw parameter count; a 247M parameter model (BLIP-Base) substantially outperformed a 1,098M parameter configuration (ViT-Base + GPT2-Large).

The study offers practical guidance for the medical AI community:

• Architecture Selection: For high-accuracy requirements, unified VLP models like BLIP are superior. For resource-constrained environments, Swin-Tiny + GPT2-Small provides an optimal balance of performance and efficiency.
• Training Stability: The findings warn against coupling small encoders with very large decoders on limited medical data, as this leads to optimization divergence.
• Diagnostic Utility: Caption length calibration is proposed as a rapid, parameter-free diagnostic criterion to identify non-converging models before full metric computation.

The authors acknowledge limitations, noting that the dataset is relatively small and sourced from a single institution, which may limit generalization. They conclude that while the current metrics provide standardized benchmarks, future work must incorporate clinical expert evaluation and explore larger, more diverse datasets to fully validate the clinical utility of these automated systems.