Parameter-Efficient VLMs for Gastrointestinal Endoscopy: Medical Image Generation and Clinical Visual Question Answering

Technical Summary: Parameter-Efficient VLMs for Gastrointestinal Endoscopy

Problem Statement
The deployment of advanced Artificial Intelligence (AI) in gastrointestinal (GI) endoscopy is currently hindered by three primary barriers: a scarcity of annotated medical datasets, strict patient privacy regulations that limit data sharing, and the prohibitive computational costs associated with conventional full fine-tuning of large models. These limitations result in AI systems that often suffer from bias, poor generalization to rare pathologies, and an inability to scale effectively in clinical practice. Furthermore, existing Visual Question Answering (VQA) models frequently produce "hallucinations"—plausible but incorrect clinical information—while synthetic data generation methods often fail to balance high fidelity with semantic alignment to medical prompts.

Methodology
To address these challenges, the authors propose a dual-pipeline framework utilizing Parameter-Efficient Fine-Tuning (PEFT) techniques. The system is designed to operate on the Kvasir-VQA dataset (6,500 images) and consists of two parallel components:

1. Clinical Visual Question Answering (VQA) Pipeline:

   • Architecture: The system adopts the Florence-2 vision-language model. It employs a frozen DaViT (Data-efficient Vision Transformer) encoder to extract hierarchical visual features and a BART-style decoder for text generation.
   • PEFT Strategy: Instead of retraining the entire model, the vision encoder is frozen, and only the language components are adapted. Clinical questions are prefixed with a domain-specific token (<MedVQA>) to activate relevant representations.
   • Mechanism: A cross-attention mechanism allows the model to dynamically attend to specific image regions based on textual queries, supporting spatial reasoning and object localization. The model is trained using token-level cross-entropy loss with AdamW optimization.

2. Privacy-Preserving Synthetic Image Generation Pipeline:

   • Architecture: This pipeline utilizes Stable Diffusion 2.1 as the base text-to-image model.
   • PEFT Strategy: Low-Rank Adaptation (LoRA) is integrated into the attention layers of the diffusion model. Specifically, the pre-trained weight matrices ($W_0$) are frozen, and low-rank trainable matrices ($A$ and $B$) are inserted such that $W = W_0 + BA$.
   • Optimization: The authors empirically determined that a rank-4 LoRA configuration offers the optimal balance between efficiency and representational power, significantly reducing the number of tunable parameters and the risk of overfitting on small medical datasets.
   • Process: The model generates high-fidelity GI images from clinical text prompts (e.g., "clinical colonoscopy image with a polyp") while preserving patient privacy by avoiding the use of real patient data for training the generative model.

Key Contributions
The paper outlines three primary contributions:

1. Implementation and Analysis of PEFT for VQA: The successful adaptation of the Florence-2 model to GI endoscopy using PEFT, accompanied by a detailed performance analysis demonstrating that fine-tuning on private datasets yields superior results compared to public datasets.
2. Robust Synthetic Image Generation Framework: The design of a LoRA-enhanced Stable Diffusion pipeline augmented with advanced prompt engineering to generate clinically accurate, privacy-preserving GI images for dataset expansion.
3. Comprehensive Evaluation: A thorough assessment of both methodologies in the context of GI diagnostics, supported by ablation studies on LoRA ranks and comparisons against state-of-the-art models.

Results

• VQA Performance: The PEFT-based Florence-2 model achieved a ROUGE-1 score of 0.92 and a ROUGE-L score of 0.91. BLEU scores improved by 0.08–0.24, indicating enhanced n-gram accuracy for medical terminology. The model demonstrated consistent performance across diverse question types (yes/no, counting, localization) and showed better generalization on private datasets compared to public ones.
• Synthetic Generation Performance: The rank-4 LoRA synthesis achieved a Fidelity score of 0.290, an Agreement score of 0.730, and a Fréchet BiomedCLIP Distance (FBD) of 1450.
• Comparative Analysis:
  • In VQA, the proposed method outperformed VQA-RAD and MedVQA benchmarks in ROUGE-L (0.910).
  • In image generation, while FLUX and Kandinsky 2.2 showed higher Fidelity or Agreement scores respectively, the proposed model demonstrated superior semantic alignment with a significantly lower FBD (1450) compared to Kandinsky (2060.58) and MSDM (1531.62).
• Efficiency: The PEFT approach reduced trainable parameters by approximately 89.9% and decreased training time to 4–5 hours on standard GPU hardware, enabling deployment on standard hospital infrastructure without specialized computational resources.

Significance and Claims
The authors claim that this framework establishes a robust solution for enhancing VQA and synthetic data generation in clinical AI. By combining PEFT with synthetic data, the approach addresses the critical bottlenecks of data scarcity and privacy. The lower FBD score indicates that the generated images possess better image-text coherence than competing models, making them valuable for expanding training databases without violating patient privacy.

The paper positions this work as a step toward democratizing access to high-quality medical AI, allowing for the integration of sophisticated diagnostic tools into healthcare organizations with limited computational resources. While the authors acknowledge limitations in complex numerical counting tasks, they assert that the system is ready for clinical pilot testing, particularly in educational contexts and as a diagnostic support tool, ultimately aiming to improve patient care through accessible and ethically aligned medical AI.