Technical Summary: Aloe-Vision – Robust Vision-Language Models for Healthcare

1. Problem Statement

The development of Large Vision-Language Models (LVLMs) for healthcare faces three critical bottlenecks that hinder their deployment in safety-critical clinical environments:

1. Data Scarcity and Quality: High-quality, medical image-text data is severely limited, restricting the scale and diversity required to train models that generalize across modalities, pathologies, and anatomical structures.
2. Evaluation Contamination: Existing medical VQA benchmarks (e.g., PathVQA, VQA-RAD) are often noisy, have been publicly available for years, and risk data leakage, leading to unreliable or overly optimistic performance assessments.
3. Robustness Failures: State-of-the-art LVLMs exhibit significant vulnerabilities to adversarial, ambiguous, or misleading prompts. In clinical settings, where models must interpret visual data alongside narratives, such failures are unacceptable.

The field currently lacks fully open, reproducible systems where all components (data, weights, training recipes) can be inspected to build trustworthy foundations for biomedical applications.

2. Methodology

2.1. Aloe-Vision-Data: A Balanced Training Mixture

To address data limitations, the authors constructed Aloe-Vision-Data, a large-scale, quality-filtered mixture designed for direct fine-tuning. The dataset integrates four distinct categories:

• Multimodal Medical: For visual clinical reasoning.
• General Multimodal: To preserve general visual capabilities.
• Medical Text-Only: For domain-specific knowledge.
• General Text-Only: To maintain conversational fluency.

Key Data Engineering Strategies:

• Token-Based Rebalancing: Instead of balancing by sample count, the mixture is weighted by loss-contributing tokens (assistant tokens only). This prevents longer samples (e.g., Chain-of-Thought traces) from dominating the training signal, ensuring uniform influence across modalities.
• Semi-Automatic Quality Filtering: A two-signal filtering process removes low-quality annotations:
  1. LVLM Tagging: A 72B model scores samples (1–5) based on coherence between image, question, and answer.
  2. Answer Perplexity: A 7B model computes perplexity; very low scores flag trivial answers (visible in images), while high scores indicate noise.
• Leakage Prevention: A 64-bit perceptual hashing (pHash) procedure removes near-duplicate images between training and evaluation sets, eliminating 6,273 samples to ensure evaluation integrity.
• Final Composition: The resulting mixture contains ~3.57M samples and ~640M loss tokens, with medical multimodal data comprising 41.8% and medical text-only 29.7% of the loss tokens.

2.2. Model Training

The Aloe-Vision family consists of two models (7B and 72B parameters) fine-tuned from the Qwen2-VL-Instruct family.

• Architecture: Retains the native multi-image interface and Qwen-style chat template.
• Training: Single-stage Supervised Fine-Tuning (SFT) on alternating user/assistant dialogues, optimizing next-token cross-entropy.
• Adversarial Robustness (AR): A variant (Aloe-Vision-AR) undergoes an additional SFT stage using adversarial samples derived from the FracAtlas dataset (a modality not present in the main benchmark) to test cross-modal robustness transfer.

2.3. Evaluation Framework

The authors introduce a comprehensive evaluation suite to avoid contamination and assess holistic quality:

• CareQA-Vision: A novel benchmark derived from Spanish medical and nursing residency entrance exams (MIR and EIR). It contains 301 expert-curated vision questions (2020–2024) translated to English. Crucially, these cases are unseen by the models, offering a contamination-free assessment of clinical reasoning.
• Adversarial Benchmark (HEART): Uses 24,311 samples to test robustness against four attack types: Sycophancy (prompted labels), Captions (incorrect text embedded in images), Prompt (incorrect text in prompts), and Legends (mismatched legends).
• Standard Benchmarks: Includes PathMMU, GMAI-MMBench, OmniMedVQA, and general benchmarks (MMMU, MMLU) to measure general capabilities.

3. Key Contributions

1. Aloe-Vision-Data: The first ready-to-train, balanced mixture explicitly curated for LVLM fine-tuning, released with full transparency regarding category ratios and token-based weighting.
2. Aloe-Vision Models: Openly released 7B and 72B medical LVLMs with full weights, training recipes, and data, achieving competitive performance with state-of-the-art (SOTA) systems.
3. CareQA-Vision: A contamination-resistant benchmark for evaluating performance on entirely unseen medical cases, addressing the leakage issues prevalent in older datasets.
4. Robustness Analysis: A systematic evaluation demonstrating that standard benchmark performance does not guarantee reliability under adversarial conditions, and that targeted robustness training can mitigate these vulnerabilities.

4. Results

4.1. Standard Performance

• Scaling Trends: Increasing model size from 7B to 72B yields consistent performance improvements across all benchmarks.
• Competitive Performance: Aloe-Vision-72B achieves strong results, often outperforming other open-source medical LVLMs (e.g., HuatuoGPT-Vision, Lingshu) on medical multimodal and text-only tasks.
• Generalization: The models perform particularly well on CareQA-Vision, indicating successful generalization to unseen medical distributions.
• Comparison with Proprietary Models: While GPT-5.2 leads on text-only benchmarks, several open models (including Aloe-Vision) surpass it on specific medical vision-language tasks (e.g., OMVQA, PathMMU), suggesting targeted domain adaptation can narrow the gap to proprietary systems.

4.2. Adversarial Robustness

• Vulnerability: Standard LVLMs (including SOTA models like Lingshu and Hulu-Med) suffer catastrophic performance drops when exposed to adversarial inputs, particularly caption-based attacks (e.g., Qwen2-VL-7B dropping from 51.6% to 0.2% accuracy).
• Effectiveness of AR Training: The Aloe-Vision-AR variants consistently outperform their standard counterparts under adversarial conditions. Notably, robustness learned from a single modality (FracAtlas) successfully transfers to unseen medical specialties and image modalities.
• Detection vs. Classification: Detection tasks (bounding boxes) proved more resilient to textual manipulation than classification tasks, suggesting spatial grounding offers partial protection.

4.3. Reasoning Capabilities

The study observed that models explicitly post-trained for reasoning (e.g., GLM-4.5V, Chiron-o1) did not consistently outperform standard models across the benchmark suite. This suggests current medical vision-language benchmarks are predominantly knowledge-based and may not yet require multi-step reasoning to achieve high scores.

5. Significance and Claims

The paper claims that Aloe-Vision provides a fully open and reproducible foundation for advancing trustworthy medical vision-language modeling. Its significance lies in:

• Reproducibility: By releasing the complete data mixture, training recipes, and weights, the authors enable the community to inspect, evaluate, and improve upon the system, addressing the "black box" nature of many SOTA medical AI efforts.
• Reliability: The work underscores that high accuracy on standard benchmarks is insufficient for clinical deployment. The introduction of adversarial testing and the demonstration of robustness training highlight the necessity of evaluating models under misleading conditions.
• Data Integrity: The rigorous leakage prevention and the creation of CareQA-Vision offer a path toward more reliable, contamination-free evaluation of medical AI.

The authors conclude that while Aloe-Vision achieves competitive performance, future research must systematically disentangle the contributions of model scale, data scale/quality, and training curricula to better characterize performance trade-offs in medical LVLMs.