⚡ electrical engineering

DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation

The paper introduces DREAM, a novel framework that leverages a two-stage adaptive multi-modal fusion mechanism to integrate retinal images with ophthalmologist-curated clinical keywords, achieving state-of-the-art medical report generation performance even with limited data.

Original authors: Nagur Shareef Shaik, Teja Krishna Cherukuri, Dong Hye Ye

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nagur Shareef Shaik, Teja Krishna Cherukuri, Dong Hye Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a highly skilled detective trying to solve a mystery, but you have two very different sources of information: a blurry photograph of a crime scene and a list of clues written by a witness.

The Problem:
In the world of medical AI, we have a similar situation. We want computers to look at retinal eye scans (the photos) and write a detailed medical report (the story).

Old AI models were like detectives who only looked at the photo. They often missed tiny, crucial details because the photos are complex and the "crime scenes" (diseases) can look very similar.
Newer, giant AI models (called Large Vision-Language Models) are like detectives who read the entire library of books before looking at the photo. They are smart, but they are so huge and heavy that they are slow, expensive to run, and sometimes they "hallucinate"—making up facts that aren't there because they are trying too hard to guess.
The Missing Piece: In real life, doctors don't just look at the eye; they also have a patient's file with specific keywords (e.g., "diabetes," "blurred vision," "family history"). Current AI struggles to mix the photo and these keywords together dynamically. It treats them like a static pile of data, not a conversation.

The Solution: DREAM
The paper introduces DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion). Think of DREAM as a super-smart, agile detective assistant that knows exactly how to use both the photo and the clues to write the perfect report, even if it doesn't have a massive library of books to memorize.

Here is how DREAM works, broken down into three simple steps:

1. The "Abstractor": The Translator

Imagine the photo is in "Visual Language" and the doctor's notes are in "Text Language." They don't speak the same dialect.

What it does: The Abstractor acts like a translator. It takes the doctor's keywords and uses them to "highlight" the important parts of the eye scan.
The Analogy: If the keyword is "bleeding," the Abstractor tells the AI, "Hey, look right here in the photo where the red spot is!" It makes the AI pay attention to the specific parts of the image that match the clues, ignoring the rest.

2. The "Adaptor": The Smart Switchboard

Sometimes the photo is very clear, and sometimes the doctor's notes are the most important part. A rigid system would treat both equally, which is a mistake.

What it does: The Adaptor is a dynamic switchboard. It has a "volume knob" for the photo and a "volume knob" for the text.
The Analogy: If the photo is blurry but the doctor wrote "severe swelling," the Adaptor turns the volume up on the text and down on the photo. If the photo is crystal clear but the notes are vague, it does the opposite. It decides, in real-time, which source of information is more trustworthy for that specific patient.

3. The "Contrastive Alignment": The Fact-Checker

Even with good translation and switching, AI sometimes gets creative and makes things up (hallucinations).

What it does: This module acts as a strict editor. During training, it constantly checks: "Does the story we are writing actually match the whole picture of the patient's condition?"
The Analogy: It's like a teacher grading a student's essay. If the student writes a beautiful story about a "broken leg" but the X-ray shows a "healthy knee," the Fact-Checker slaps a big red "F" on the paper and says, "No, the story must match the reality." This forces the AI to stick to the truth.

Why is this a big deal?

It's Lightweight: Unlike the giant, expensive AI models that need supercomputers, DREAM is like a compact, high-performance sports car. It runs fast on standard hospital computers.
It's Accurate: In tests, DREAM wrote better medical reports than any other model, including those that are 100 times bigger. It made fewer mistakes and caught more subtle diseases.
It's Flexible: It works not just on eye scans, but also on other medical images (like X-rays and MRIs), proving it's a smart tool for all of medicine, not just one specialty.

In Summary:
DREAM is a new way for computers to read medical images. Instead of just staring at a picture or blindly guessing, it listens to the doctor's notes, dynamically decides which clues matter most, and double-checks its work to ensure the final report is accurate, safe, and ready to help save sight.

1. Problem Statement

Automating the generation of medical reports for retinal images is critical for diagnosing conditions like diabetic retinopathy and glaucoma. However, current Large Vision-Language Models (LVLMs) face significant challenges in this specialized domain:

Data Scarcity & Overfitting: Medical datasets are small compared to natural image corpora, causing generalist LVLMs to overfit or miss subtle pathologies.
Static Fusion Limitations: Existing retinal captioning methods often use static fusion strategies (e.g., fixed concatenation) that cannot dynamically adjust the weight of visual data versus clinical keywords based on the specific context of an image.
Computational Cost: State-of-the-art generalist LVLMs (e.g., LLaVA-Med, mPLUG-Owl2) are computationally expensive, requiring billion-parameter backbones and multi-GPU setups, making them impractical for resource-constrained clinical environments.
Hallucinations: Models often generate factually incorrect reports or misidentify lesions due to a lack of grounding in clinical reality.

2. Methodology: The DREAM Framework

DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion) is a novel framework designed to generate high-fidelity medical reports using limited data. It employs a two-stage dynamic fusion mechanism and a contrastive alignment objective.

A. Representation Learning

Visual Encoder: Uses a pre-trained EfficientNetV2B0 enhanced by a Guided Context Attention (GCA) module. GCA captures both global context and fine-grained local details to enhance sensitivity to lesion-specific features.
Keyword Encoder: A Transformer encoder processes expert-defined clinical keywords (e.g., referral tags, structured metadata) to generate contextual embeddings, capturing nuanced relationships between terms.

B. Two-Stage Dynamic Fusion

The core innovation lies in how visual and textual modalities are integrated:

Abstractor Module (Dynamic Retinal Enhancement):
- Projects image and keyword features into a shared embedding space.
- Performs bidirectional cross-attention:
  - Visual-to-Language: Queries visual features with linguistic context to highlight pathology-relevant cues.
  - Language-to-Visual: Queries keywords with visual context to suppress keywords lacking visual support (mitigating hallucinations).
- Output: An enriched feature set ( $F_1$ ) that prioritizes pathology-relevant details.
Adaptor Module (Adaptive Multi-modal Fusion):
- Introduces a learnable modality indicator ( $M_{ind}$ ) acting as a soft gate. This allows the model to dynamically weigh the importance of visual vs. linguistic features for each specific sample rather than using a static strategy.
- Uses a decoupled cross-modal attention mechanism: While the Query projection is shared, the Key and Value projections remain modality-specific. This preserves the distinct statistical characteristics of dense pixel data and sparse categorical keywords while integrating them into a unified representation ( $F_2$ ).

C. Contrastive Alignment & Generation

Contrastive Alignment Module: To ensure the fused representation is grounded in clinical reality, the model uses an InfoNCE contrastive loss. This aligns the fused multi-modal representation with the global semantics of the ground-truth report, pulling matching pairs together and pushing non-matching pairs apart. This mitigates factual hallucinations.
Language Model: The final report is generated by LLaMA-X, a compact, efficient adaptation of LLaMA. It utilizes optimizations like Rotary Positional Encodings (RoPE), Grouped Query Attention (GQA), and SwiGLU feed-forward networks to ensure deployment efficiency on commodity hardware.

3. Key Contributions

Abstractor Module: Introduces bidirectional cross-attention to dynamically align visual features with clinical keywords, enhancing pathology detection and suppressing unsupported keywords.
Adaptor Module: Proposes a learnable modality indicator and decoupled cross-modal attention, enabling per-sample adaptive fusion rather than static concatenation.
Contrastive Alignment: Implements a global semantic grounding objective (InfoNCE) to align latent representations with ground-truth reports, significantly reducing factual errors.
Efficiency: Achieves state-of-the-art performance with a lightweight architecture (~0.36B parameters, 5 GFLOPs), making it suitable for single-GPU clinical deployment compared to billion-parameter baselines.

4. Experimental Results

The model was evaluated on the DeepEyeNet dataset (15,710 retinal images) and the ROCOv2 dataset for cross-domain generalization.

State-of-the-Art Performance: DREAM achieved a BLEU-4 score of 0.241 on DeepEyeNet, outperforming the previous best model (GCS-M3VLT) by 0.010 and the top fine-tuned LVLM (mPLUG-Owl2) by 0.027. It also achieved the highest BERT-F1 (0.91), indicating superior semantic relevance.
Ablation Studies: Incremental testing confirmed that each component contributes significantly:
- Adding the Abstractor improved BLEU-4 by 0.018.
- Adding the Adaptor improved BLEU-4 by 0.013 and significantly boosted CIDEr (semantic richness).
- Contrastive Alignment provided the final boost to 0.241 BLEU-4 and 0.91 BERT-F1.
Generalization: On the ROCOv2 radiology dataset, DREAM achieved a BLEU-4 of 0.228, demonstrating that its adaptive fusion principles generalize beyond ophthalmology.
Qualitative Analysis: Visualizations show DREAM's attention maps precisely focus on central pathological lesions, whereas baseline models often hallucinate incorrect patient details or misidentify pathologies.

5. Significance

DREAM represents a significant leap forward in automated medical reporting by addressing the trade-off between specialization and efficiency.

Clinical Trust: By dynamically integrating expert keywords and grounding outputs via contrastive learning, the model produces reports that are not only grammatically correct but clinically accurate and free from severe hallucinations.
Deployment Viability: Its lightweight design allows for integration into standard hospital infrastructure without the need for massive GPU clusters, facilitating access to specialized care in under-resourced regions.
Adaptability: The framework's ability to dynamically arbitrate between modalities based on image context makes it robust to variations in image quality and keyword availability, a critical requirement for real-world clinical workflows.