Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge

This paper introduces EyExIn, a data-efficient framework that enhances retinal Vision Language Models by employing a dual-stream encoding strategy and a deep expert injection mechanism to bridge perception and reasoning gaps, thereby achieving state-of-the-art precision in ophthalmic diagnosis while preventing hallucinations.

The Gist

Imagine you are trying to teach a brilliant, well-read librarian (the AI) how to diagnose eye diseases by looking at photos of the back of the eye (retina).

The problem is that this librarian has read millions of books about general topics but has never actually looked at a real eye in a medical setting. When you show them a photo, they might guess the disease based on what they've read in books, rather than what they actually see in the picture. This leads to two big mistakes:

1. The "Blind Spot" (Perception Gap): They miss tiny, subtle clues (like a tiny burst blood vessel) because their eyes aren't trained for medical details.
2. The "Daydream" (Reasoning Gap): As they think harder, they start ignoring the photo entirely and just make up a diagnosis based on their general knowledge, often inventing diseases that aren't there or missing real ones.

The paper introduces a new system called EyExIn (Eye Expert Injection) to fix this. Here is how it works, using simple analogies:

1. The "Dual-Eye" Strategy (Expert-Aware Dual-Stream)

Instead of giving the librarian just one pair of eyes, EyExIn gives them two pairs of glasses working together:

• The "General Glasses": These look at the big picture. They see the overall shape of the eye, the color, and the main structures (like the optic disc). This is like looking at a landscape painting and seeing the mountains and rivers.
• The "Microscope Glasses": These are specialized for doctors. They zoom in on the tiny, dangerous details like micro-aneurysms or tiny leaks that the general glasses would miss.

The Magic Glue (Semantic-Adaptive Gated Fusion):
If you just mash these two views together, the tiny details might get lost in the noise of the big picture. EyExIn uses a smart "traffic cop" (a Gated Fusion module).

• If the librarian sees a healthy area, the traffic cop says, "Ignore the microscope, just look at the general view."
• If the librarian sees a suspicious spot, the traffic cop says, "Stop! Zoom in with the microscope glasses right now!"
  This ensures the AI focuses its attention exactly where the disease is, filtering out the background noise.

2. The "Anchoring" System (Deep Expert Injection)

Even with the right glasses, the librarian might still get distracted. As they start writing their report (the "reasoning" part), they might forget what they saw in the photo and start guessing based on their memory of textbooks.

EyExIn solves this with "Vision Anchors."
Imagine the librarian is writing a story on a long scroll. Every few paragraphs, a heavy, unbreakable anchor is dropped onto the scroll, pinning the current sentence to the original photo.

• How it works: The system takes the "Microscope" view and injects it directly into the middle of the AI's thinking process. It's like a persistent reminder that says, "Hey, don't forget! There is a leak right here in the image. Base your next sentence on THIS, not on what you think you know."
• This prevents the AI from "hallucinating" (making things up) because the visual evidence is physically "anchored" to its thoughts.

3. The Results: A Trustworthy Doctor

The researchers tested EyExIn against massive, famous AI models (like the "GPT-5" or "Gemini" of the future).

• The Old Way: The big AIs were confident but often wrong. They would miss tiny diseases or invent fake ones because they were relying too much on their "book smarts" and not enough on the actual photo.
• The EyExIn Way: Even though it is a smaller model, it acts like a seasoned specialist. It catches the tiny details, ignores the noise, and refuses to guess unless the photo proves it.

In a nutshell:
EyExIn is like taking a smart but inexperienced intern and giving them specialized medical glasses and a permanent tether to the patient's actual eye scan. This stops them from daydreaming and forces them to make diagnoses based strictly on what is actually there, making them a much safer and more reliable tool for doctors.

Technical Summary

Here is a detailed technical summary of the paper "Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge" (EyExIn).

1. Problem Statement

The paper addresses the critical limitations of current Large Vision Language Models (LVLMs) in the domain of ophthalmic diagnosis. While general-purpose LVLMs show promise, their clinical deployment is hindered by a lack of domain-specific knowledge, manifesting in two structural deficiencies:

• The Perception Gap: General-purpose visual encoders (pre-trained on natural images) fail to resolve fine-grained pathological cues (e.g., microaneurysms, subtle nerve fiber defects). They often pass ambiguous tokens to the language model, leading to missed diagnoses.
• The Reasoning Gap: In deep transformer layers, sparse visual evidence is progressively overridden by massive pre-trained language priors. This causes the model to "hallucinate" plausible but non-existent lesions or fabricate clinical text based on language patterns rather than actual image evidence, posing severe safety risks.

Existing solutions relying on "brute-force" data scaling (massive instruction tuning or RLHF) are impractical for ophthalmology due to the scarcity, privacy sensitivity, and high cost of expert-annotated fundus images.

2. Methodology: EyExIn Framework

The authors propose EyExIn, a data-efficient framework designed to anchor retinal VLMs with expert knowledge. The architecture introduces a Deep Expert Injection mechanism to bridge the perception and reasoning gaps.

A. Expert-Aware Dual-Stream Encoding

To address the Perception Gap, the visual input is processed through two decoupled streams rather than a single encoder:

1. General Stream (Anatomical Context): Uses a frozen foundation encoder (e.g., Qwen2.5-VL) to extract global features ($F_{gen}$), preserving macroscopic anatomical structures and holistic colorimetric variations (e.g., optic disc pallor).
2. Expert Stream (Pathological Semantics): Uses a contrastively pre-trained fundus foundation encoder to extract fine-grained features ($F_{exp}$) highly sensitive to subtle lesions. These are projected to match the general stream's dimension.

B. Semantic-Adaptive Gated Fusion

To integrate these streams without diluting fragile lesion signals, a Semantic-Adaptive Gated Fusion module is employed:

• It computes a token-wise weight map ($\alpha$) using a lightweight semantic router.
• The fusion is a convex interpolation: $F_{fused} = (1 - \alpha) \odot F_{gen} + \alpha \odot F'_{exp}$.
• Mechanism: The gate dynamically amplifies expert features ($\alpha \to 1$) in pathological regions while suppressing them ($\alpha \to 0$) in healthy background regions, maximizing the visual Signal-to-Noise Ratio (SNR).

C. Adaptive Deep Expert Injection (Vision Anchors)

To address the Reasoning Gap and prevent visual signal decay in deep layers, the framework bypasses standard prompt-level integration. Instead, it embeds fused visual features directly into intermediate LLM layers as persistent residual biases:

• Mechanism: For visual tokens at layer $l$, the model computes a spatial routing map ($g_l$) to detect representation decay.
• Injection: The expert features are injected as a residual bias: $H'_{vis} = H_{vis} + \tanh(\gamma_l) \cdot (g_l \odot F_{fused})$.
• Benefits:
  • Discrepancy-Aware Routing: Selectively bypasses normal anatomical backgrounds, refreshing only pathological representations when language priors threaten to override visual evidence.
  • Zero-Initialization: The scaling parameter $\gamma_l$ is initialized to zero, isolating the pre-trained LLM from uncalibrated projections during early training to ensure robust convergence and prevent catastrophic forgetting.

3. Key Contributions

1. Dual-Stream Architecture: A novel decoupling of visual representation into general anatomical and specialized pathological streams to resolve fine-grained features.
2. Semantic-Adaptive Gated Fusion: A dynamic module that isolates subtle lesions from background noise, optimizing the visual signal for the LLM.
3. Adaptive Deep Expert Injection: A mechanism that establishes persistent "Vision Anchors" in intermediate LLM layers, forcing the reasoning stack to remain strictly grounded in visual evidence rather than language priors.
4. Data Efficiency: The framework achieves state-of-the-art performance with limited data (150K images) compared to massive proprietary systems, utilizing parameter-efficient fine-tuning (LoRA).

4. Experimental Results

The model was evaluated on four benchmarks: TM4K (private clinical dataset), JSIEC, Retina, and ODIR.

• Performance: EyExIn (7B parameters) consistently outperformed massive proprietary systems (Qwen3-VL-Max, ChatGPT-5.2, Gemini3-Pro) and fine-tuned open-source baselines (LLaVA, Qwen2.5-VL).
  • Closed VQA: Achieved SOTA F1-scores of 78.07% on TM4K and 80.66% on JSIEC.
  • Open-ended VQA: Demonstrated superior precision (e.g., 96.15% on the Retina dataset), significantly reducing false positives and hallucinations.
• Ablation Studies:
  • Replacing simple addition with Gated Fusion improved Precision by ~12.5% by filtering noise.
  • Replacing unconditional injection with Adaptive Deep Injection improved Precision by ~7.5% by preserving syntactic fluency while anchoring pathology.
• Qualitative Analysis: In case studies (e.g., Central Serous Chorioretinopathy and Retinal Vein Occlusion), EyExIn correctly identified subtle detachments and quantitative metrics (e.g., Cup-to-Disc ratio) that proprietary models missed or hallucinated as "Normal Fundus."

5. Significance

This work represents a significant advancement in trustworthy medical AI. By explicitly addressing the structural deficiencies of general-purpose LVLMs through Deep Expert Injection, EyExIn demonstrates that high-fidelity clinical reasoning can be achieved without massive data scaling. The framework provides a robust solution for:

• Reducing Diagnostic Errors: Minimizing both false positives (hallucinations) and false negatives (missed subtle lesions).
• Clinical Trust: Ensuring AI outputs are strictly grounded in visual evidence, a prerequisite for deployment in real-world ophthalmology.
• Resource Efficiency: Offering a viable path for specialized medical AI development in data-scarce environments.