Noise-Aware Visual Representation Learning for Medical Visual Question Answering

Technical Summary: Noise-Aware Visual Representation Learning for Medical Visual Question Answering

Problem Statement
Medical Visual Question Answering (Med-VQA) aims to enable AI systems to interpret medical images and answer clinical queries, offering significant potential for decision support. Current state-of-the-art approaches typically connect frozen, off-the-shelf vision encoders (e.g., CLIP) with Large Language Models (LLMs) via lightweight mapping networks (such as MLPs or query-based transformers) to reduce computational costs. However, these methods often overlook the robustness of visual representations against noise. Medical images frequently contain noise and irrelevant variations introduced during acquisition and transmission. When standard projection-based adapters map these noisy visual embeddings directly into the LLM's input space, they propagate both clinically relevant information and noise, potentially degrading diagnosis and downstream analysis. Existing Med-VQA research has relatively underexplored the refinement of visual embeddings prior to their projection into the language model.

Methodology
The authors propose a noise-aware Med-VQA framework that incorporates a Denoising Autoencoder (DAE) before the visual mapping stage. The framework follows a two-stage training paradigm:

1. Stage 1: Denoising Representation Learning

   • Input: Visual embeddings ($v$) are extracted from a frozen CLIP vision encoder.
   • Corruption: During pretraining, Gaussian noise ($\epsilon \sim \mathcal{N}(0, \sigma^2I)$) is added to the clean embeddings to create corrupted inputs ($\tilde{v} = v + \epsilon$). The study evaluates noise levels of $\sigma \in \{0.1, 0.5, 1.0\}$.
   • Architecture: A denoising autoencoder, consisting of an encoder and a decoder with a dimensional bottleneck (compressing 512 dimensions to 128 via a 256-dimensional hidden layer), is trained to reconstruct the original clean embeddings from the corrupted inputs.
   • Objective: The model is optimized using a Smooth L1 reconstruction loss. The goal is to force the encoder to learn robust latent representations ($z$) that capture stable underlying structures while suppressing noise-related variations.
   • Outcome: After pretraining, the decoder is discarded, and only the encoder is retained for the next stage.

2. Stage 2: Generative Med-VQA Training

   • Mapping: The robust latent representations ($z$) produced by the frozen DAE encoder are projected into the LLM embedding space using a 3-layer Multi-Layer Perceptron (MLP). These are reshaped into "visual prefix tokens" ($P_v$).
   • Integration: These visual prefix tokens are concatenated with textual embeddings (question and context) to form the input sequence for a pretrained LLM (GPT-2 XL).
   • Fine-Tuning: The framework employs Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. This allows the LLM to adapt to the medical domain and the visual prefix tokens by updating only a small subset of attention parameters, while keeping the backbone LLM and the DAE encoder frozen.
   • Objective: The model is trained with a standard language modeling objective (cross-entropy) to generate answers autoregressively.

Key Contributions

• Two-Stage Noise-Aware Learning: The paper introduces a decoupled training paradigm where denoising representation learning is separated from generative alignment. This allows the model to explicitly learn robustness to irrelevant perturbations before engaging in cross-modal alignment.
• Structured Robustness Evaluation: The authors propose a systematic evaluation method where noise is injected into visual embeddings during inference to assess the resilience of different architectures (Baseline, Autoencoder, and Denoising Autoencoder) under both clean and corrupted conditions.
• Empirical Validation: The method is evaluated on two diverse benchmarks: SLAKE (radiology images) and PathVQA (histopathology images), covering both open-ended and Yes/No question types.

Experimental Results
The proposed DAE framework demonstrates superior robustness compared to baselines that directly project visual embeddings and standard Autoencoder (AE) variants without noise injection:

• Robustness Gains: On the SLAKE dataset, the DAE approach improved the average accuracy under noisy evaluation settings from 0.642 (Baseline LoRA) to 0.735 and from 0.473 (Baseline Frozen) to 0.713. On PathVQA, improvements were observed from 0.514 to 0.568 (LoRA) and 0.348 to 0.518 (Frozen).
• Clean Performance: While the DAE framework achieved competitive or state-of-the-art performance on clean benchmarks (e.g., matching the highest accuracy of 0.756 on SLAKE with LoRA), the primary advantage was observed under noisy conditions. The DAE model maintained higher accuracy as noise levels increased, whereas baseline and standard AE models showed significant performance drops.
• Optimal Corruption Level: Experiments indicated that moderate noise corruption during Stage 1 training ($\sigma = 0.5$) provided the best trade-off between clean accuracy and robustness to inference-time noise.

Significance and Claims
The paper claims that learning robust visual representations through embedding-level denoising is a promising strategy for enhancing Med-VQA systems, particularly those trained on relatively small medical datasets. The findings suggest that explicitly addressing noise and irrelevant variations in visual embeddings before they are mapped into the LLM embedding space can significantly improve the stability and effectiveness of answer generation. The authors note that while their approach uses Gaussian noise as a controlled proxy for corruption, future work could explore more realistic medical imaging artifacts. The study concludes that denoising-based representation learning offers a viable path to more resilient medical AI without requiring full retraining of large backbone models.