💻 computer science

Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

This paper introduces a lightweight, frequency-domain dual-branch fusion framework for medical Visual Question Answering that leverages complementary spectral features from a frozen BiomedCLIP encoder and question-adaptive filtering to enhance multimodal alignment and improve performance on standard medical VQA benchmarks.

Original authors: Yusra Tariq, Rakesh Chandra Joshi

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yusra Tariq, Rakesh Chandra Joshi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery where the clues are hidden in two very different languages: one is a picture, and the other is a sentence. This is the world of "Visual Question Answering," a branch of artificial intelligence where computers try to act like detectives. Usually, when a computer looks at a picture, it scans it like a human does, piece by piece, looking for shapes and objects. But in the medical world, the clues are often much sneakier. A doctor doesn't just need to know that there is a spot on an X-ray; they need to understand the texture of that spot, how sharp its edges are, and how the density of the tissue changes around it. These details are like the subtle ripples in a pond versus the big splash of a rock.

For a long time, computers have been trying to solve these medical mysteries by mixing the picture and the question together in a "spatial" way—basically, looking at the image and the text side-by-side and hoping they click. But this paper suggests that approach might be missing the bigger picture. The authors propose a different way of thinking: instead of just looking at the image as a grid of pixels, what if we could listen to the image's "music"? In science, there is a tool called the Fourier transform that can turn a picture or a sound into a spectrum of frequencies. Think of it like taking a complex song and separating it into the deep, rumbling bass notes (which tell you about the overall structure) and the high-pitched, crisp treble notes (which tell you about the fine details and textures). This paper asks: what if we could tune into the right frequencies based on the specific question being asked?

The researchers at Amity University in India have built a new system called a "Frequency-Domain Dual-Branch Fusion" model to test this idea. Imagine the computer as a chef trying to cook a perfect meal (the answer) using two ingredients: a medical image and a patient's question. Most chefs just chop everything up and throw it in a pot together. This new system, however, acts like a master sound engineer. It takes the image and the question and converts them into "sound waves" (frequencies). Then, it uses the question as a remote control to adjust the volume of different parts of the sound. If the question asks about the general shape of an organ, the system turns up the "bass" (low frequencies) to hear the big structure. If the question asks about a tiny, fuzzy texture of a lesion, it turns up the "treble" (high frequencies) to hear the fine details.

The team trained this "sound engineer" AI using a massive library of medical images and questions. They found that by letting the question guide which frequencies to listen to, the computer could combine the visual and textual clues much more effectively than before. When they tested this on two famous medical puzzle sets, the system showed it could indeed find better answers. For example, on a large dataset called SLAKE, the system got the right answer about 69% of the time, which was a noticeable improvement over a version of the system that didn't use this frequency trick.

However, the authors are careful to note that this isn't a magic wand that solves everything yet. While the system is great at open-ended questions where it needs to describe textures or specific details, it sometimes stumbles on simple "yes or no" questions about whether an organ is present, especially in complex images with many overlapping parts. In those cases, the system sometimes got too excited about the high-frequency details and gave a messy answer instead of a simple one. The paper suggests that while tuning into the "frequency" of the data is a powerful new tool for medical AI, it still needs to be refined to handle every type of medical mystery perfectly. It's a promising new direction that shows listening to the right "notes" in an image can help computers understand the human body a little better.

Technical Summary: Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

Problem Definition
Medical Visual Question Answering (Med-VQA) requires models to answer clinical questions by grounding them in fine-grained visual evidence from medical images. Unlike general VQA, which often relies on object recognition, Med-VQA demands the interpretation of subtle anatomical structures, lesion boundaries, tissue textures, and diffuse pathological changes. Current state-of-the-art approaches primarily rely on spatial-domain cross-attention mechanisms to fuse visual and textual representations. However, the authors argue that these methods often entangle global structural patterns with fine-grained texture within the same representation, potentially failing to distinguish subtle textural abnormalities critical for clinical decision-making. Furthermore, spatial cross-attention incurs quadratic computational costs as token sequences grow.

Methodology
The paper proposes a Dual-Branch Frequency-Domain Fusion Framework that transforms visual and textual features into the frequency domain to perform question-guided spectral filtering and cross-modal interaction before reconstructing enhanced spatial representations. The architecture consists of four main stages:

Feature Extraction:
- Visual: A frozen BiomedCLIP vision encoder extracts patch-level features at two depths: an early intermediate block ( $T_e$ ) retaining local texture and edge information, and a final block ( $T_s$ ) carrying globally contextualized semantics.
- Textual: A BioBART encoder produces contextual question representations ( $H_q$ ).
- Pretraining Alignment: Prior to joint training, the early and late visual projections are aligned with the question representation using a symmetric InfoNCE objective to ensure they enter the fusion module within the same semantic space.
Dual-Branch Frequency-Domain Fusion (FSRU):
- Spectral Transform: The early visual tokens are reshaped into a 2D grid and processed via a 2D Real FFT, while the question sequence undergoes a 1D Real FFT.
- Uni-modal Spectrum Compression: Each branch is filtered by a bank of learnable complex filters. The mixture weights are derived from the pooled magnitude spectrum, allowing the effective filter to adapt per sample.
- Cross-Modal Emphasize-and-Suppress: A gating mechanism modulates one branch's spectrum based on the other. The question spectrum generates a gate to filter the image spectrum, and vice versa. These gates are bounded (between 0.1 and 0.9) to allow strong attenuation without fully discarding frequency content.
- Inverse Transform and Fusion: The gated spectra are mapped back to the spatial domain via inverse FFTs and combined with their pre-transform inputs via residual connections.
Gated Multimodal Integration:
- The frequency-enhanced streams are additively blended with the original late-semantic visual stream and the original textual stream using learnable scalar gates. This ensures the spectral contribution is neither negligible nor dominant.
- Source-type embeddings are added to distinguish visual from textual tokens before concatenation.
Answer Generation:
- The fused representation is fed into a BioBART decoder for autoregressive answer generation.
- An auxiliary classification head predicts the coarse answer type (yes/no, open, other) to regularize the encoder.
- The total loss function combines generation loss, answer-type classification loss, contrastive alignment loss, and a diversity term to prevent representational collapse.

Key Contributions

Dual-Branch Frequency-Domain Fusion: The authors introduce a module that performs spectral filtering and cross-modal gating conditioned on the input question, enabling adaptive selection of global low-frequency structure and fine-grained high-frequency detail.
Dual-Depth Visual Representation: The strategy combines early texture-sensitive features with late semantic features from a frozen BiomedCLIP encoder, aligning them with question representations via contrastive pretraining.
Empirical Validation: The framework is pre-trained on PMC-VQA and fine-tuned on VQA-RAD and SLAKE benchmarks, demonstrating that frequency-aware fusion improves performance over a non-frequency baseline.

Results
The model was evaluated on the VQA-RAD and SLAKE benchmarks using Exact Match (EM) and Token F1 metrics.

Performance: The proposed model achieved an overall EM of 40.21% on VQA-RAD and 69.21% on SLAKE.
Comparison: The authors note that while the model demonstrates competitive performance, it does not surpass current state-of-the-art methods (such as M3AE, M2I2, MUMC, and PeFoMed), which benefit from stronger multimodal pretraining strategies and specialized architectures.
Ablation Study: Removing the FSRU module resulted in a performance drop (e.g., EM on VQA-RAD fell from 40.21% to 36.66%), confirming the module's positive contribution to feature fusion.
Qualitative Analysis: t-SNE visualizations showed that embeddings became more compact and separable after frequency-domain fusion. Case studies indicated the model correctly identified anatomical structures and abnormalities missed by the ablated version, particularly for open-ended, texture-sensitive questions.
Limitations: The authors acknowledge that FSRU is not uniformly beneficial. In some cases, particularly with closed-ended organ-presence questions or dense multi-organ CT slices, the gating mechanism occasionally degraded predictions by over-emphasizing competing high-frequency content, leading to incorrect binary answers or malformed multi-token spans.

Significance and Claims
The paper claims that frequency-domain multimodal fusion is a promising direction for generative medical VQA. The primary significance lies in demonstrating that transforming representations into the frequency domain allows for the decoupling of global structure and fine-grained texture, enabling question-conditioned adaptive filtering. The authors position their work not as a replacement for existing state-of-the-art systems, but as a practical foundation for evaluating the effectiveness of spectral feature fusion. They conclude that while the current lightweight architecture yields improvements over a non-frequency baseline, future work involving larger vision-language backbones and more extensive medical pretraining is necessary to fully realize the potential of this approach and mitigate specific failure modes in dense visual contexts.

Technical Summary: Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

More like this