Structure Causal Models and LLMs Integration in Medical Visual Question Answering

1. Problem Statement

Medical Visual Question Answering (MedVQA) aims to answer questions based on medical images to assist in diagnosis and treatment. However, current approaches face two critical challenges:

• Data Bias and Spurious Correlations: Medical datasets are often imbalanced (e.g., more samples of "infiltration" than "effusion"). This leads to confounders where models learn statistical associations rather than true causal relationships. For instance, a model might associate a specific lung region with a disease simply because that region frequently appears with that disease in the training set, ignoring the actual pathology.
• Relative Confounding: A confounder for one question (e.g., a specific visual pattern) might be the true causal factor for a different question. Standard models fail to distinguish these relative biases.
• LLM Limitations: While Large Language Models (LLMs) improve text generation, they often produce irrelevant or non-standard answers in MedVQA due to a lack of structured guidance and an inability to align complex visual features with medical reasoning.

2. Methodology

The authors propose a Causal Inference Framework (CIF) integrated with a Prompt Module (PM) to address these issues. The core architecture combines Structural Causal Models (SCM) with modern LLMs.

A. Causal Analysis & Front-Door Adjustment

The authors model the MedVQA task using a Directed Acyclic Graph (DAG) where:

• $I$ (Image) and $Q$ (Question) are inputs.
• $A$ (Answer) is the output.
• $C$ represents unobserved confounders (biases in data) that create spurious paths ($I \leftarrow C \rightarrow A$).

To eliminate these confounders without observing $C$ directly, the authors employ Front-Door Adjustment:

1. Mediator Introduction: They introduce two mediators, $M_i$ (Visual Mediator) and $M_q$ (Textual Mediator), to block the backdoor paths.
2. Multi-Variable Resampling: They use Mutual Information to quantify correlations between modalities and employ a resampling strategy to ensure the mediators capture true causal relationships rather than spurious ones.
3. De-confounding Networks: Two specialized sub-networks, Visual De-confounding ($Y$) and Textual De-confounding ($H$), process the features.
   • $Y$ combines local visual features, global visual features, and fused features to extract true visual causal features ($F'_i$).
   • $H$ combines textual features and cross-modal features to extract true textual causal features ($F'_q$).
4. Causal Probability Estimation: The final answer probability is estimated as $P(A|do(I, Q))$, effectively cutting off the influence of confounders.

B. Prompt Module (PM)

To guide the LLM and ensure standardized, medically relevant outputs:

• Dynamic Prompt Generation: The system generates a series of auxiliary QA pairs based on the input image using pre-trained backbones (CLIP and LLaMA).
• Structured Instructions: The input to the LLM includes:
  1. Task instructions (e.g., "You are a doctor...").
  2. Generated exemplar QA pairs (acting as few-shot learning).
  3. The actual question.
• This strategy forces the LLM to focus on causal patterns rather than relying solely on textual priors.

C. Architecture & Training

• Backbones: Uses frozen CLIP ViT-B/16 for visual encoding and LLaMA (7B/13B) for text encoding.
• Feature Mapping: A Feature Mapping Module (FMM) extracts both global (Down-sampling Transformer) and local (Attention-based top-k tokens) features.
• Loss Functions:
  • Cross-entropy loss for closed-ended questions.
  • Negative log-likelihood for open-ended questions.
  • Causal Consistency Loss ($L_{cau}$): A KL-divergence loss ensuring the predictions from causal features ($F'$) align with the original features, reinforcing the causal alignment.

3. Key Contributions

1. Novel Causal Framework (CIF): The first framework to explicitly model and eliminate relative confounders in MedVQA using a multi-variable resampling front-door adjustment. It captures how different questions influence visual features differently.
2. Prompt Strategy for MedVQA: A novel prompt module that integrates generated QA pairs and structured instructions to guide LLMs, significantly improving their ability to handle complex medical reasoning and generate standardized answers.
3. Comprehensive Validation: Extensive experiments across five diverse MedVQA datasets (VQA-RAD, SLAKE, PathVQA, PMC-VQA, ProbMed) demonstrating robustness and generalizability.

4. Experimental Results

The method was evaluated on five datasets, outperforming State-of-the-Art (SOTA) methods like LLaVA-Med, MedVInT, and M2I2.

• Performance Gains:
  • VQA-RAD: Achieved 83.1% overall accuracy (13B model), surpassing the previous best by 1.5%.
  • SLAKE: Achieved 91.0% overall accuracy, a 2.3% improvement over SOTA. Notable gains were seen in complex question types (e.g., "Else" category improved by 25% for open questions).
  • PathVQA: Improved open-ended question accuracy by 1.4% and overall by 1.2%.
  • ProbMed: Showed massive gains, with the combined CIF+PM approach improving accuracy from a baseline of ~53% to 70.7% (13B model).
• Ablation Studies:
  • Removing the Front-Door Adjustment (FDA) resulted in significant performance drops, proving the gains are due to causal inference, not just parameter count.
  • The combination of CIF and PM yielded the best results, confirming their complementary roles (CIF removes bias; PM guides generation).
• Qualitative Analysis: Visualizations (Fig. 5 & 6) show that without CIF, models focus on spurious regions (e.g., misidentifying healthy lungs as cancerous due to statistical bias). With CIF, attention shifts to the true causal regions (e.g., correctly identifying pleural effusion in the lower left lung).

5. Significance

This work represents a paradigm shift in MedVQA by moving from statistical association to causal reasoning.

• Clinical Reliability: By eliminating spurious correlations, the model reduces the risk of misdiagnosis caused by data bias, making it more reliable for clinical decision support.
• Generalizability: The framework is robust across different imaging modalities (X-ray, CT, MRI) and disease types, addressing the "relative confounding" problem that plagues previous methods.
• LLM Integration: It provides a blueprint for effectively integrating LLMs into high-stakes medical domains by using causal features and prompt engineering to ensure outputs are both accurate and clinically valid.

In summary, the paper proposes a robust solution that leverages causal inference to "clean" medical data biases and prompt engineering to "guide" LLM reasoning, achieving state-of-the-art performance in medical visual question answering.