FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment

Technical Summary: FAIR_XAI

FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment investigates the reliability, bias, and interpretability of Vision-Language Models (VLMs) when used for clinical depression prediction.

1. Problem Statement

The integration of Multimodal Foundation Models (specifically VLMs) into mental health assessment offers scalable potential for monitoring wellbeing. However, these models act as "black boxes," creating a transparency-fairness dilemma. There is a critical lack of research regarding:

• How zero-shot VLM performance shifts between controlled laboratory settings and unconstrained, naturalistic clinical environments.
• Whether systematic demographic biases (gender and race) are inherent to specific VLM architectures.
• Whether Explainable AI (XAI) interventions can actually mitigate bias or if they merely provide "procedural transparency" without improving equitable outcomes.

2. Methodology

The researchers implemented a dual-pipeline framework to evaluate two state-of-the-art, small-scale VLMs: Qwen2-VL-2B and Phi-3.5-Vision.

• Datasets:
  • AFAR-BSFT: A controlled laboratory dataset (dyadic coaching sessions).
  • E-DAIC: A naturalistic, "in-the-wild" dataset (semi-clinical interviews).
• Dual-Pipeline Approach:
  1. Generative Classification Pipeline: A zero-shot approach where multimodal features (audio, visual, and text) are serialized into a natural language prompt. The model is asked to provide a reasoning trace in a <thinking> block before a <classification> label.
  2. Embedding-Based Pipeline (E-DAIC only): A supervised approach where frozen VLM text encodings are fused with audio and facial MLP embeddings, passed through a classification head.
• Intervention Framework:
  • Prompt-Based Intervention: Appending Chain-of-Thought (CoT) directives and explicit "fairness rule sets" (e.g., "ignore demographic factors") to the generative prompt.
  • Counterfactual Fairness Algorithm (CFA): A training-time intervention for the embedding pipeline that uses integrated gradients to penalize features highly correlated with sensitive attributes (gender/race).
• Evaluation Metrics: The study used a multi-dimensional audit including Predictive Validity (Accuracy, F1, Cohen’s $\kappa$), Outcome Fairness (Equal Opportunity $\Delta EO$, Disparate Impact $DI$), and Explainability Quality ($\Delta REF$, measuring the consistency of reasoning traces).

3. Key Contributions

• Cross-Domain Benchmark: Established how environmental noise and data distribution shifts impact VLM diagnostic reliability.
• Fairness Audit: Identified that bias is not universal but architecture-specific (e.g., Qwen2-VL being more gender-biased vs. Phi-3.5-Vision being more race-biased).
• XAI-Fairness Analysis: Demonstrated the "paradox of transparency," where improving the logic of an explanation does not necessarily result in a fairer prediction.
• Actionable Roadmap: Provided a set of clinical recommendations (Table 3) for the responsible deployment of VLMs.

4. Key Results

• Performance Disparity (RQ1): Performance varied wildly by environment. Phi-3.5-Vision achieved 80.4% accuracy on the naturalistic E-DAIC dataset, whereas Qwen2-VL collapsed to 33.9%. In laboratory settings, both models tended to over-predict depression (high sensitivity, low specificity).
• Architecture-Specific Bias (RQ2):
  • Qwen2-VL exhibited higher gender-based disparities.
  • Phi-3.5-Vision exhibited higher racial disparities.
  • Modality Fusion: Adding modalities did not always help; for Qwen2-VL, the "text + facial" configuration produced the highest observed bias ($\Delta EO = 0.418$).
• The XAI Paradox (RQ3):
  • On AFAR-BSFT, interventions improved the quality of reasoning ($\Delta REF$ decreased) but amplified racial bias and worsened disparate impact.
  • On E-DAIC, fairness prompting achieved "perfect" equal opportunity for Qwen2-VL, but only by sacrificing accuracy so severely that the model became clinically useless.
  • The CFA (Counterfactual Fairness) approach led to model collapse, where the model simply predicted "not depressed" for everyone to satisfy fairness constraints.

5. Significance

The paper serves as a cautionary tale for the clinical deployment of foundation models. It proves that procedural transparency (XAI) is not a proxy for equity. A model can provide a highly logical-sounding explanation for a decision that is fundamentally biased. The study concludes that future VLM development must move beyond zero-shot prompting toward domain-invariant fine-tuning and must jointly optimize for accuracy, demographic parity, and clinical viability.