FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

1. Problem Statement

Context: Multi-modal Large Language Models (MLLMs) are increasingly used in safety-critical medical settings, such as generating radiology reports from chest X-rays or answering visual questions in dermatology.
The Issue: While powerful, these models often exhibit demographic disparities (bias based on age, race, and gender). In medical imaging, these biases are insidious because:

• Latent Leakage: Unlike text-only models where bias appears in explicit pronouns, medical images contain latent demographic signals (e.g., acquisition protocols, device types, or subtle anatomical variations correlated with race/age) that models exploit as "shortcuts."
• Inapplicability of Standard Methods: Traditional fairness techniques (resampling, reweighting) assume bias stems from class imbalance, which is often not the case in medical datasets. Adversarial methods often cause catastrophic forgetting, degrading overall clinical performance.
• Evaluation Gap: Standard fairness metrics (e.g., demographic parity on binary labels) do not apply to open-ended text generation, where the same clinical finding can be phrased in many ways, and omissions are as harmful as errors.

2. Methodology: FairLLaVA

The authors propose FairLLaVA, a parameter-efficient fine-tuning (PEFT) strategy designed to mitigate group disparities without compromising overall model performance.

Core Mechanism: Mutual Information Minimization

The central hypothesis is that model bias arises when hidden states contain information about sensitive demographic attributes (age, race, gender) that is not causally related to the diagnosis. FairLLaVA aims to make the model's internal representations demographic-invariant.

• Architecture:

  • Base Model: Uses a frozen pre-trained MLLM (e.g., LLaVA-Rad) with a frozen image encoder.
  • Parameter-Efficient Tuning: Only Low-Rank Adapters (LoRA) are added to the Language Model (LM) backbone, and the multi-modal projector is fine-tuned. This keeps computational costs low.
  • Demographic Attribute Classifier (DAC): A lightweight, variational MLP trained simultaneously to predict demographic attributes from the model's pooled hidden states.

• Training Process (Two-Stage Optimization):

  1. Stage 1 (Alignment): Standard visual-language alignment using Cross-Entropy (CE) loss to ensure the model follows instructions.
  2. Stage 2 (Fairness-Aware Fine-tuning):
     • DAC Training: The DAC is trained to predict the demographic attribute $a$ from the hidden state $h$ (minimizing classification loss $L_{DAC}$).
     • Mutual Information (MI) Regularization: The DAC is then frozen, and the main model (LoRA + Projector) is updated to minimize the Mutual Information between the hidden states and the demographic attributes. This is achieved by maximizing the probability that the DAC fails to predict the attribute from the hidden state (minimizing $L_{DIM}$).
     • Total Loss: $L_{total} = \lambda_1 L_{DAC} + \lambda_2 L_{DIM} + \lambda_3 L_{LLM}$.

• Key Innovation: By minimizing the mutual information, the model is forced to discard "demographic shortcuts" and rely solely on diagnostic visual cues, effectively regularizing the representation space to be invariant to sensitive attributes.

3. Key Contributions

1. FairLLaVA Framework: A novel, architecture-agnostic, parameter-efficient debiasing method that uses mutual information minimization to remove demographic shortcuts from MLLM hidden states.
2. Equity-Scaled Metric (ES-M): The authors extend fairness evaluation to generative tasks by proposing the Equity-Scaled Metric:
   $$ES\text{-}M = \frac{M_{all}}{1 + \Delta M}$$
   Where $M_{all}$ is the average performance across all groups, and $\Delta M$ is the performance gap (fairness disparity). This metric ensures that a model is not considered "fair" if it achieves low gaps simply by performing poorly across all groups.
3. Comprehensive Evaluation: Extensive experiments on three large-scale datasets:
   • MIMIC-CXR: Chest X-ray report generation (Age, Race, Gender).
   • PadChest: Chest X-ray report generation (Spanish/English).
   • HAM10000: Dermoscopy visual question answering (Age, Gender).

4. Experimental Results

FairLLaVA was benchmarked against state-of-the-art medical MLLMs (LLaVA-Rad, MedGemma, CheXagent), general-purpose MLLMs (Qwen, DeepSeek), and classical fairness baselines (Resampling, Reweighting, Adversarial Classifiers).

• Performance on MIMIC-CXR:
  • FairLLaVA-All (jointly debiasing all attributes) achieved 7 out of 12 best Equity-Scaled scores across metrics like BLEU, RadGraph-F1, and GREEN.
  • It significantly outperformed the "Adversarial MLP Classifier" baseline, which suffered from catastrophic forgetting (dropping overall GREEN scores by ~4x).
  • It reduced fairness gaps while improving or maintaining overall clinical report quality, unlike resampling/reweighting which often improved one group at the expense of others.
• Generalization: The method proved robust across different modalities (grayscale X-rays vs. RGB skin lesions) and languages (English vs. Spanish).
• Ablation Studies:
  • Layer Selection: Debiasing using middle-layer hidden states provided the best balance between reducing gaps and maintaining utility.
  • Joint Training: Training the DAC jointly with the main model (rather than pretraining it) prevented the aggressive removal of clinically relevant features, avoiding catastrophic forgetting.
• Counterfactual Analysis: On matched patient pairs (same clinical findings, different demographics), FairLLaVA showed significantly smaller fairness gaps, proving it relies on clinical evidence rather than demographic cues.

5. Significance and Impact

• Clinical Trust: By reducing disparities in diagnostic narratives, FairLLaVA addresses a critical barrier to the trustworthy adoption of AI in healthcare, ensuring equitable care across demographic groups.
• Efficiency: The method is parameter-efficient, requiring minimal computational overhead compared to full fine-tuning or heavy adversarial training, making it scalable for large medical models.
• Methodological Shift: It moves beyond simple frequency-based debiasing to address intersectional, latent dependencies in visual data, offering a new paradigm for fairness in generative multi-modal AI.
• Practicality: The approach is "plug-and-play," compatible with existing MLLM architectures and fine-tuning pipelines, facilitating immediate adoption in medical AI research and deployment.

In conclusion, FairLLaVA demonstrates that it is possible to achieve high-fidelity, equitable medical AI by explicitly regularizing the model's internal representations to be independent of sensitive demographic attributes, without sacrificing the quality of clinical reasoning.