WeNLEX: Weakly Supervised Natural Language Explanations for Multilabel Chest X-ray Classification

1. Problem Statement

Current approaches for generating Natural Language Explanations (NLEs) for medical image classification (specifically chest X-rays) rely on fully supervised learning. These methods require datasets where every image has a corresponding human-annotated explanation for each diagnosis. This approach presents three critical limitations:

• Lack of Faithfulness: The generated explanations reflect the human annotator's reasoning rather than the model's actual decision-making process. They are "plausible" but not "faithful" to the black-box model.
• Annotation Burden: Creating one NLE per image per diagnosis is extremely costly and time-consuming for medical experts.
• Inflexibility: Fully supervised models struggle to explain incorrect predictions (post-hoc) because ground-truth explanations for errors do not exist in the dataset. Furthermore, they cannot easily adapt to different target audiences (e.g., laypeople vs. radiologists) without re-annotating data.

The authors propose WeNLEX, a weakly supervised framework that generates NLEs requiring only a small database of ground-truth explanations per diagnosis (as few as 5), rather than per image.

2. Methodology

WeNLEX is designed to generate NLEs that are both plausible (coherent to humans) and faithful (reflecting the model's internal logic). It operates in two modes: Post-hoc (explaining a pre-trained classifier) and In-model (jointly training the classifier and the explanation generator).

Architecture

The system utilizes a Vision-to-Adapter and Soft Prompt mechanism to feed a frozen, pre-trained Text-only Transformer Encoder-Decoder (based on PubMedBERT) with:

1. Visual features from the chest X-ray.
2. The prediction vector (diagnosis and evidence labels) from the Vision Classifier.
3. The textual label of the diagnosis being explained.

Key Loss Functions

To ensure the four desired properties of NLEs (Plausibility, Faithfulness, Image-Relevance, Adaptability), WeNLEX employs a multi-objective loss function:

1. Plausibility Loss (Distribution Alignment):

   • Instead of matching generated NLEs to specific ground-truth NLEs (which is impossible in weak supervision), the model aligns the distribution of generated NLE embeddings with a small database of ground-truth NLE embeddings.
   • The authors compare two approaches: Wasserstein GAN (WGAN-GP) and Maximum Mean Discrepancy (MMD). Experiments show MMD minimization is superior, preventing mode collapse and producing more diverse explanations.

2. Faithfulness Loss (Feature Reconstruction & Classification):

   • Reconstruction: To ensure the NLE describes the specific image features used by the model, the generated NLE is passed through a "Text-to-Image" module to reconstruct an image. The features of this reconstructed image are compared to the original image features in the classifier's feature space (L2 distance).
   • Classification: The reconstructed image is fed back into the classifier. The classifier's output must match the original prediction vector (not the ground-truth label). This forces the NLE to contain the specific visual evidence the model used to make its decision.

3. Image-Relevance:

   • Guaranteed by feeding visual features directly into the NLE generator and enforcing the feature reconstruction loss.

4. Adaptability:

   • Achieved by simply swapping the small ground-truth NLE database. The authors demonstrate this by training a "Layman" version using simplified text, without changing the model architecture.

Training Strategy

• Post-hoc: The vision classifier is frozen; only the NLE generator and auxiliary modules are trained.
• In-model: The vision classifier and NLE generator are trained jointly. The classifier is updated via backpropagation through the NLE generator and Text-to-Image modules, effectively making the model "self-explanatory."

3. Key Contributions

• First Weakly Supervised Framework: WeNLEX is the first method to generate NLEs for multilabel chest X-ray classification using only a few ground-truth examples per diagnosis, drastically reducing annotation costs.
• Faithfulness via Reconstruction: The novel use of a "Text-to-Image" reconstruction loop in the feature space ensures explanations are faithful to the model's reasoning, not just human annotations.
• Performance Enhancement: The In-model setting improves the classification AUC by 2.21% compared to a standalone classifier, proving that interpretability can enhance downstream task performance.
• Audience Adaptability: The framework can generate explanations for non-experts (laypeople) simply by changing the reference database, without retraining the core architecture.

4. Experimental Results

Experiments were conducted on the MIMIC-NLE dataset (derived from MIMIC-CXR).

• Comparison with Fully Supervised Baselines:

  • WeNLEX (In-model) achieved a 67.34% AUC, outperforming the fully supervised baseline (64.08%) and the post-hoc WeNLEX (65.13%).
  • Faithfulness: Measured by "Flip %" (how often the model changes its prediction when the NLE-relevant regions are masked) and $\Delta p$ (logit difference). WeNLEX showed high flip rates, indicating the NLEs accurately capture the decision rationale.
  • Plausibility: Measured by CLEV (Clinical Evidence) and CXBS (BERTScore). The MMD-based approach achieved a CLEV of 9.401, significantly outperforming the adversarial approach.
  • Simulatability: Measured by how well a separate Vision-Language model (CheXagent) could reproduce the classifier's predictions using the NLE. WeNLEX achieved 90.46% accuracy in this metric.

• Ablation Studies:

  • Database Size: Using only 5 ground-truth NLEs per diagnosis yielded results comparable to using 10 or 20, demonstrating robustness.
  • Feature Layer: Using intermediate layers (DenseBlock 2) for reconstruction provided the best balance between faithfulness and diversity.
  • Layman Adaptation: A version trained on simplified text achieved a readability level of 10th grade (vs. 13th for medical text) while maintaining reasonable faithfulness metrics, though plausibility scores dropped due to the shift in vocabulary.

5. Significance

WeNLEX addresses the "faithfulness gap" in medical AI explainability. By decoupling the explanation generation from the need for exhaustive human annotations, it offers a scalable solution for real-world medical settings.

• Trust: It ensures that explanations reflect the AI's actual logic, which is crucial for debugging and clinical trust.
• Efficiency: It reduces the annotation burden by orders of magnitude (5 examples per class vs. 1 per image).
• Synergy: It challenges the notion that interpretability comes at the cost of performance; in fact, joint training improved the classifier's accuracy.
• Accessibility: It bridges the gap between technical AI outputs and non-expert users, facilitating better patient communication.