Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media

1. Problem Statement

During crisis situations (e.g., natural disasters), social media platforms like Twitter generate vast amounts of multimodal data (text and images) crucial for situational awareness and relief efforts. While existing methods can classify these posts into humanitarian categories (e.g., infrastructure damage, affected persons, rescue efforts), they suffer from two main limitations:

1. Lack of Transparency: Most high-performing models operate as "black boxes," making their decision-making processes opaque. This hinders trust and deployment in sensitive humanitarian contexts where transparency is mandated (e.g., by the UN).
2. Modality Imbalance in Explainability: Current explainable AI (XAI) approaches for crises focus primarily on extracting textual rationales. They largely ignore image interpretability or rely on post-hoc strategies (explaining a decision after the fact) rather than designing models that are interpretable by nature. Furthermore, annotating image rationales (e.g., bounding boxes or heatmaps) is costly and labor-intensive.

The core challenge addressed is: How can we build a multimodal classification system that is interpretable by design, extracts rationales for both text and images, and avoids the high cost of manual image annotation?

2. Methodology: VLTCrisis

The authors propose VLTCrisis (Vision and Language Transformer-based approach for Crisis-related tweet classification), an interpretable-by-design framework that utilizes Cross-Modal Rationale Transfer. The architecture operates in two main stages:

A. Rationale Extraction Network (Stage 1)

This stage learns joint representations and extracts rationales for both modalities.

• Encoder: The model uses a pre-trained ViLT (Vision-and-Language Transformer) to encode input tweets. Text is tokenized, and images are segmented into fixed-size patches. ViLT generates embeddings for tokens, patches, and a pooled representation of the whole sequence.
• Text Rationale Extraction: A GRU layer followed by a Sigmoid layer predicts binary labels for each text token (1 = rationale, 0 = non-rationale). This is trained as a multi-task learning problem:
  • Primary Task: Predict text rationales.
  • Auxiliary Task: Predict the humanitarian class label (to ensure rationales are task-specific).
  • Loss Function: A weighted combination of Binary Cross-Entropy (for rationales) and Cross-Entropy (for classification).
• Image Rationale Extraction (Cross-Modal Transfer): Instead of manual annotation, the model derives image rationales by mapping them to the predicted text rationales.
  • It computes an alignment score between the embeddings of predicted text rationale tokens and image patches.
  • The Inexact Proximal Point Method (IPOT) is used to perform soft matching (approximating optimal transport) between text tokens and image patches.
  • This generates an image rationale heatmap, where high scores indicate patches that visually support the text rationale and the predicted class.

B. Tweet Classification Network (Stage 2)

To ensure the model is "interpretable by design," the final classification is solely based on the extracted rationales, not the raw input.

• Masking: Non-rationale text tokens are replaced with a special token (*), and non-rationale image patches are blurred using the heatmap.
• Prediction: The masked text and blurred image are fed back into the ViLT encoder. The final class prediction is made based on these filtered inputs. If the model performs well on this stage, it proves the extracted rationales are sufficient and faithful to the decision.

3. Key Contributions

1. Interpretable-by-Design Framework: Unlike post-hoc methods, VLTCrisis forces the classification decision to rely entirely on extracted rationales, ensuring the explanation is intrinsic to the model's logic.
2. Cross-Modal Rationale Transfer: The paper demonstrates a novel method to generate image rationales (heatmaps) by transferring knowledge from annotated text rationales. This significantly reduces the annotation burden, as image rationales are derived automatically without human labeling.
3. Joint Multimodal Learning: The approach shows that text and images are complementary. Joint learning aligns concepts across modalities, yielding better representations (5–6% improvement) than unimodal approaches.
4. Zero-Shot Adaptability: The model demonstrates strong generalization capabilities, achieving ~80% accuracy on unseen crisis datasets without fine-tuning.

4. Experimental Results

Experiments were conducted on the CrisisMMD benchmark dataset (containing 5,263 training samples across 5 humanitarian classes).

• Classification Performance:

  • VLTCrisis achieved a Macro-F1 score of 0.822, outperforming state-of-the-art baselines.
  • It improved Macro-F1 by 2–35% compared to existing unimodal and multimodal baselines (e.g., VCNN, FMLS, IncBERT).
  • It significantly outperformed fine-tuned LLMs (like LLORA) on this specific, smaller, and imbalanced dataset.

• Rationale Quality (Plausibility):

  • Text: Achieved a Token-F1 of 0.826, outperforming GPT-4 (0.552) and other interpretable baselines.
  • Image: Human evaluation showed that the image patches extracted by VLTCrisis helped users identify classes with 74% accuracy, compared to 62% for a post-hoc baseline (RNET).
  • Human Evaluation: Multimodal rationales (text + image) reduced task difficulty for human annotators by 11–13% compared to single-modality rationales.

• Faithfulness:

  • Comprehensiveness: When rationales were removed (masked), performance dropped by 51.4%, indicating the rationales captured the most critical information.
  • Sufficiency: When only rationales were used, the performance drop was minimal (7.2%), proving the rationales alone are sufficient for accurate classification.

• Zero-Shot Performance: On a new, unseen dataset (DMD), the model achieved 80% accuracy and high alignment between text and image rationales (Q4 score of 0.83).

5. Significance

This paper addresses a critical gap in humanitarian AI by moving beyond "black box" performance to trustworthy, transparent decision-making.

• Operational Impact: By providing clear visual and textual evidence for why a tweet is classified as "infrastructure damage" or "rescue effort," humanitarian organizations can trust and act on automated alerts more quickly.
• Efficiency: The cross-modal transfer technique solves the "annotation bottleneck" for image explanations, making scalable, explainable crisis monitoring feasible.
• Generalizability: The model's ability to perform well in zero-shot scenarios suggests it can be rapidly deployed for emerging crises where labeled data is scarce.

In summary, VLTCrisis establishes a new standard for multimodal crisis informatics by proving that high accuracy and deep interpretability can be achieved simultaneously through a design that leverages the synergy between text and image rationales.