Interactive Mars Image Content-Based Search with Interpretable Machine Learning

Technical Summary: Interactive Mars Image Content-Based Search with Interpretable Machine Learning

Problem Statement
The NASA Planetary Data System (PDS) hosts millions of images from planetary missions, creating a data archive where manually searching for specific products is infeasible. While content-based classification systems exist to automate this search, standard deep learning models often function as "black boxes," lacking transparency regarding the visual evidence used for decision-making. This opacity creates a gap between the mental models of Atlas users (scientists and the public) and the classifiers, hindering trust and preventing users from validating the evidence. Furthermore, existing post-hoc explanation methods (e.g., GradCAM) can be unreliable in the presence of repetitive patterns common in planetary imagery, such as rocks or terrain, and fail to explain why a region was attended to, only where.

Methodology
The authors propose replacing the non-interpretable content-based classifier on the PDS Image Atlas with an inherently interpretable system based on the Prototypical Part Network (ProtoPNet) architecture (Chen et al., 2019).

• Architecture: The system utilizes a standard deep network backbone (VGG19 or ResNet18) for feature extraction, followed by a prototypical layer. This layer learns a set of prototypes (exemplars) that represent each image class. A fully connected layer then weights these prototypes to make the final classification.
• Interpretability Mechanism: For a test image, the system identifies regions most similar to the learned prototypes. These regions are visualized as bounding boxes or heatmaps, showing the specific evidence the model used. The classification decision is transparent: it is a linear combination of the similarity scores between the test image patches and the prototypes.
• Novel Enhancement (Diversity Loss): The authors identified that standard ProtoPNet training often results in a lack of diversity among prototypes (e.g., multiple prototypes looking identical or originating from the same training image). To address this, they introduced a diversity-enhancing term ($Div$) to the loss function. This term minimizes the similarity between distinct prototypes within the same class, ensuring the model learns a broader range of visual evidence.
• Training Strategy: The model, termed Proto-MSLNet, was trained on the MSL surface dataset (images from the Curiosity rover's Mastcam and MAHLI instruments). The training process employs a temporal split (training on past sols, validating/testing on future sols) to simulate realistic generalization. The total loss function combines cross-entropy, clustering cost, separation cost, and the new diversity cost.

Key Contributions

1. Deployment of an Interpretable System: The paper outlines a plan to deploy Proto-MSLNet on the PDS Image Atlas, replacing the existing non-interpretable classifier to support scientific discovery and user curiosity.
2. Diversity-Enhanced Training: The authors introduce and evaluate a diversity cost term that significantly improves the variety of evidence utilized by the classifier, particularly for classes with initially low prototype diversity (e.g., "Night Sky").
3. Quantitative and Qualitative Evaluation of Evidence: Unlike prior work that relies on agreement with human annotations, this study quantifies the correctness and diversity of the learned prototypes. They demonstrate that the system can identify erroneous evidence (e.g., background features acting as negative evidence) and mislabeled data.
4. User Feedback Loop: The proposed system includes a mechanism for users to report misclassifications or erroneous evidence, creating a feedback loop to refine the model.

Results

• Performance Trade-off: The interpretable Proto-MSLNet models showed a slight drop in test accuracy (approximately 2.5 percentage points) and an increase in the abstention rate (images where the model is not confident enough to classify, rising from ~15.5% to ~24.5% for ResNet18) compared to their non-interpretable counterparts. This is attributed to the constraints imposed by the prototypical layer.
• Diversity Improvements: The inclusion of the diversity loss term resulted in a 2 percentage point rise in test accuracy and significantly increased the diversity of prototypes for the "Night Sky" class (from ~4 unique prototypes to a higher count), while classes already rich in diversity (e.g., "Float Rock") saw no change.
• Qualitative Insights:
  • Correctness: The system successfully identified that "Night Sky" prototypes focused on emptiness, leading to misclassification of "Sun" images when the sun was not prominent. Conversely, correct classifications highlighted the bright, round shape of the sun.
  • Negative Evidence: The model learned to use background features as negative evidence (e.g., rocky backdrops with negative weights) to distinguish calibration targets.
  • Class Overlap: Analysis revealed overlaps between classes like "Wheel Joint" and "Wheel," attributed to camera positioning and rotation angles, and identified mislabeled instances in the dataset.

Significance and Claims
The paper claims that an inherently interpretable system is vital for bridging the gap between user mental models and planetary image classifiers. By making the evidence transparent, the system allows users to:

1. Validate the visual cues used by the classifier.
2. Identify erroneous evidence or data quality issues through user adjudication.
3. Engage actively with the data, fostering a feedback loop that can improve classifier performance over time.

The authors modestly assert that while there is a minor performance cost, the ability to debug spurious evidence and the establishment of a user-driven feedback mechanism are essential for the long-term improvement of content classification across NASA's planetary missions. Future work will focus on user studies to optimize the visualization of prototypes and the integration of user feedback into model refinement protocols.