Technical Summary: Explainable Concept Generation through Vision-Language Preference Learning

Problem Statement

Understanding the internal representations of deep neural networks (DNNs) is critical for model improvement and regulatory assessment. While concept-based explanation methods, such as Testing with Concept Activation Vectors (TCAV), offer human-interpretable insights by linking high-level visual concepts (e.g., "stripes") to model decisions, they suffer from a significant bottleneck: manual concept curation.

Existing approaches require practitioners to manually guess and collect candidate concept image sets. This process is labor-intensive, prone to overlooking critical concepts, and limited by human cognitive biases. Furthermore, alternative "retrieval" methods that extract concepts directly from test images (e.g., via segmentation) often fail to capture high-level abstractions, tending to produce patches of the target class (e.g., a patch of a zebra) rather than the underlying features (e.g., stripes) that drive the classification. There is a lack of automated methods capable of generating diverse, high-level concepts that truly matter to a specific DNN's decision-making process without human intervention.

Methodology: Reinforcement Learning-based Preference Optimization (RLPO)

The authors propose RLPO, a framework that redefines concept set creation as an image generation problem. Instead of retrieving or manually curating images, RLPO uses a generative model guided by deep reinforcement learning to synthesize concept images that maximize TCAV scores.

Core Components

1. Generative Model (Stable Diffusion): The system utilizes a pre-trained text-to-image model (Stable Diffusion v1-5) to generate concept images based on text prompts.
2. Deep Reinforcement Learning (DQN): A Deep Q-Network (DQN) controls the generation process.
   • Action ($a_t$): Selecting a seed prompt from a predefined list (extracted via a Vision-Question Answering model).
   • State ($s_t$): The preferred concept images generated from the previous prompt.
   • Reward ($r_t$): Calculated based on the TCAV scores of the generated images.
3. Preference Optimization (RLPO): Unlike standard preference optimization that relies on human ranking, RLPO uses TCAV scores as the preference signal. The system compares two groups of generated images ($G_1$ and $G_2$); the group with the higher TCAV score is preferred.
4. Fine-Tuning via LoRA: To guide the generative model toward producing more relevant concepts, the system employs Low-Rank Adaptation (LoRA). The weights of the Stable Diffusion model are updated ($w_{t+1} \leftarrow w_t + \lambda ab$) based on the preference feedback, iteratively refining the model to generate images that trigger specific neural network activations.
5. Dynamic Reward Scaling: A scaling factor $\xi_t$ is introduced to adjust rewards over time, ensuring that the agent receives appropriate feedback as it progresses from "proxy states" (low TCAV) to "explainable states" (high TCAV).

Key Contributions

1. Automated Concept Generation: The paper introduces RLPO, a method to automatically "generate" concepts that are relevant to a neural network's decision process, bypassing the need for manual guessing or retrieval.
2. Discovery of Non-Intuitive Concepts: The method successfully identifies diverse concepts that are challenging for humans or retrieval-based methods to anticipate (e.g., background elements like "mud" or "running" trees for a zebra classifier, not just "stripes").
3. Comprehensive Validation: The authors provide both qualitative and quantitative evidence (computational metrics and human surveys) demonstrating that RLPO effectively reveals internal network representations.
4. Modularity and Generalizability: The framework is designed to be modular, allowing components like the generative model to be swapped. The authors demonstrate its applicability beyond computer vision, extending it to NLP sentiment analysis tasks.

Experimental Results

The method was evaluated on multiple classifiers (GoogleNet, InceptionV3, ViT, Swin) trained on ImageNet, as well as a ResNet-18 on the CelebA dataset and a TextCNN on IMDB.

• Search Strategy: RLPO outperformed $\epsilon$-greedy search strategies, exhibiting higher entropy and better exploration of the action space (Table 1).
• Scoring Mechanism: Using TCAV for preference feedback (RLPO-XAIF) proved superior to human feedback (RLPO-HF) and AI feedback (RLPO-AIF) in terms of execution time and computational cost, while providing model-specific explanations (Table 2).
• Concept Diversity and Novelty:
  • Exploration Gap (EG): Human surveys revealed a significant gap between concepts humans could identify and those generated by RLPO. For laymen, the EG was 91.54% for RLPO versus 6.54% for retrieval methods, indicating RLPO finds concepts humans do not typically consider (Table 3).
  • Diversity: Generated concepts showed lower cosine similarity and higher Euclidean distance compared to retrieval-based methods, confirming they represent abstract features rather than image patches (Table 4).
• Validation:
  • C-Deletion: Removing generated concepts (e.g., "stripes") from input images caused a significant drop in model accuracy, validating their importance (Fig. 6).
  • Engineer Utility: In a study with 19 ML engineers, 94.7% agreed that RLPO explanations helped them understand the model better, and 84.2% found new insights. Engineers often missed concepts (like "running" or "mud") that RLPO identified as important.
• Bias Detection: Applied to a "Blonde" vs. "Not Blonde" classifier, RLPO revealed that the model relied heavily on "female face" concepts, highlighting gender bias in the training data.
• NLP Generalization: The framework was successfully adapted to generate word synonyms for sentiment analysis, demonstrating cross-domain applicability.

Significance and Claims

The paper claims that RLPO addresses the fundamental limitation of current concept-based XAI methods: the reliance on human intuition for concept selection. By framing concept creation as a generation problem optimized via reinforcement learning, the authors demonstrate that it is possible to automatically articulate diverse, high-level concepts that drive neural network decisions.

The authors position this work as a tool for:

1. Improving Models: Engineers can use generated concepts (e.g., identifying that a model focuses on background "blurry" textures for tigers) to fine-tune models and reduce reliance on spurious correlations.
2. Understanding Bias: The method can uncover undesirable biases learned during training that are not obvious to human observers.
3. Generalizability: The approach is not limited to vision; it can be adapted to other modalities like NLP.

The authors acknowledge limitations, noting that the quality of generated concepts depends on the initial seed prompts and the pre-training data of the generative model, which may introduce its own biases. They suggest future work could involve integrating domain expert preferences to further align generated explanations with specific application needs.