Technical Summary: LVLM-Aided Alignment of Task-Specific Vision Models

Problem Statement

In high-stakes domains such as medicine and manufacturing, small, task-specific vision models are often preferred over Large Vision Language Models (LVLMs) due to their lower computational requirements and established explainability methods. However, these models frequently fail to align with human domain knowledge. Instead of learning robust core features, they often rely on spurious correlations (shortcuts) present in training data. While Explainable AI (XAI) techniques can identify these shortcuts, interpreting the resulting heatmaps is difficult for domain experts. Furthermore, existing methods for correcting these models typically require fine-grained, instance-wise feedback (e.g., pixel-level masks) from experts, which is prohibitively time-consuming for specialized professionals like medical doctors.

Methodology: LVLM-Aided Visual Alignment (LVLM-VA)

The authors propose LVLM-VA, a novel two-step approach that leverages the generalization capabilities of an LVLM to act as a bidirectional translator between human domain knowledge and model behavior. The method aligns small vision models with human specifications without requiring fine-grained annotations.

1. Detecting Spurious Correlations

The process begins by identifying training samples where the model relies on shortcuts.

• Sampling Strategy: Instead of processing the entire dataset, the method employs an unsupervised sampling strategy based on output entropy. The authors hypothesize that models learn shortcuts more easily than robust features, resulting in lower output entropy on shortcut-dependent samples. The alignment dataset ($D_{align}$) consists of the $N$ samples with the lowest entropy.
• PPEPS-WGM (Positive Predictive Effect Probabilistic Segmentation via Weighted Gaussian Mixtures): To facilitate the LVLM's analysis, the method generates explanation maps (using DeepLIFT-SHAP) and applies a custom segmentation technique. Unlike standard segmentation (e.g., SAM) which follows object boundaries, PPEPS-WGM clusters regions based on their positive predictive effect. It fits a weighted Gaussian mixture to the distribution of positive attribution, steering the segmentation toward regions that most influence the model's prediction, regardless of object boundaries.
• Critic & Judge Pair:
  • The Critic (LVLM): Receives the original image, the ground truth label, a class-level human specification ($V_k$), and the PPEPS-WGM segmentation map. Using a Chain-of-Thought prompt, the Critic analyzes whether specific clusters correspond to relevant core features or spurious shortcuts (e.g., hospital tags or colored bandages).
  • The Judge (LLM): Takes the free-form reasoning of the Critic and maps it to a binary verdict ($R_j$) for each cluster, determining if it is spurious. This step standardizes the output for automated training.

2. Visual Alignment

Once spurious clusters are identified, the method corrects the model using the Right for the Right Reasons (RRR) loss function.

• Correction Masks: The binary verdicts from the Judge are automatically translated into instance-wise correction masks ($A$). Pixels belonging to clusters deemed spurious are masked out.
• Training: The original model is fine-tuned using a combination of standard cross-entropy loss and the RRR loss. The RRR loss penalizes gradients in regions identified as spurious, forcing the model to rely on relevant features.
• Efficiency: This process eliminates the need for manual instance-wise mask generation, requiring only high-level class descriptions from human experts.

Key Contributions

1. LVLM-VA Framework: A novel approach for aligning small vision models with human domain knowledge by translating class-level human specifications into instance-wise corrections via an LVLM.
2. PPEPS-WGM: A segmentation method designed specifically to support spurious feature detection by clustering based on positive predictive effect rather than visual object boundaries.
3. Reduced Human Effort: The method demonstrates that effective alignment can be achieved without fine-grained feedback, relying instead on scalable class-level descriptions.

Experimental Results

The authors evaluated LVLM-VA on three datasets: a synthetic DecoyMNIST dataset and two real-world medical datasets (ISIC skin lesions and Knee Osteoarthritis radiographs).

• Synthetic Setting (DecoyMNIST): The method successfully shifted the model's attention from artificial spurious decoys (grey patches) to the actual digits. The alignment metric showed that the model's attribution mass moved significantly toward relevant features, approaching the performance of models trained with ground-truth instance-wise masks.
• Medical Datasets:
  • Skin Lesions: The model learned to ignore colored bandages (spurious features) that were correlated with malignancy in the training set but not in the test set.
  • Knee Radiographs: The model learned to ignore hospital tags (L/R markers) that were correlated with specific knee conditions in the training data.
• Performance Metrics: Across both medical datasets, LVLM-VA significantly improved the Worst Group Accuracy (WGA)—a standard metric for robustness against spurious correlations—while maintaining or slightly improving the Average Group Accuracy (AGA).
• Comparisons: LVLM-VA outperformed non-human-centric baseline methods such as Sub-sampling Groups (SUBG), Deep Feature Reweighting (DFR), and Just Train Twice (JTT). While SUBG improved WGA, it severely degraded overall accuracy. LVLM-VA was the only method to improve WGA without sacrificing overall performance.
• User Study: A study with 18 domain experts showed an 88% agreement rate with the LVLM's identification of spurious clusters and 87% agreement with the LVLM's reasoning, validating the method's alignment with human expectations.

Significance and Claims

The paper claims that LVLM-VA provides a synergistic interface between generative AI (LVLMs) and discriminative task-specific models. By translating human domain knowledge into natural language and then into instance-level critiques, the method allows experts to steer models effectively without the prohibitive cost of pixel-level annotation.

The authors emphasize that this approach addresses the "brittle behavior" of models deployed in the real world by reducing dependence on spurious features and group-specific biases. They position LVLM-VA as a practical solution for high-stakes domains where interpretability and trust are paramount, offering a scalable alternative to current methods that rely on extensive manual feedback or purely statistical group balancing. The method is shown to be adaptable to different LVLMs (including open-source models) and robust to variations in hyperparameters like the number of clusters.