Technical Summary: Open-Linguistic Concept Unified Learning for Cross-Site Interpretable Dermatology Image Diagnosis

1. Problem Statement

The paper addresses a critical bottleneck in the clinical deployment of Computer-Aided Diagnosis (CAD) systems for dermatology: cross-site concept heterogeneity. While Concept Bottleneck Models (CBMs) offer interpretability by predicting human-understandable concepts before final diagnosis, they suffer from rigid, dataset-specific adaptations.

• Heterogeneous Taxonomies: Different clinical sites and imaging modalities (e.g., dermoscopic vs. clinical photographs) utilize distinct, non-overlapping concept vocabularies. Dermoscopic images rely on micro-structural concepts (e.g., "pigment network," "streaks"), while clinical photographs rely on macroscopic concepts (e.g., "plaque," "crust").
• Generalization Failure: Existing CBMs and Vision-Language Models (VLMs) require costly, dataset-specific label engineering and retraining to adapt to new sites or modalities. They lack a unified semantic space, preventing the transfer of clinician interventions (e.g., correcting a concept prediction) across different imaging types.
• Interpretability Gaps: Current models often output opaque probabilities or rely on fixed text templates that fail to capture the nuance of uncertain clinical contexts, limiting their ability to support "human-in-the-loop" decision-making.

2. Methodology: UniCon Framework

The authors propose UniCon, an open-linguistic unified concept learning framework designed to create a shared semantic representation space across heterogeneous modalities without dataset-specific retraining. The framework operates in two interconnected stages:

Stage 1: Instance-Level Multi-Faceted Semantic Alignment

To construct a language-robust and boundary-sensitive semantic space, UniCon generates Multi-Faceted Semantic Specifications (MSS) for each image using a large vision-language model (Qwen-VL). Instead of single-template prompts, MSS generates four complementary textual slots for every concept:

1. Description ($T_{desc}$): Canonical definitions of morphological features.
2. Synonyms ($T_{syn}$): Alternative terminologies to handle linguistic variations across cohorts.
3. Edge-cases ($T_{edge}$): Descriptions of atypical presentations to improve robustness in uncertain regions.
4. Counter-examples ($T_{neg}$): Explicit descriptions of visually similar but pathologically distinct signs to serve as negative constraints.

The model employs Global Image-Text Contrastive Optimization with a structured multi-margin objective. This includes:

• Intra-Textual Anchoring: Ensuring positive anchors (description, synonyms, edge-cases) form a compact semantic neighborhood.
• Semantic-to-Visual Anchoring: Aligning the image embedding with the semantic centroid while repelling it from counter-examples.
• Inter-Sample Class Separation: Pushing representations of different diagnostic classes apart.

Stage 2: Concept-Level Alignment and Bottleneck Aggregation

This stage aggregates cross-cohort knowledge into a Unified Concept Prototype Codebook (UCPC) and performs dynamic filtering.

• Unified Concept Prototype Codebook (UCPC): For each concept and its discrete states (e.g., "pigment network" = {typical, atypical, absent}), the model averages embeddings from all available datasets. This creates a cohort-agnostic representation that marginalizes out dataset-specific linguistic biases.
• Learnable Evidence Aggregation (LEA): Visual queries probe the UCPC. The model computes cosine similarities between visual features and codebook slots. It uses learned weights to aggregate positive evidence (description, synonyms, edge-cases) and penalize negative evidence (counter-examples) to produce calibrated concept scores.
• Image-Conditioned Reliability Gating (ICRG): A lightweight MLP predicts a reliability score for each concept based solely on visual features. This gate dynamically filters the concept bottleneck, suppressing concepts that are not applicable to the specific image modality (e.g., suppressing "pigment network" for a low-resolution clinical photo) and enabling cross-modal corrections.

Optimization Strategy

The framework is optimized via a three-step joint strategy:

1. Concept Representation Alignment: Optimizing encoders and aggregation modules while freezing the classification head, using cohort-specific masks to ensure gradients only flow for observable concepts.
2. Diagnostic Head Initialization: Training the classification head on the filtered, high-confidence concept representations.
3. End-to-End Fine-Tuning: Jointly fine-tuning the entire network with a smaller learning rate.

3. Key Contributions

1. Unified Semantic Space: A shared concept prototype codebook that seamlessly coordinates heterogeneous concept systems (dermoscopic and clinical) across modalities, eliminating the need for dataset-specific retraining or label engineering.
2. Open-Linguistic Semantic Specifications: A multi-faceted approach (Definition, Synonym, Edge-case, Counter-example) that overcomes sparse textual label limitations, improving boundary sensitivity in uncertain clinical contexts.
3. Robust Intervention Interface: A reliability-gated bottleneck aggregation mechanism that enables consistent, adjustable, and transparent clinician interventions across diverse clinical populations and imaging modalities.

4. Experimental Results

The framework was evaluated on public datasets including Derm7pt, SkinCon (which comprises the Fitzpatrick17k and DDI sub-cohorts), and PH2.

• Diagnostic Performance: UniCon achieved state-of-the-art performance on both Derm7pt and SkinCon datasets.
  • Derm7pt: F1-score of 0.828, AUROC of 0.903.
  • SkinCon: F1-score of 0.854, AUROC of 0.950.
  • It consistently outperformed baselines like CBM, PCBM, LaBo, and MICA, particularly in threshold-independent metrics (AUROC, AUPRC) crucial for imbalanced medical data.
• Concept Prediction: UniCon demonstrated superior precision and F1-scores in predicting intermediate concepts, ensuring the downstream classifier is driven by genuine evidence rather than spurious correlations.
• Zero-Shot Inference: On the unseen PH2 dataset, UniCon surpassed all baselines in both disease diagnosis and concept prediction, demonstrating robust generalization without target-domain exposure.
• Cross-Modal Retrieval: The model achieved the best cross-modal retrieval performance (AVG 0.279) on SkinCon, significantly outperforming BioMedCLIP and other VLMs.
• Intervention Capabilities: Test-time interventions showed that modifying concept states (e.g., forcing "pigment network" to be present) predictably altered the final diagnosis. The model demonstrated the ability to bridge modalities, allowing clinicians to apply dermoscopic concepts to clinical images and vice versa.

5. Significance and Claims

The paper claims that UniCon successfully bridges the gap between disparate clinical taxonomies, unlocking unprecedented cross-site intervention capabilities. By moving away from rigid, dataset-specific concept sets, UniCon provides a transferable, interpretable, and controllable framework for dermatology CAD.

The authors emphasize that their approach:

• Decouples interpretable concept learning from final diagnostic decisions.
• Enables clinicians to guide model decisions using holistic expertise, regardless of the underlying imaging modality.
• Facilitates scalable, multi-center clinical deployment by removing the barrier of costly, repetitive label engineering for new sites.

The work positions itself as a step toward evidence-based, clinically verifiable prediction systems that maintain transparency and allow for human-in-the-loop error correction in real-world, heterogeneous clinical environments.