💻 computer science

Assessing Multimodal Chronic Wound Embeddings with Expert Triplet Agreement

This paper introduces TriDerm, a multimodal framework that integrates wound imagery, boundary masks, and expert reports to generate clinically meaningful embeddings for rare skin disorders, achieving a 73.5% agreement with expert triplet judgments and outperforming existing foundation models.

Original authors: Fabian Kabus, Julia Hindel, Jelena Bratulić, Meropi Karakioulaki, Ayush Gupta, Cristina Has, Thomas Brox, Abhinav Valada, Harald Binder

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fabian Kabus, Julia Hindel, Jelena Bratulić, Meropi Karakioulaki, Ayush Gupta, Cristina Has, Thomas Brox, Abhinav Valada, Harald Binder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of fingerprints, your clues are pictures of skin wounds and doctors' notes. The mystery? A rare and tricky skin condition called RDEB.

This disease is like a "long-tail" problem in the world of medicine. It's so rare that the giant, pre-trained AI models (the "super-smart detectives" of the tech world) haven't seen enough examples of it to be good at solving it. They might recognize a generic rash, but they can't tell the difference between two very specific, subtle types of RDEB wounds that a human expert would spot instantly.

The authors of this paper wanted to fix this. They built a new system called TriDerm to help computers understand these wounds better, using a clever mix of pictures, text, and a game-like method to train the AI.

Here is how they did it, broken down into simple concepts:

1. The "Which One Looks More Like This?" Game (Triplet Judgments)

Usually, to teach an AI, you give it a picture and say, "This is a wound." But for rare diseases, you don't have thousands of labeled pictures.

Instead, the researchers used a dermatologist (a skin expert) to play a game. They showed the expert three pictures:

Picture A (The Reference)
Picture B (The Reference)
Picture C (The Mystery)

The expert had to answer: "Does Picture C look more like Picture A or Picture B?"

This is powerful because it doesn't force the expert to put a rigid label on the wound (like "Type 1" or "Type 2"). Instead, it captures the feeling of similarity. It's like asking a wine expert, "Does this new wine taste more like the Cabernet or the Merlot?" rather than asking them to define the exact chemical composition. The AI learns from these "A vs. B" choices to build a map of what wounds look like.

2. The Two Eyes of TriDerm

The system has two "eyes" to look at the problem: one for Images and one for Text.

The Image Eye (The Spotlight):
Standard AI looks at a whole photo and averages everything out. But a wound might be just a small spot on a large leg. The researchers gave their AI a "spotlight" (called Attention Pooling). Instead of looking at the whole leg, the AI focuses intensely only on the wound area, ignoring the healthy skin around it. They also taught it using a "non-contrastive" method, which is like teaching a student by showing them two slightly different photos of the same wound and saying, "These are the same," without needing to show them a totally different wound to say, "This is different." This works great when you don't have many photos to begin with.
The Text Eye (The Translator):
Doctors write notes describing wounds: "red," "deep," "scabbed," "healing." The researchers used a giant AI language model (like a super-smart chatbot) as a "Synthetic Expert." They asked the chatbot the same "A vs. B" game using the doctors' text notes. The chatbot's answers were then used to create a mathematical map of the text, turning words into a shape that matches the visual map.

3. The Magic of Mixing (Fusion)

Here is the best part: Vision and Text are like two different detectives.

The Image detective sees the shape and color.
The Text detective reads the history and the specific medical terms.

Sometimes the image is blurry, but the text is clear. Sometimes the text is vague, but the image is sharp. The researchers combined these two detectives. They created a system that weighs the "confidence" of each detective. If the image is clear, it listens more to the image. If the text is detailed, it listens more to the text.

The Result

When they tested their new system, TriDerm, against the experts:

The old, off-the-shelf AI models got about 68% of the "A vs. B" games right.
Their new TriDerm system got 73.5% right.

That might not sound like a huge jump, but in medicine, that extra 5.6% means the AI is much closer to a human expert's intuition. It successfully learned to see the subtle differences in these rare wounds that the big, generic models missed.

Why This Matters

This paper is a blueprint for how to teach AI about rare diseases when you don't have massive databases. Instead of needing millions of labeled photos, you just need a few experts to play a simple "which looks more similar?" game. By combining what the AI sees with what it reads, and training it with human intuition, we can build tools that actually help doctors treat patients with rare conditions.

In short: They taught a computer to understand rare skin wounds by playing a similarity game with a human doctor, using a "spotlight" for photos and a "translator" for notes, and then letting the two work together to make the best diagnosis possible.

1. Problem Statement

The paper addresses the challenge of retrieving and assessing similar cases for Recessive Dystrophic Epidermolysis Bullosa (RDEB), a rare, chronic genetic skin disorder characterized by severe skin fragility and non-healing wounds.

Limitations of Current Models: Off-the-shelf foundation models (both vision and language) fail to capture the fine-grained, clinically meaningful features required for this "long-tail" disease. They often lack the precision to differentiate subtle phenotypic variations in wound severity and tissue damage.
Data Scarcity: RDEB is rare, resulting in small datasets where traditional supervised learning is difficult.
Validation Gap: Validating embedding spaces for rare diseases is challenging because standard categorical labels are insufficient. Clinical similarity is often graded and overlapping, resisting discrete categorization. There is a need for a structured, expert-anchored assessment method that reflects the nuanced reasoning of dermatologists.

2. Methodology: TriDerm

The authors propose TriDerm, a multimodal framework designed to learn interpretable wound representations from small cohorts by integrating wound imagery, boundary masks, and expert reports. The core innovation lies in using expert triplet judgments (ordinal comparisons) rather than categorical labels to guide and evaluate the models.

A. Data Collection

Dataset: 53 wound camera images from 21 RDEB patients, yielding 120 labeled wounds.
Annotations: Includes wound boundary masks and free-text clinical descriptions (granulation, fibrin, inflammation, depth, etc.).
Expert Triplet Judgments: A dermatologist performed 513 triplet comparisons. Given two reference wounds and a query wound, the expert indicated which reference the query was more similar to. This encodes ordinal similarity structure.

B. Visual Representation (TriDerm Vision)

Backbone: Uses a frozen DermLIP-ViT-B foundation model.
Wound-Level Attention Pooling: Instead of global image pooling, the model extracts features specifically from wound regions defined by masks. It employs a two-layer MLP with attention pooling to aggregate spatial features, creating a specific representation for each wound.
Learning Objective: Trained using VICReg (Variance-Invariance-Covariance Regularization), a non-contrastive self-supervised loss.
- Why VICReg? Unlike contrastive learning, it does not require negative samples, making it highly effective for small datasets where constructing diverse negative pairs is difficult.
- The loss enforces consistency between augmented views while preventing representational collapse and reducing dimension redundancy.

C. Text Representation (TriDerm Text)

Synthetic Expert Protocol: Instead of fine-tuning a Large Language Model (LLM) directly (which requires large data), the authors treat the LLM as a "synthetic expert."
Prompting: The LLM (GPT-OSS-120B) is prompted with wound descriptions and asked to perform triplet comparisons ("Is wound A more similar to B or C?").
Soft Ordinal Embedding (SOE): The binary responses from the LLM are used to fit a continuous embedding space via SOE. This minimizes a hinge loss to ensure the embedding distances reflect the LLM's (and by extension, the medical logic's) ordinal judgments.
Efficiency: This approach recovers medically meaningful representations from only 50 unlabeled text descriptions.

D. Multimodal Fusion

The paper evaluates two fusion strategies that operate on pairwise distance matrices without additional training:

Uncertainty-Aware Fusion: Weights the vision and text modalities based on the variance of distances for a specific sample. High variance implies clear separation (high confidence), while low variance implies uncertainty. Vision is weighted slightly higher ( $\alpha=0.7$ ) due to empirically stronger performance.
Similarity-Based Fusion: Converts distances to similarities and computes the element-wise product ( $S_{fused} = S_V \odot S_T$ ). This acts as a filter, requiring agreement between modalities to confirm a match.

3. Key Contributions

Triplet Paradigm for Rare Diseases: Introduces a method to evaluate medical reasoning in visual, textual, and multimodal representations using expert ordinal triplet comparisons, which better capture graded phenotypic relationships than categorical labels.
TriDerm Framework:
- Vision: Adapts foundation models to RDEB using wound-level attention pooling and non-contrastive learning (VICReg).
- Text: Proposes a synthetic-expert protocol using LLMs and Soft Ordinal Embeddings (SOE) to derive embeddings from minimal data.
Multimodal Synergy: Demonstrates that vision and text capture complementary aspects of wound phenotypes, and fusing them significantly outperforms single-modality approaches.
Open Science: The authors release the expert annotation tool, model code, and representative dataset samples.

4. Results

The models were evaluated based on Triplet Agreement with the dermatologist's ground truth.

Text Performance: The proposed SOE method achieved 67.4% balanced agreement, outperforming the best baseline (RoBERTa) by 3.7 percentage points. Notably, MedGemma-27B performed well on text (61.3%) but failed on vision (50.3%, near random chance).
Vision Performance: TriDerm (Vision only) achieved 71.6% agreement, surpassing the best baseline (DermLIP) by 3.7 percentage points.
Multimodal Fusion:
- Uncertainty-aware fusion achieved the highest performance: 73.5% balanced agreement.
- This represents a 5.6 percentage point improvement over the best off-the-shelf single-modality foundation model.
- Cohen's $\kappa$ for the fused model reached 0.446 (moderate agreement), compared to ~0.32–0.40 for unimodal methods.
Qualitative Analysis: Nearest-neighbor retrieval showed that the embeddings correctly grouped wounds by clinical features (e.g., superficial granulating lesions vs. deep ulcers; deep fibrotic ulcers vs. superficial inflamed wounds).

5. Significance and Conclusion

Clinical Relevance: The study proves that for rare diseases with limited data, adapting foundation models via self-supervised objectives (VICReg) and ordinal constraints (Triplets/SOE) is more effective than standard supervised fine-tuning or using raw foundation model outputs.
Data Efficiency: The framework successfully learns from extremely small cohorts (53 images, 50 text descriptions) by leveraging the structural knowledge of experts and LLMs.
Complementarity: The results highlight that visual and textual modalities provide independent, complementary signals regarding wound phenotypes. Fusing them reduces the error rate of single modalities.
Future Impact: The proposed "Triplet Paradigm" offers a robust evaluation metric for medical AI in scenarios where labeled data is scarce but expert similarity judgments are available, bridging the gap between AI capabilities and clinical decision-making in rare disease management.