💻 computer science

In-Context Learning for Wound Classification with Small Multimodal Language Models

This study demonstrates that small multimodal language models can effectively perform training-free wound image classification through retrieval-based in-context learning, achieving high accuracy on public datasets by leveraging query-conditioned support examples and maximal marginal relevance reranking without the need for task-specific retraining.

Original authors: George Martvel, Oskar Gustafsson, John Pavia, Ernst Ahlberg

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: George Martvel, Oskar Gustafsson, John Pavia, Ernst Ahlberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to recognize different types of wounds, like a scrape, a burn, or a deep cut. Usually, to teach a robot this, you have to show it thousands of labeled pictures and force it to memorize them through a process called "training." It's like making a student study for weeks for a single test. But what if the robot could just look at a few examples right before the test and figure it out on the spot? This is the idea behind "In-Context Learning" (ICL). Think of it like handing a student a cheat sheet with three perfect examples of a "burn" and a "cut" right before they see the new picture. They don't need to memorize the whole textbook; they just need to look at the examples and say, "Oh, this new one looks like the burn example."

Now, imagine this robot is small and fits on a laptop or a phone, rather than being a giant supercomputer in a cloud. These are called "Small Multimodal Language Models" (SMLMs). They are great for privacy because you don't have to send your private medical photos to a big server; the robot can do the thinking right there on your device. The big question scientists have been asking is: Can these small, local robots actually do a good job with just a few examples, or do they need the giant supercomputers and massive training data to get it right?

This paper puts that idea to the test. The researchers wanted to see if they could use these small, local AI models to classify wound images without retraining them every time. They set up an experiment using two public collections of wound photos: one with about 1,469 images and another with 560. Instead of just asking the AI to guess (which is called "zero-shot" and often leads to wild guesses), they tried giving it a "cheat sheet" of examples. But here's the trick: they didn't just pick random examples. They used a smart search system to find the examples that looked most similar to the new wound image.

The results were quite exciting for the small robots. When the AI was given a smartly chosen set of examples (specifically, 10 examples found by a search system called kNN, and sometimes refined with a method called MMR to make sure the examples were diverse), it got much better at its job. On the larger dataset, the best small model, a 27-billion-parameter version of Qwen 3.5, reached an accuracy of 0.872. That means it got the diagnosis right more than 87% of the time. On the trickier, smaller dataset, it still managed about 68% accuracy.

The study also found that the size of the robot mattered. The bigger small models (like the 27B and 9B versions) were able to use the examples to improve their guesses significantly, often beating even a standard computer program that just votes based on the closest match. However, the tiniest models struggled a bit more, sometimes just copying the examples without really understanding the nuance. The researchers also discovered that you don't need a huge cheat sheet; adding more than 8 to 10 examples didn't help much and just made the process slower.

Crucially, the paper shows that this approach works without needing to retrain the model for every new task. If the doctors decide to change the labels or the types of wounds they are looking for, you just swap out the examples in the prompt, and the model adapts instantly. While the performance isn't perfect and depends heavily on having good examples and a decent-sized model, the study suggests that small, local AI models can be a practical, privacy-friendly way to help classify wounds, offering a flexible alternative to the heavy, data-hungry systems of the past.

Technical Summary: In-Context Learning for Wound Classification with Small Multimodal Language Models

Problem Definition
Wound image classification is traditionally treated as a task-specific supervised learning problem. This paradigm requires substantial manually labeled data and necessitates retraining whenever the label space or deployment context changes. Such requirements are particularly burdensome in medical settings where expert labels are scarce, privacy constraints limit data sharing, and clinical definitions evolve rapidly. While Large Multimodal Language Models (LMLMs) offer zero-shot capabilities, they often require server-side inference, raising privacy concerns for sensitive patient data. Conversely, Small Multimodal Language Models (SMLMs) suitable for local or edge deployment often struggle with complex medical tasks without fine-tuning. This study investigates whether SMLMs can serve as a practical, privacy-preserving alternative for wound classification using retrieval-based in-context learning (ICL), thereby avoiding task-specific retraining.

Methodology
The study evaluated eleven SMLMs from the Qwen 3.5, Ministral 3, and Gemma 4 families, ranging from 0.8B to 27B parameters. Experiments were conducted on two public datasets: the Kaggle wound dataset (1,469 images, 10 classes) and the Medetec dataset (560 images, 9 classes).

The core methodology involved constructing prompts for the SMLMs using three strategies for selecting support examples (few-shot context):

Zero-shot: No support examples provided.
Random Few-shot: Support examples sampled uniformly from the dataset.
Retrieval-based ICL: Support examples selected via embedding-based retrieval using a frozen OpenAI CLIP ViT-B/32 encoder. Two retrieval strategies were tested:
- kNN: Selecting the $K$ nearest neighbors based on cosine similarity.
- kNN+MMR: Selecting neighbors using Maximal Marginal Relevance to balance relevance and diversity.

To isolate the contribution of the model's reasoning from the retrieval mechanism itself, the authors compared SMLM performance against a retrieval-only weighted-kNN baseline. Additionally, the study assessed the impact of support set size (reducing the pool to 75%, 50%, and 25%) and support context size (varying $K$ from 2 to 20) on performance and inference cost. Supervised baselines (ViT, ConvNeXt, EfficientNet, ResNet) with full fine-tuning, LoRA, and head-only tuning were also evaluated for comparison.

Key Results

Retrieval is Critical: Query-conditioned retrieval (kNN and kNN+MMR) consistently outperformed both zero-shot and random few-shot prompting. On the Kaggle dataset, the best model (Qwen 3.5 27B with kNN+MMR) achieved 0.872 accuracy, compared to 0.580 for zero-shot. On the more challenging Medetec dataset, the same configuration reached 0.678 accuracy versus 0.367 for zero-shot.
Model Scale and Reasoning: Larger models (e.g., Qwen 3.5 27B, 9B; Gemma 4 26B) significantly outperformed the retrieval-only weighted-kNN baseline. Statistical analysis (McNemar tests) confirmed that these models did not merely "parrot" the retrieved examples but utilized internal knowledge to correct errors made by the retrieval-only control. For instance, on Kaggle, Qwen 3.5 27B preserved 96% of the control's correct predictions while "rescuing" 61.6% of the control's errors. Smaller models (e.g., Qwen 3.5 0.8B) often failed to surpass the retrieval-only baseline, suggesting they lacked the capacity to effectively integrate retrieved context.
Robustness to Data Scarcity: Retrieval-based ICL demonstrated robustness to reduced support sets. While performance degraded as the support pool shrank, the best SMLMs with ICL outperformed supervised baselines trained on the same reduced data fractions. For example, on the Medetec dataset with only 25% of the support data, the best SMLM (Qwen 3.5 27B) achieved ~0.53 accuracy, surpassing the best supervised model (ConvNeXt-Tiny) which dropped to ~0.41.
Context Size and Cost: Increasing the support context size yielded positive but diminishing returns. Most performance gains saturated between 8 and 10 support images. Beyond this point, accuracy gains were minimal while inference latency and token counts increased linearly.

Significance and Claims
The paper claims that retrieval-based ICL allows SMLMs to perform adaptable wound image classification without the need for task-specific retraining or fine-tuning. This approach offers a viable compromise between task adaptability, computational cost, and classification performance, particularly in privacy-sensitive settings where data cannot be sent to external cloud providers.

The authors emphasize that while SMLMs do not necessarily outperform large remote models or fully fine-tuned systems in all scenarios, they provide a "useful privacy-preserving compromise." The study demonstrates that compact, retrieved contexts can support practical deployment on local hardware, provided the model scale and retrieval strategy are appropriately matched to the task difficulty. The work highlights that the choice of support examples is central to effective ICL, and that larger SMLMs can effectively leverage retrieved context to exceed simple nearest-neighbor voting, making them a flexible solution for medical imaging tasks where label sets and definitions may change.

Technical Summary: In-Context Learning for Wound Classification with Small Multimodal Language Models

More like this