🤖 AI

Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection

ReCAP is a language-free medical anomaly detection framework that replaces static CLIP-based anchors with input-conditioned visual prototypes and a non-parametric memory, achieving state-of-the-art performance across zero-shot and few-shot settings while significantly reducing inference latency.

Original authors: Yibo Wan, Jinyu Cai, Seekiong-Ng

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yibo Wan, Jinyu Cai, Seekiong-Ng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to spot a fake painting in a massive museum. Usually, you'd need a guidebook that says, "A real Van Gogh looks like this," and "A fake one looks like that." But in the world of medical imaging, the "museum" is the human body, and the "paintings" are X-rays, MRIs, and CT scans. The problem is that the human body is incredibly diverse; a normal liver looks very different from a normal brain, and a normal lung looks different from a normal eye. Furthermore, the "fakes" (diseases) can be tiny, weird, and hard to describe. Traditional computer programs often try to learn a single, rigid rule for what "normal" looks like. But just like a detective who memorizes one specific type of forgery and misses all the others, these rigid rules often fail when they encounter a new type of patient or a new kind of camera. Scientists have been trying to fix this by teaching computers to understand language (like the words "healthy" or "sick"), but language is often too vague to catch subtle medical glitches.

This paper introduces a new detective tool called ReCAP. Instead of using a static, unchanging rulebook or vague words to define what is normal, ReCAP acts like a super-smart, adaptable guide that changes its mind for every single patient it sees. It uses a clever trick to shift its understanding of "normal" based on the specific picture it is looking at, without needing any text descriptions or slow, heavy calculations. The authors found that this method is not only more accurate at spotting diseases across different body parts but is also lightning-fast, making it a promising step toward helping doctors catch problems earlier and more reliably.

The Problem: The "One-Size-Fits-All" Trap

In the world of medical anomaly detection, computers are trained to find the "odd one out." They look at thousands of images of healthy organs to learn what "normal" looks like, and then they scan new images to find anything that doesn't fit. The trouble is that what looks normal for a brain MRI is totally different from what looks normal for a liver CT scan.

Older methods tried to solve this by using text prompts. They would tell the computer, "Look for things that are not 'normal' and are 'abnormal'." But words are clumsy. A "normal" brain and a "normal" liver are so different that the word "normal" doesn't really help the computer distinguish between them. Other methods tried to learn a fixed visual "anchor"—a single point in the computer's memory that represents "normal." But this is like trying to use a single map for the entire world; it works okay for a city, but it fails miserably when you try to navigate a mountain range. When the computer encounters a new type of organ or a new camera setting, that fixed anchor gets lost, and the computer starts missing the disease or crying wolf.

The Solution: ReCAP's Shape-Shifting Guide

The authors propose ReCAP (Re-Centered Anomaly Prototypes), a system that ditches the fixed anchors and the text descriptions entirely. Instead of having one rigid definition of "normal," ReCAP creates a custom definition for every single image it sees.

Think of it like a chameleon. If you put a chameleon on a green leaf, it turns green. If you put it on a red flower, it turns red. It doesn't have a single "true color"; it adapts its color to match its surroundings. ReCAP does something similar with its understanding of medical images.

Here is how it works in simple terms:

It looks at the whole picture first: Before trying to find the disease, ReCAP takes a quick look at the entire image to understand the context (e.g., "This is a brain," or "This is a liver").
It shifts its "Normal" and "Abnormal" targets: Based on that context, it slightly moves its internal "Normal" and "Abnormal" targets to fit that specific image. It's like a detective who, upon entering a new room, immediately adjusts their expectations of what a "messy room" looks like based on the furniture in that specific room.
It keeps a safety leash: The authors were worried that if the computer shifts its targets too much, it might get confused by the disease itself and decide the disease is actually normal. To stop this, they put a "leash" on the movement. The computer is allowed to adapt, but only within a safe, bounded range. It can shift its view to fit the patient, but it can't wander off so far that it forgets what a disease looks like.
It uses a memory bank (for few-shot learning): If the computer is given just a few examples of healthy patients from a new type of scan, it builds a quick "memory bank" of those specific healthy examples. This helps it remember the tiny, unique details of that specific group, which a general rulebook would miss.

What They Found

The team tested ReCAP on six different medical datasets, ranging from brain MRIs and liver CTs to eye scans and chest X-rays. They compared it against the best existing methods, including those that use text prompts and those that use fixed visual anchors.

The results were quite impressive:

Accuracy: ReCAP achieved the best results in spotting diseases (measured by a score called AUROC) in all six zero-shot settings (where it had to guess on a new type of scan without any examples) and in 23 out of 24 few-shot settings (where it had a tiny number of examples to learn from).
Precision: When it came to pinpointing exactly where the disease was (segmentation), ReCAP got the top score on all three datasets that required this level of detail.
Speed: Perhaps most surprisingly, ReCAP was incredibly fast. It was over 70% faster than the fastest existing method. While other methods took about 66 milliseconds to process an image, ReCAP did it in just 17.4 milliseconds for zero-shot tasks and 19.0 milliseconds for few-shot tasks. This is because it doesn't have to process text or run complex, slow updates while looking at the image.

Why This Matters

The paper suggests that the old way of using fixed rules or text descriptions is hitting a wall when it comes to the messy, diverse reality of human anatomy. By replacing static anchors with a smart, bounded, and adaptable system, ReCAP shows that computers can learn to spot medical anomalies more reliably and much faster.

The authors note that while this is a significant step forward, the current tests are on 2D images. The next big challenge will be seeing if this "shape-shifting guide" works on 3D volumes and more complex clinical data. But for now, ReCAP offers a promising new way to help doctors find the needle in the haystack, without needing a dictionary to do it.

Technical Summary: ReCAP for Language-Free Medical Anomaly Detection

Problem Definition

Medical anomaly detection (AD) aims to identify abnormal images and localize lesion regions under limited supervision, a critical requirement given the rarity, diversity, and high annotation costs of medical lesions. A practical detector must generalize across diverse organs (e.g., brain, liver, retina) and modalities (MRI, CT, X-ray, OCT) without requiring extensive target-domain annotations.

Existing CLIP-based approaches attempt to reduce annotation burdens by leveraging vision–language alignment. However, these methods typically rely on static anomaly anchors—either fixed text prompts (e.g., "normal" vs. "abnormal") or learned visual tokens—that remain unchanged across all test images. In cross-domain medical scenarios, where normal appearance varies significantly due to anatomical structures and imaging protocols, these static references often fail to transfer reliably. They suffer from three primary limitations:

Semantic Coarseness: Text prompts provide generic supervision that struggles to capture subtle, localized medical deviations.
Cross-Domain Miscalibration: Learned visual anchors, even without text, define fixed boundaries that may be misplaced for unseen target domains.
Lack of Instance-Level Adaptation: Compact global anchors cannot preserve the diverse normal patterns required for fine-grained localization, especially in few-shot settings.

Methodology: ReCAP

The authors propose ReCAP (Re-Centered Anomaly Prototypes), a language-free framework designed for zero-shot and few-shot medical anomaly detection. ReCAP replaces static anchors with input-conditioned visual prototypes that adapt to the global context of each query image without requiring text prompts or test-time gradient updates.

Core Components

Multi-Level Feature Extraction:
ReCAP utilizes a frozen CLIP visual encoder (ViT-L/14) augmented with lightweight segmentation and detection adapters inserted into selected transformer layers. This extracts $\ell_2$ -normalized multi-level patch tokens, preserving spatial fidelity for localization and discriminative responses for classification.
Conditional Prototype Modulation:
Instead of using fixed prototypes, ReCAP learns a pair of normal ( $q_n$ ) and abnormal ( $q_a$ ) visual prototypes. For each input image, these prototypes are re-centered based on the image's global context.
- Context Descriptor: A global context vector $c_l(x)$ is derived by average-pooling and normalizing the query's patch tokens.
- Bounded Gated Modulation: The prototypes are updated via a bounded gated residual:
  $\tilde{q}_{l}^{z} = \text{Norm}\left(q_{l}^{z} + \sigma(\beta_{l}^{z}) \cdot \eta \cdot \tanh(W_{l}^{z} c_{l}(x))\right)$
  This design is critical: because the context is pooled from the entire image (which may contain the lesion), unconstrained modulation could shift the "normal" anchor toward the anomaly, falsely classifying it as normal. The bounded nature (via $\tanh$ ) and the learnable gate ( $\sigma(\beta)$ ) ensure the adaptation remains controlled, preventing the prototype from collapsing onto anomalous content while still shifting toward the target domain's distribution.
Scoring Mechanisms:
- Prototype Branch: Anomaly scores are computed via scaled-cosine logits between query patches and the conditioned prototypes.
- Memory Branch (Few-Shot Only): To capture instance-level normal variations in few-shot settings, ReCAP constructs a non-parametric normal-reference memory bank from normal support images. Query patches are scored based on their nearest-neighbor cosine distance to this memory.
Fusion and Inference:
- Layer Fusion: Evidence from multiple transformer layers is aggregated using learnable weights and top- $k$ pooling to produce robust image-level scores and pixel-level maps.
- Branch Fusion: In few-shot settings, the prototype-based score and memory-based score are fused with a learnable weight $\lambda$ . In zero-shot settings, the memory branch is disabled ( $\lambda=0$ ).
Training Objectives:
The model is optimized using a combination of pixel-level localization loss (focal + Dice), image-level detection loss (binary cross-entropy), and a prototype separation regularizer that enforces a margin between the base normal and abnormal prototypes before conditioning.

Key Contributions

Identification of Static Anchors as a Limitation: The paper identifies that fixed anomaly references in CLIP-based methods hinder cross-domain generalization in medical imaging and proposes input-conditioned visual prototypes as a solution.
Bounded Gated Prototype Modulation: A novel mechanism that re-centers normal/abnormal prototypes according to input context while strictly constraining the shift to prevent "drift" toward anomalous content, enabling gradient-free adaptation.
Integrated Few-Shot Framework: The combination of conditional prototypes with a non-parametric normal-reference memory and multi-layer fusion, achieving state-of-the-art performance in both image-level detection and pixel-level localization across six benchmarks.

Experimental Results

ReCAP was evaluated on six medical datasets (BrainMRI, LiverCT, RESC, OCT17, ChestXray, HIS) covering five imaging domains under zero-shot and few-shot (K=2, 4, 8, 16) settings.

Zero-Shot Performance: ReCAP achieved the best image-level AUROC on all six datasets and the best pixel-level AUROC on all three segmentation datasets, outperforming both prompt-based CLIP methods and the language-free VisualAD baseline.
Few-Shot Performance: ReCAP achieved the best image-level AUROC on 23 out of 24 few-shot settings (across K=2 and K=4) and the best pixel-level AUROC on all segmentation datasets.
Efficiency: ReCAP demonstrated superior inference efficiency, reducing latency by over 70% compared to the fastest baseline (VisualAD). For instance, zero-shot inference took 17.4 ms per image compared to 66.1 ms for VisualAD, attributed to the avoidance of text encoding and complex token interactions.
Qualitative Analysis: Visualizations showed ReCAP produces more compact anomaly responses aligned with ground-truth lesions, with fewer false positives on normal anatomical structures compared to baselines like APRIL-GAN and MVFA.

Significance and Claims

The paper claims that ReCAP establishes controlled visual prototype adaptation as an effective alternative to text-based anomaly modeling in medical imaging. By eliminating the need for text prompts and test-time gradient updates, ReCAP offers a robust, language-free solution that generalizes well across unseen anatomical structures and modalities.

The authors emphasize that the bounded modulation mechanism is the key to success, allowing the model to adapt to target domains without compromising the discriminative boundary between normal and abnormal. While the current evaluation focuses on 2D benchmarks, the authors note that extending ReCAP to volumetric and multimodal clinical data remains an important future direction. The work suggests that for time-sensitive clinical screening, the combination of high accuracy and low inference latency makes ReCAP particularly suitable for practical deployment.