💻 computer science

Recurrent Contrastive Learning for Imbalanced Medical Image Classification

This paper proposes Recurrent Contrastive Learning (RCL), a novel framework that utilizes a Temporal Memory Queue to generate Temporal Anchors and progressively expand the latent support region of tail classes, thereby mitigating class imbalance and improving decision boundaries in medical image classification.

Original authors: Zhiyuan Zhu, Xinling Meng, Junxuan Yu, Jiongquan Chen, Qiongying Ni, Tuhang Shao, Yuhao Huang, Luping Zhou, Ruiyang Huang, Yuxue Wang, Rongliang Zhang, Xue Wang, Tianhong Tang, Likun Wang, Junbo Chen

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiyuan Zhu, Xinling Meng, Junxuan Yu, Jiongquan Chen, Qiongying Ni, Tuhang Shao, Yuhao Huang, Luping Zhou, Ruiyang Huang, Yuxue Wang, Rongliang Zhang, Xue Wang, Tianhong Tang, Likun Wang, Junbo Chen, Yong Jiang, Yongping Lu, Xin Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of animals in a zoo. If the zoo has 1,000 lions but only 5 tigers, the robot will get really good at spotting lions and will likely forget what a tiger looks like, or worse, it might think every cat-like animal is a lion. This is a common problem in the world of medical AI called "class imbalance." Doctors have tons of data for common diseases but very little for rare or severe ones. When AI models are trained on this uneven data, they become biased toward the common stuff and miss the rare, critical cases. To fix this, scientists usually try to either show the robot more pictures of the rare animals (resampling) or tell it to pay extra attention to them during lessons (reweighting). But there's a catch: these methods often just make the robot look harder at the few rare pictures it has, without actually giving the rare animals enough "space" in the robot's brain to stand out clearly from the common ones.

This is where a new study by Zhiyuan Zhu and his team steps in with a clever idea called Recurrent Contrastive Learning (RCL). Instead of just shuffling the deck of cards or shouting louder at the rare ones, they decided to build a "memory buffer" that helps the rare diseases expand their territory in the AI's mind. Think of it like a crowded dance floor where the popular kids (common diseases) are taking up all the space. The rare kids (rare diseases) are squished into a tiny corner. The researchers' solution is to bring in "ghost dancers" from previous practice sessions—old versions of the rare kids' moves—to stand around them and create a protective bubble. This bubble pushes the popular kids away, giving the rare diseases the room they need to be seen and recognized correctly. The team tested this on three different medical datasets, including carotid ultrasound scans, diabetic retinopathy images, and knee X-rays, and found that their method consistently helped the AI spot the rare conditions better than previous techniques.

The Problem: The "Popular Kid" Effect in Medical AI

In the world of medical imaging, not all diseases are created equal. Some, like a common cold or a mild skin rash, happen all the time. Others, like rare genetic disorders or specific stages of cancer, are much less frequent. When we train an AI to diagnose these conditions, it's like training a student who only gets to see 1,000 examples of the common cold but only 5 examples of a rare disease. Naturally, the student (the AI) becomes an expert at the common cold but gets confused when it sees the rare one. It might even mistake the rare disease for the common one because, in the AI's "mind," the rare disease doesn't have enough room to exist on its own.

The paper points out that existing solutions try to fix this by either showing the AI more examples of the rare disease (resampling) or by telling the AI to "pay more attention" to those few examples (reweighting). While these methods help a little, they don't solve the root problem: the rare diseases are still cramped into a tiny, crowded corner of the AI's feature space. They are so compact that the common diseases easily "encroach" on their territory, leading to mistakes.

The Solution: Time-Traveling Ghosts and Protective Bubbles

The authors propose a new strategy called Recurrent Contrastive Learning (RCL). To understand how it works, imagine the AI's learning process as a series of training sessions over time.

1. The Time-Traveling Memory (Temporal Memory Queue)
Usually, an AI only remembers the examples it sees in the current batch of training. RCL introduces a special "Time Machine" called a Temporal Memory Queue (TMQ). This queue doesn't just hold the current images; it saves the "feature states" (the AI's understanding of the images) from previous training sessions. It's like keeping a diary of what the AI learned last week, last month, and so on. This allows the AI to look back at how it understood the rare diseases in the past, even if it hasn't seen many new examples of them recently.

2. The Ghost Dancers (Temporal Anchors)
Here is the magic trick. When the AI is trying to learn about a rare disease (a "tail class"), it looks into its time-traveling diary. It picks out the "ghost dancers"—old feature representations of that rare disease from previous sessions. These ghosts are then used to create a Temporal Anchor (TAR).

Imagine the rare disease is a small island in a vast ocean. The common diseases are huge continents nearby. Without help, the ocean waves (the common disease features) wash over the island, making it hard to distinguish. The TARs act like a series of buoys or a protective barrier placed around the island. These buoys are made from the "ghost" versions of the island from the past. By placing these buoys around the rare disease, the AI effectively expands the island's territory. It creates a "buffer zone" that pushes the common diseases away, ensuring the rare disease has its own distinct space.

3. The Recurrent Loop
The process is "recurrent," meaning it happens over and over again. As the AI trains, it keeps updating its memory queue with new information, but it also keeps the old anchors. This creates a growing, expanding field around the rare diseases, making them more robust and easier to separate from the common ones.

What They Found: A Clearer Picture

The team tested this method on three real-world medical datasets:

Carotid Ultrasound: Images of blood vessels in the neck to check for plaque (3,404 images).
Diabetic Retinopathy: Eye images to check for vision loss (3,662 images).
Knee Osteoarthritis: X-rays of knees to grade arthritis severity (8,260 images).

In all three cases, the RCL method showed consistent improvements over the best existing methods.

On the Carotid dataset, the new method achieved a Balanced Accuracy of 87.10%, beating the previous best.
On the Knee dataset, it improved the Balanced Accuracy by 9.09% compared to just using the base AI model without the special memory tricks.
On the Eye dataset, the method achieved a Quadratic Weighted Kappa score of 0.92, which is a very high score indicating excellent agreement with human experts.

The researchers also visualized what was happening inside the AI's brain using a technique called t-SNE (which is like a map that shows how close or far apart different things are). They saw that without their method, the rare diseases were squished into tiny, crowded dots. With RCL, the rare diseases had expanded into larger, well-defined areas with clear "buffer zones" between them and the common diseases. The "ghost dancers" (anchors) successfully created a protective field that prevented the common diseases from crowding the rare ones out.

Why This Matters

This paper suggests that we don't just need more data for rare diseases; we need smarter ways to use the data we already have. By using the AI's own history to build a protective field around rare conditions, we can make medical AI more fair and accurate. This is crucial because, in the real world, missing a rare but severe disease can be life-threatening. The authors note that while their method works well, it currently requires some manual tuning of settings (like how many "ghosts" to keep in the memory). Future work will aim to make this process automatic. But for now, this "time-traveling memory" approach offers a promising new way to ensure that the rarest patients get the attention they deserve from our AI doctors.

Technical Summary: Recurrent Contrastive Learning for Imbalanced Medical Image Classification

Problem Statement

Medical image classification frequently encounters severe class imbalance due to the inherent disparities in disease incidence, where rare conditions (tail classes) have significantly fewer samples than common conditions (head classes). Existing approaches, such as class resampling and loss reweighting, primarily operate within the observed feature distribution. While these methods adjust how features are weighted or sampled, they fail to explicitly enlarge the latent support region of tail classes. Consequently, tail-class representations remain overly compact and are easily encroached upon by head classes in the feature space, leading to biased decision boundaries and reduced inter-class separability. This is particularly critical as tail classes often correspond to severe or rare diseases where accurate identification is vital for clinical decision-making.

Methodology: Recurrent Contrastive Learning (RCL)

The authors propose Recurrent Contrastive Learning (RCL), a framework that addresses class imbalance from a "field-expansion" perspective. Rather than merely reweighting existing data, RCL actively expands the support region of tail classes in the latent space. The methodology consists of three core components:

1. Backbone and Feature Extraction

The framework utilizes DINOv3 with LoRA (Low-Rank Adaptation) adapters as the backbone.

Architecture: An input image is processed to extract a 1280-dimensional class token ( $f_i$ ).
Heads: A classifier head ( $G(\cdot)$ ) outputs predictions for cross-entropy supervision, while a projection head ( $P(\cdot)$ ) maps features to a 256-dimensional normalized embedding ( $z_i$ ) for contrastive learning.
Training Strategy: The DINOv3 backbone is frozen, and only the LoRA adapters and the heads are optimized, ensuring efficient adaptation with few trainable parameters.

2. Temporal Memory Queue (TMQ)

To overcome the limitations of batch-level or FIFO queue-based contrastive learning, RCL introduces a Temporal Memory Queue (TMQ).

Function: The TMQ preserves corpus-level features across multiple training epochs, providing diversified global references.
Mechanism: In each epoch $m$ , the projection embedding $z_i^{(m)}$ updates the stored feature $q_i^{(m)}$ in the memory bank $Q^{(m)}$ via an Exponential Moving Average (EMA):
$q_i^{(m)} = (1 - \lambda)q_i^{(m-1)} + \lambda z_i^{(m)}$
Storage: The updated banks are enqueued over time, forming a sequence $TMQ = \{Q^{(m)}\}_{m=0}^M$ . This allows the model to access historical global embeddings, ensuring that positive and negative pairs for contrastive learning are temporally diverse and aligned with the evolving representation space.

3. Temporal Anchors (TARs) and Recurrent Contrastive Loss

The core innovation is the construction of Temporal Anchors (TARs) to form an "anchor field" around tail classes.

Anchor Construction: For tail classes, the model retrieves features from historical memory banks (epochs $m-st$). It applies K-means clustering to intra-class features and selects the bottom- $k_t$ features (those furthest from the cluster center). These selected features are perturbed with Gaussian noise to create the anchor set $A$ .
Field Expansion: These anchors are recurrently reused to occupy the surrounding region of tail-class features in the latent space. This creates a buffer margin that suppresses head-class encroachment.
Loss Function: The framework employs a recurrent contrastive loss ( $L_{RCL}$ ) that incorporates both current batch features and the constructed anchors:
$L_{RCL} = -\log \frac{\sum_{q \in Q_+^{(m)}} \exp(\text{sim}(z_i^{(m)}, q)/\beta) + \sum_{a \in A_+} \exp(\text{sim}(z_i^{(m)}, a)/\beta)}{\sum_{q \in Q^{(m)}} \exp(\text{sim}(z_i^{(m)}, q)/\beta) + \sum_{a \in A} \exp(\text{sim}(z_i^{(m)}, a)/\beta)}$
The final loss is a combination of cross-entropy and the recurrent contrastive loss: $L = L_{ce} + \alpha L_{RCL}$ .

Key Contributions

The paper outlines three primary contributions:

RCL Framework: A novel imbalanced learning framework that enlarges tail-class support regions by injecting recurrently constructed temporal anchors into contrastive learning, effectively mitigating head-class encroachment.
Temporal Memory Queue (TMQ): A mechanism to preserve corpus-level historical embeddings across training phases, which are utilized to construct TARs for tail-class field expansion.
Empirical Validation: Extensive experiments on three imbalanced medical datasets (a private multi-center carotid dataset, APTOS 2019, and KneeOA) demonstrating consistent improvements over strong baselines.

Experimental Results

The method was evaluated on three datasets using Balanced Accuracy (Bacc), F1-score, Accuracy, and Quadratic Weighted Kappa (QWK).

Performance: RCL achieved the best Balanced Accuracy on all three datasets.
- Carotid: 87.10% Bacc (vs. 79.75% for vanilla DINOv3).
- APTOS 2019: 69.01% Bacc and 0.922 QWK (surpassing the top-ranked CNN-Ensemble method from the original challenge).
- KneeOA: 70.19% Bacc (vs. 61.10% for vanilla DINOv3).
Backbone Agnosticism: When using ResNet50 instead of DINOv3, RCL still yielded substantial improvements, even surpassing the performance of DINOv3 fine-tuning on the Carotid and KneeOA datasets.
Qualitative Analysis: t-SNE visualizations confirmed that RCL creates a more balanced feature distribution. Specifically, the "anchor field" expanded the tail-class boundary and occupied intermediate regions between head and tail classes, forming an effective buffer that pushed tail-class centers away from head-class distributions.

Significance and Claims

The paper claims that RCL offers a significant shift in addressing class imbalance by moving beyond simple reweighting or resampling. By interpreting representation learning from a field-expansion perspective, the method explicitly enlarges the latent support region of tail classes. This approach prevents tail-class representations from being "encroached upon" by dominant head classes, thereby improving inter-class separation and decision boundary fairness.

The authors note that while the framework is effective, it currently relies on manual hyperparameter tuning (e.g., sample steps and anchor selection ratios). They identify the development of adaptive anchor selection as a direction for future work to address this limitation. The code is made available to facilitate further research in this domain.