💻 computer science

Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos

This paper proposes a noise-aware contrastive learning framework that leverages the inherent temporal structure of colonoscopy videos to learn robust polyp representations without costly manual annotations, achieving state-of-the-art performance across multiple downstream tasks with a lightweight encoder trained on a small dataset.

Original authors: Luca Parolari, Pietro Gori, Lamberto Ballan, Carlo Biffi, Loic Le Folgoc

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Parolari, Pietro Gori, Lamberto Ballan, Carlo Biffi, Loic Le Folgoc

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize different people in a crowded room, but you don't have any name tags. You only have a continuous video of the room.

In the world of colonoscopies (a medical camera procedure used to look inside the colon), the "people" are polyps (small growths that can turn into cancer), and the "room" is the inside of a patient's colon. The video is a long, messy stream where the camera moves, the lighting changes, and the polyps wiggle or get covered by debris.

Here is how the paper solves the problem of teaching the computer to recognize these polyps without needing a human doctor to label every single one.

The Problem: The "Labeling" Bottleneck

Usually, to teach a computer to recognize things, you need a teacher. In this case, a human expert would have to watch the entire video, find every polyp, and draw a box around it, then say, "This box at 0:05 is the same polyp as the box at 0:15."

The paper argues this is too slow, expensive, and requires too much expert time. It's like trying to teach a child to recognize their friends by having a parent write a note for every single photo the child takes.

The Solution: "Time is the Teacher"

The authors realized that colonoscopy videos have a natural rhythm. A doctor looks at a polyp, maybe removes it, and then moves to the next one. They don't usually jump back and forth randomly.

The Analogy: Imagine you are watching a movie. If you see a character on screen at minute 10, and then see them again at minute 10:05, it is almost certainly the same character. If you see them at minute 10 and then again at minute 45, it might be the same character, but it's less certain (maybe they left and came back, or maybe it's a different person who looks similar).

The paper uses this time gap as a "self-supervision" signal.

The Safe Bet: If two video clips are right next to each other in time, they are likely the same polyp.
The Risky Bet: If two clips are far apart in time, they might be the same polyp, but they might also be two different polyps that look alike.

The Innovation: The "Noise-Aware" Filter

Here is the tricky part: The "Risky Bet" is often wrong. If the computer assumes two distant clips are the same polyp when they are actually different, it gets confused and learns the wrong things. This is called "noisy data."

The authors invented a special Noise-Aware Loss (a mathematical rule for learning).

The Metaphor: Imagine a teacher grading a student's homework. Usually, if a student gets an answer wrong, the teacher marks it down. But in this new system, the teacher is smart enough to say, "I know this question is tricky and the answer key might be wrong. I won't penalize the student too hard for getting it wrong, but I will still reward them for getting the easy, obvious questions right."
How it works: The system creates "bags" of video clips. It starts with clips that are very close in time (very safe). As the training goes on, it slowly starts adding clips that are further apart in time (riskier). The "Noise-Aware" rule tells the computer: "Focus on the strong similarities, but don't let the weak, possibly wrong similarities ruin your learning."

The Results: Small Team, Big Wins

The team trained their system using a very small amount of data: just 27 videos.

The Comparison: They compared their system to:
- Other AI that tries to learn without labels (Self-Supervised).
- AI that was trained with full human labels (Supervised).
- Massive "Foundation Models" (huge AI brains trained on millions of images, like the "Dino" models).
The Outcome: Their small, lightweight system beat the other "no-label" methods by a huge margin. It also matched or even beat the massive, heavy Foundation Models on tasks like:
- Retrieval: Finding the same polyp again in a different part of the video.
- Re-identification: Deciding if two clips show the same polyp.
- Size Estimation: Guessing if a polyp is small or large.
- Histology: Guessing if a polyp is likely cancerous (adenoma) or not.

Why It Matters

The paper claims this is a "lightweight" solution. It's like building a highly efficient, smart car engine that runs on a tiny amount of fuel (27 videos) but performs just as well as a massive, fuel-guzzling engine (the huge Foundation Models). This means it could potentially run on smaller devices or be used more easily in real-world hospitals without needing supercomputers or armies of doctors to label data.

In short: They taught a computer to recognize colon polyps by using the flow of time as a guide, but they added a safety filter so the computer wouldn't get confused when the time clues were misleading. They did this with very little data and beat much larger, more complex systems.

Technical Summary: Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos

Problem Statement

Robust representation learning of polyp tracklets (sequences of consecutive detections of the same polyp) is essential for various AI-assisted colonoscopy applications, including polyp characterization, automated reporting, and lesion retrieval. While supervised contrastive learning has proven effective, it relies on "polyp entity" annotations—labels linking detections to the same underlying polyp throughout a video. Generating these labels is costly, time-consuming, and requires specialized clinical expertise, limiting the scalability of supervised approaches.

Existing self-supervised methods attempt to bypass this by splitting tracklets into disjoint segments or clustering them. However, these approaches often fail to capture the full semantic consistency of a polyp entity across a video. Conversely, recent supervised methods that utilize explicit entity annotations achieve superior performance, highlighting a significant gap between supervised and self-supervised learning. The challenge is to narrow this gap without relying on manual entity annotations, despite the highly variable visual environment of colonoscopy (changes in viewpoint, scale, illumination, motion blur, and occlusion).

Methodology

The proposed framework leverages the inherent sequential workflow of colonoscopy procedures to derive self-supervised associations from temporal structure. The core assumption is that observations captured close in time are likely to depict the same polyp entity.

1. Temporal Structure as Self-Supervision

The method constructs a dataset of tracklets $\{x_i\}$ from ground-truth polyp detections. For each anchor tracklet $i$ within a video $v_i$ , a "bag" of candidate positive tracklets is constructed from other tracklets in the same video ( $v_j = v_i$ ).

Ranking: Tracklets are sorted by their temporal distance from the anchor.
Bag Construction: Instead of selecting only the temporally closest tracklets (which yields redundant views), the method samples $K$ distinct ranks from an exponential distribution $P_\tau(r) \propto \exp(-r/\tau)$ . The temperature parameter $\tau$ controls the diversity of the bag: low $\tau$ selects close neighbors (high similarity, low noise), while high $\tau$ selects distant neighbors (high diversity, higher risk of incorrect associations).
Curriculum Learning: To balance the trade-off between view diversity and association correctness, a curriculum variable $c \in [0, 1]$ is introduced. The temperature $\tau$ is gradually increased from a minimum to a maximum value over the course of training using a cosine schedule. This allows the model to start with safe, highly similar associations and progressively learn from more diverse, potentially noisier temporal associations.

2. Noise-Aware Contrastive Loss

Since temporal proximity does not guarantee that two tracklets belong to the same polyp (e.g., two different polyps appearing in quick succession), the resulting "bags" may contain "impure" samples (tracklets from different polyps). Treating all samples in a bag as positives can degrade representation learning.

The authors propose a noise-aware contrastive loss that relaxes the standard contrastive constraint. Instead of requiring all positive samples in a bag to be similar to the anchor, the loss enforces that at least one tracklet in the bag is more similar to the anchor than any tracklet in the bag is to any other anchor.

Mathematical Formulation: The loss function utilizes a log-sum-exp approximation of the max operator. It maximizes the similarity between the anchor and the best-matching positive in the bag while minimizing the similarity between the anchor and negatives from other anchors.
Effect: This formulation allows the model to effectively "ignore" noisy (incorrect) temporal associations within a bag, preventing them from degrading the learned embedding space.

3. Architecture

The framework uses a lightweight encoder consisting of a ResNet-50 for frame embedding extraction, followed by a stack of three transformer encoder layers to aggregate global tracklet representations. The loss can be applied to the final tracklet embedding, per-frame embeddings, or both.

Key Contributions

Self-Supervised Framework: A novel approach that exploits the sequential workflow of colonoscopy to construct self-supervision signals from temporal structure, eliminating the need for manual polyp entity annotations.
Noise-Aware Contrastive Loss: A loss function designed to mitigate the impact of noisy temporal associations by ensuring that only the most consistent positive samples within a bag drive the learning process.
Comprehensive Evaluation: An extensive evaluation across four public datasets (REAL-Colon, SUN, PolypSize, PolypsSet) on multiple downstream tasks, demonstrating performance that matches or exceeds recent foundation models.

Results

The method was trained on only 27 videos (85 polyps) from the REAL-Colon dataset and evaluated on four downstream tasks:

Polyp Retrieval: Measured by mean Average Precision (mAP) and Hit Rate.
Re-identification (ReID): Measured by AUROC and AUPR.
Size Estimation: Binary classification (diminutive vs. non-diminutive) via non-linear probing.
Histology Classification: Binary classification (adenoma vs. non-adenoma) via non-linear probing.

Performance Highlights:

vs. Self-Supervised Baselines: The proposed method outperformed prior self-supervised approaches (e.g., TPC, MVE, SFE) with significant margins, including a 23.69% relative improvement in retrieval mAP and a 24.4% improvement in histology accuracy.
vs. Fully Supervised Baselines: The method consistently achieved superior or comparable performance to fully supervised baselines (which rely on expensive entity annotations). For instance, it improved AUPR by 18.56% over the best supervised baseline in ReID.
vs. Foundation Models: The lightweight encoder (25.6M parameters) trained on 27 videos matched or surpassed large general-purpose foundation models (e.g., DinoV2-giant with 1136M parameters) and endoscopy-specific foundation models across all tasks.
Ablation Studies: The study confirmed that both the exponential rank-based sampling and the curriculum learning strategy contribute to performance gains. Applying the loss to both frame and tracklet embeddings yielded the best overall results.

Significance and Claims

The paper claims that this work enables scalable representation learning in colonoscopy by significantly reducing reliance on manual annotations while maintaining high performance on clinical applications. By leveraging the procedural structure of colonoscopy videos and introducing a mechanism to handle the inherent noise in temporal self-supervision, the authors demonstrate that robust polyp tracklet representations can be learned efficiently. The ability to match or exceed the performance of massive foundation models using a lightweight architecture trained on a small dataset suggests a practical path toward deploying AI-assisted tools in clinical settings where data annotation resources are limited.