💻 computer science

Understanding Model Behavior in Monocular Polyp Sizing

This paper audits monocular polyp size classification models across diverse datasets and architectures, revealing that their performance relies on examination-behavior cues rather than true metric scales, and identifies metric scale accuracy and segmentation mask robustness as two independent bottlenecks that current depth estimation and calibration methods fail to overcome.

Original authors: Xinqi Xiong, Andrea Dunn Beltran, Junmyeong Choi, Sarah K. McGill, Marc Niethammer, Roni Sengupta

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinqi Xiong, Andrea Dunn Beltran, Junmyeong Choi, Sarah K. McGill, Marc Niethammer, Roni Sengupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess the size of a balloon floating in a dark room, but you only have one camera and no ruler. If the balloon is small but very close to your camera, it looks big. If it's huge but far away, it looks small. This is the exact problem doctors face when looking at polyps (tiny growths) inside the colon using a standard camera. They have to guess if a polyp is "small" (under 5mm) or "large" (over 5mm) just by looking at the image, which leads to a lot of mistakes.

This paper is like a diagnostic audit or a "truth test" for computer programs (AI) that try to solve this guessing game. The researchers wanted to know: Are these AI programs actually measuring size, or are they just cheating by noticing patterns in how doctors move the camera?

Here is the breakdown of their findings using simple analogies:

1. The "Cheating" Pattern (Shortcuts)

The researchers found that current AI models hit a "glass ceiling." No matter how smart the model is or what kind of data they feed it, they all stop improving at about 75% accuracy.

Why? Because the AI isn't learning geometry (math); it's learning behavior.

The Analogy: Imagine a student taking a test who doesn't know the math but notices that the teacher always stands very close to the small problems and far away from the big ones. The student learns to guess "Small" when the teacher is close and "Big" when the teacher is far.
The Reality: In colonoscopy, doctors naturally move the camera closer to tiny polyps to get a better look and stay further back for big ones. The AI learned this habit. It's not measuring the polyp; it's guessing based on how close the camera is. This is called "shortcut learning."

2. The "Magic Ruler" Test (Oracle Scale)

To see if the AI could do better if it actually had a ruler, the researchers gave the computer a "Magic Ruler" (perfect, ground-truth distance data) that no real camera has yet.

The Result: When they gave the AI this perfect distance info, its accuracy jumped from 75% to nearly 89%.
The Catch: This proves the AI is capable of solving the problem, but it's currently stuck because it lacks a real metric reference (a ruler).
Current Tech Failure: The researchers tried using existing "3D depth estimation" tools (software that tries to guess distance from a flat image). These tools failed to help, acting like a broken ruler that gives random numbers. They didn't improve the score at all.

3. The "Blurry Glasses" Problem (Mask Quality)

Even if you give the AI a perfect ruler, it still needs to know exactly where the polyp is. This is called "segmentation" (drawing a box around the object).

The Analogy: Imagine you have a perfect ruler, but you are wearing foggy glasses that make the balloon look like it's in a different spot or a different shape.
The Result: When the researchers used a standard AI to draw the box around the polyp (instead of a perfect human-drawn box), the "Magic Ruler" stopped working. The accuracy dropped back down to the baseline.
The Lesson: The AI needs two things to work: a perfect ruler AND perfect glasses (a very accurate way to find the polyp). If either one is bad, the whole system fails.

4. The "Flashlight" Trick (Lighting)

The researchers also tested if they could guess distance by how bright the light looks (since light gets dimmer the further away it is).

The Result: This didn't work. Why? Because modern medical cameras have "Auto-Exposure." They automatically brighten the image if the camera moves away, keeping the brightness constant. This breaks the math the AI was trying to use.

The Bottom Line

The paper concludes that we cannot just throw more data or smarter computers at this problem to fix it. We are stuck because of two independent bottlenecks:

We lack a reliable "ruler" in standard video to tell the AI the true distance.
We lack "perfect glasses" (segmentation) that can find the polyp accurately when the video quality changes or the camera moves.

The researchers have released their "audit toolkit" (a set of tests) so other scientists can check if their new AI models are actually measuring size or just cheating by guessing based on camera habits. Until we solve the ruler and the glasses problems, AI polyp sizing will remain stuck at that 75% accuracy mark.

Technical Summary: Understanding Model Behavior in Monocular Polyp Sizing

Problem Formulation
Accurate stratification of colorectal polyp size is critical for clinical surveillance decisions, specifically distinguishing between diminutive polyps (≤5 mm) and those requiring closer follow-up (>5 mm). However, monocular colonoscopy suffers from inherent geometric scale ambiguity: under a pinhole camera model, the apparent diameter ( $A$ ) of a polyp is a function of its true diameter ( $S$ ) and the camera distance ( $Z$ ) ($A = fS/Z$). Without a metric reference, a small polyp viewed up close is visually indistinguishable from a large polyp viewed from a distance.

This ambiguity is compounded by clinical examination behavior. Endoscopists naturally move the camera closer to diminutive polyps to confirm their status while viewing larger lesions from a wider field of view. This behavior creates a spurious correlation between apparent scale and the size label in naturalistic training data. Consequently, automated models may rely on "shortcuts"—cues correlated with examination behavior rather than true geometric reasoning—leading to systematic errors when deployment conditions differ from training distributions.

Methodology and Experimental Setup
The authors present a diagnostic audit of binary polyp size classification (≤5 mm vs. >5 mm) across two public, multi-center datasets: Real-Colon and SUN-SEG. The combined dataset contains 232 unique polyps (147 ≤5 mm, 85 >5 mm) from 159 patients. To prevent data leakage, experiments utilize patient-stratified 5-fold cross-validation.

The study evaluates three distinct input modalities and model families to isolate the contributions of architecture versus data:

Appearance: A ResNet18 and a ViT-B operating on masked RGB images.
Relative Depth: A three-block CNN (CNN3) operating on relative depth maps generated by PPSNet.
Photometry/Geometry: A two-layer MLP using 51 handcrafted geometric and photometric features.

To diagnose the limitations of current models, the authors introduce three targeted interventions:

Data-Model Bias Audit: Comparing performance across modalities and architectures to identify convergence points.
Metric Scale Ladder: Applying "oracle" scale corrections at varying granularities (per-frame, per-polyp, and global) to relative depth maps. This quantifies the theoretical performance gain if perfect metric scale were available. They also test off-the-shelf metric depth estimators (MetricCol) and global calibration factors.
Mask Quality Robustness: Replacing ground-truth bounding boxes with predictions from a segmentation model (PolypPVT) and applying bounding box perturbations (±20%) to assess how segmentation errors propagate to sizing tasks.

Key Results

Performance Plateau and Shortcut Reliance: Across all architectures and input modalities (RGB, relative depth, photometry), models converge to a similar performance ceiling with a Macro-F1 score of approximately 0.75. Diagnostic analysis of the MLP features reveals that models rely heavily on "apparent size" (the ratio of bounding box area to image area), which conflates true scale with distance. This suggests the plateau is driven by missing metric information and reliance on examination-behavior cues rather than insufficient model capacity.
Limited Gains from Current Depth Estimation: Providing oracle per-frame scale information yields a significant performance boost (+16.1 percentage points, from 0.725 to 0.886), indicating substantial headroom exists if scale ambiguity is resolved. However, current metric depth estimators (MetricCol) and global calibration methods fail to recover this gain in clinical data, showing negligible improvement over relative depth baselines. The oracle gain is concentrated in "shortcut-inconsistent" frames where apparent size contradicts the true physical size, confirming that scale ambiguity is the primary failure mode in these cases.
Segmentation as an Independent Bottleneck: Segmentation quality acts as a critical, independent bottleneck. When ground-truth masks are replaced with predictions from a segmentation model trained on different data, the performance of the depth-based sizing model drops by 15.3 percentage points (0.725 to 0.572). Even with oracle per-frame scale, using predicted masks recovers performance only to the level of the unscaled baseline (0.736). This demonstrates that in-domain segmentation benchmarks do not predict downstream sizing robustness under distribution shift.
Photometry Limitations: Attempts to use photometric cues (background luminance) as a proxy for distance failed, recovering only 17% of the oracle-scale headroom. This is attributed to Auto-Exposure Control in clinical endoscopes, which adjusts sensor gain to maintain stable brightness, breaking the inverse-square law relationship between brightness and distance.

Significance and Contributions
The paper argues that progress in monocular polyp sizing is currently limited by three independent factors: the lack of accurate per-frame metric scale, the fragility of segmentation masks under domain shift, and the scarcity of large-scale, multi-center datasets with reliable size annotations.

The authors claim that their work provides a diagnostic framework for future research, offering reusable evaluation tools such as:

Oracle Scale Ladders: To quantify the theoretical upper bound of performance given perfect scale information.
Shortcut Partitions: To isolate and analyze cases where examination behavior creates misleading cues.
Mask Substitution: To audit the robustness of sizing pipelines to segmentation errors.

The study concludes that while current models perform adequately when apparent size aligns with true size, they degrade significantly when clinical behavior creates contradictions. Future advancements will likely depend on clinically grounded scale references (e.g., instrument-based anchors), improved segmentation robustness, and increased data diversity, rather than simply scaling model capacity. The code and tools are made publicly available to facilitate further auditing of polyp sizing pipelines.