⚡ electrical engineering

Investigating the Visual Cues of CNNs for Vascular Segmentation: A Case Study in Microscopy and Fundus Imaging

This study investigates the visual cues driving CNN-based vascular segmentation in microscopy and fundus imaging, revealing that models rely primarily on pixel intensity within a limited 20-pixel receptive field rather than texture or global shape, despite maintaining high accuracy even when these cues are removed.

Original authors: Weslley dos Santos Silva, Cesar Henrique Comin

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weslley dos Santos Silva, Cesar Henrique Comin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints, you are looking at a map of tiny, winding rivers inside the human body. These rivers are blood vessels, and finding them on a blurry photo is crucial for doctors to spot diseases like diabetes or heart trouble. For years, computers have gotten really good at drawing these maps using a special kind of brainy software called a Convolutional Neural Network, or CNN for short. Think of a CNN as a super-smart robot that learns to recognize patterns by looking at thousands of pictures. But here's the catch: the robot is a bit of a "black box." We know it gets the answer right, but we don't know how it decides what is a river and what is just a rock. Does it look at the shape of the river? Does it look at the color? Or is it just guessing based on a tiny speck of light? This paper is like a magnifying glass that peeks inside the robot's brain to see exactly what clues it is using to solve the puzzle.

The researchers wanted to figure out if these computer brains are actually "seeing" the vessels the way a human doctor does, or if they are just cheating by looking for easy tricks. To do this, they set up a series of clever experiments using two different types of pictures: one set from a microscope looking at tiny mouse brain vessels, and another set from a camera looking at the back of a human eye. They asked three big questions: Does the robot care more about the texture (the grainy look) or the brightness (how light or dark the pixels are)? Can it figure out the shape of the vessel if you only show it a thin outline? And how much of the surrounding picture does it need to see to get the job done?

Here is what they found. First, they tested if the robot cared about the "grain" of the image or just the brightness. They took little square pieces of the image and scrambled the pixels so the texture was completely gone, like shuffling a deck of cards until the order makes no sense. Surprisingly, the robot still did a pretty good job! This suggests that while the brightness of the pixels is the most important clue, the robot is actually quite smart and can find other hidden patterns even when the texture is destroyed. It's like if you could still recognize your friend's face even if you painted their skin a solid color, as long as the lighting was right.

Next, they tried to see if the robot could draw the whole river just by looking at a thin line down the middle (a centerline) or a hollow outline. They stripped away all the inside details of the vessels, leaving only the shape. The result? The robot got confused. It tried to draw the vessels, but it often made them too thick or filled in the wrong spots. It turns out that for these specific types of pictures, the robot cannot just look at the shape and guess the rest. It really needs to see the inside of the vessel to know how wide it is. It's like trying to guess the width of a highway just by looking at the white lines in the middle; without seeing the asphalt, you might guess it's a tiny driveway or a massive superhighway.

Finally, they wondered how much "context" the robot needs. Does it need to see the whole picture, or just a tiny patch? They tested the robot by feeding it smaller and smaller pieces of the image. They discovered that the robot only really needs to see a square patch about 32 by 32 pixels wide to do its best work. Once the patch gets bigger than that, the robot doesn't get any better. It seems the robot has a "sweet spot" where it looks at a small area—roughly 20 pixels wide—and that is enough to make a perfect decision. It doesn't need to see the whole forest to find a single tree.

In short, this paper suggests that for these specific medical images, the computer brains are relying heavily on how bright the pixels are and a small amount of local detail, rather than the big, global shape of the vessels. While this might sound like a limitation, it actually helps scientists build better, faster, and more trustworthy medical tools. By knowing exactly what clues the robot is using, doctors can be more confident that the robot isn't just guessing, but is actually looking at the right things to help save lives.

Technical Summary: Investigating the Visual Cues of CNNs for Vascular Segmentation

Problem Statement

Blood vessel segmentation is a critical procedure for diagnosing conditions ranging from diabetic retinopathy to neurodegenerative diseases. However, Convolutional Neural Networks (CNNs) used for this task are often treated as "black boxes," lacking transparency regarding the specific visual features that drive their decisions. While CNNs achieve high accuracy, it remains unclear whether they rely on anatomical continuity (global shape), local pixel intensity, or texture patterns. This lack of interpretability hinders clinical integration, model debugging, and the ability to ensure robustness against data variations. Furthermore, the relationship between the network's Effective Receptive Field (ERF) and the spatial context required to capture the complex, branching geometry of vascular trees is not systematically understood in this domain.

Methodology

The authors conducted a systematic, three-part investigation using two distinct imaging domains: fluorescence microscopy (VessMAP dataset) and retinal fundus photography (DRIVE dataset). The study employed targeted experiments to isolate and quantify the influence of texture, shape, and receptive field.

1. Quantifying Texture and Intensity Influence

To determine the relative importance of intensity versus texture, the authors developed a patch-based classification task.

Patch Extraction: $9 \times 9$ pixel patches were extracted from VessMAP images. Two scenarios were tested: patches containing only vessel pixels ( $f=100\%$ ) and patches containing vessel borders with varying fractions of vessel pixels ( $f=75\%, 50\%$ ).
Perturbations: Three variations of the dataset were created to remove specific cues:
- Intensity Normalization ( $D_{-i}$ ): Pixel values were normalized to zero mean and unit standard deviation, removing absolute intensity and variance.
- Pixel Shuffling ( $D_{-t}$ ): Pixel positions within each patch were randomly shuffled, destroying spatial texture while preserving intensity values.
- Homogenization ( $D_{-i,-t}$ ): Both normalization and shuffling were applied.
Evaluation: A simple residual network was trained to classify these patches as vessel or background.

2. Isolating Shape Cues

To assess the ability of CNNs to segment vessels based solely on geometry, the authors created two datasets where internal vessel appearance was removed:

Contour Dataset ( $D_c$ ): Input images contained only the vessel contours (edges), with original pixel intensities preserved at the contour locations.
Centerline Dataset ( $D_s$ ): Input images contained only the vessel centerlines, with original intensities preserved at these points.
Evaluation: U-Net and W-Net architectures were trained to predict full segmentation masks from these sparse shape inputs.

3. Analyzing Receptive Field (RF) Requirements

The study examined the relationship between model architecture, receptive field size, and performance using two approaches:

Architectural Variation: 160 U-Net configurations were trained with varying kernel sizes ($1, 3, 5, 7$), dilation ratios ($2, 3, 7$), and downsampling strategies. Theoretical Receptive Field (TRF) and Effective Receptive Field (ERF) were measured for each.
Context Restriction: The base model was trained on non-overlapping patches of varying sizes ( $4 \times 4$ to $512 \times 512$ ) to determine the minimum context required for optimal performance.

Key Results

Texture vs. Intensity

Intensity Dominance: Pixel intensity was found to be more critical than local texture. Removing intensity information ( $D_{-i}$ ) caused a larger accuracy drop than removing texture via shuffling ( $D_{-t}$ ).
Robustness to Perturbation: Surprisingly, even when both intensity and texture cues were removed (Homogenized dataset $D_{-i,-t}$ ), the networks maintained classification accuracy above 75%, suggesting they leverage other statistical features not explicitly analyzed.
Role of Borders: Including vessel borders in patches significantly improved classification accuracy, indicating that local edge-based context is a strong cue.

Shape Cues

Insufficiency of Shape Alone: CNNs failed to reliably reconstruct full vessel geometry from shape cues alone.
- Using contours ( $D_c$ ), the best model (W-Net) achieved a Dice score of 54.4%.
- Using centerlines ( $D_s$ ), the best model (U-Net) achieved a Dice score of 57.0%.
- In contrast, models trained on full data achieved ~90% Dice.
Oversegmentation: Models relying solely on shape cues exhibited high sensitivity but very low specificity (e.g., 12% specificity for U-Net on $D_c$ ), leading to significant oversegmentation. This indicates that current CNNs cannot extrapolate vessel caliber from boundary geometry without internal appearance information.

Receptive Field and Context

ERF Saturation: The Effective Receptive Field (ERF) utilized by the networks converged to approximately 20 pixels for the tasks considered.
Kernel Size: Performance increased with average kernel size but plateaued around an average kernel size of 3 ( $3 \times 3$ kernels). Larger kernels did not yield significant gains, suggesting the ERF does not expand to utilize the full theoretical field when not necessary.
Patch Size:
- For VessMAP (microscopy), performance saturated at a patch size of $32 \times 32$ , indicating that local context is sufficient.
- For DRIVE (fundus), performance did not saturate within the tested range, suggesting a modest benefit from larger global contexts, likely due to the presence of global anatomical landmarks (e.g., optic disk).

Significance and Claims

The paper claims to provide a quantitative foundation for auditing and refining deep learning systems in vascular imaging. Its primary contributions are:

Diagnostic Template: The methodology offers a structured approach to disentangle the influences of shape, texture, and context on model decisions.
Empirical Evidence on RF: The finding that the ERF saturates at ~20 pixels suggests that lightweight, efficient CNNs can be tailored for local vessel delineation without the computational overhead of unnecessarily deep networks or global attention mechanisms (like Transformers).
Interpretability Roadmap: The results highlight that models rely heavily on local intensity and texture rather than global shape extrapolation. This insight is proposed as a basis for developing Explainable AI (XAI) frameworks that can attribute decisions to specific cues (intensity vs. shape), helping clinicians determine if a model is identifying vessels based on anatomy or being misled by local artifacts.

The authors remain modest, acknowledging that these findings are empirical observations derived from specific datasets (VessMAP, DRIVE) and architectures (U-Net variants), and may not represent universal properties of all neural networks across all medical modalities.