🤖 machine learning

CONSIGN: Conformal Segmentation Informed by Spatial Groupings via Decomposition

The paper introduces CONSIGN, a conformal prediction framework that leverages spatial groupings to transform heuristic pixel-wise scores into statistically valid, less conservative, and more interpretable uncertainty estimates for image segmentation across medical and general datasets.

Original authors: Bruno Viti, Elias Karabelas, Martin Holler

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bruno Viti, Elias Karabelas, Martin Holler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a medical scan, like an MRI of a heart, and an AI is trying to tell you exactly where the heart muscle ends and the blood begins. The AI is usually very confident, pointing to every single pixel and saying, "This is heart," or "This is blood."

But what if the AI is wrong? In high-stakes fields like medicine, knowing how sure the AI is, is just as important as the prediction itself.

This paper introduces a new method called CONSIGN (Conformal Segmentation Informed by Spatial Groupings via Decomposition). Here is how it works, explained through simple analogies.

The Problem: The "Lonely Pixel" Mistake

Most current AI systems treat every pixel in an image like a lonely person standing in a crowd, making decisions in total isolation.

The Old Way: If the AI is unsure about a specific pixel, it might say, "I think this is a heart, but I'm 50% sure it's blood." To be safe, it creates a "prediction set" (a list of possibilities) that includes both heart and blood for that pixel.
The Flaw: Because the AI treats every pixel independently, it ignores the fact that pixels are neighbors. In a real image, pixels don't change labels randomly; they form smooth shapes. If the AI is unsure about one pixel, it's likely unsure about the pixels right next to it too.
The Result: The old method creates a "prediction set" that is huge and messy. It's like saying, "For this entire image, the answer could be any combination of heart and blood pixels." This is statistically safe, but it's so broad it's not very useful. It's like a weather forecast that says, "Tomorrow, it will either rain, snow, be sunny, or be a tornado," which is technically true but doesn't help you plan your day.

The Solution: The "Group Hug" Approach (CONSIGN)

The authors, Bruno Viti, Elias Karabelas, and Martin Holler, realized that pixels in an image are like a choir. They don't sing random notes; they sing together in harmony. If one singer is off-key, the neighbors are likely off-key too.

CONSIGN changes the game by looking at the image as a whole group rather than a collection of lonely pixels.

Listening to the Choir (SVD): The method uses a mathematical tool called Singular Value Decomposition (SVD). Think of this as listening to the choir and identifying the main "themes" or "patterns" of uncertainty. Instead of asking, "Is pixel #1 unsure?" it asks, "What are the main ways the whole image could be wrong?"
- Analogy: Imagine a sheep and a cow standing next to each other. If the AI is confused, it might confuse the boundary between them. The "main theme" of uncertainty isn't just one pixel; it's the entire wavy line separating the two animals.
Building a Smarter Safety Net: Instead of listing every possible pixel combination, CONSIGN builds a prediction set based on these main "themes."
- It says, "The AI might be wrong about the boundary between the sheep and the cow, so let's include a few variations of that specific boundary."
- It ignores impossible scenarios, like a pixel on the sheep suddenly becoming a pixel on the cow without the rest of the sheep changing.
The "Guaranteed" Promise: The paper uses a statistical framework called Conformal Prediction. Think of this as a "money-back guarantee" for the AI's confidence.
- The user sets a rule: "I want to be 95% sure that the true answer is in our list of possibilities."
- CONSIGN adjusts its "safety net" size until it mathematically guarantees that 95% of the time, the real answer is inside the list.
- Because it understands the "group" nature of pixels, it can make this safety net much smaller and tighter than the old "lonely pixel" methods, while still keeping the same 95% guarantee.

The Results: Tighter, Smarter Predictions

The authors tested CONSIGN on medical images (hearts and lungs) and general object images (animals and vehicles).

Smaller Lists: The "prediction sets" (the list of possible answers) generated by CONSIGN were significantly smaller and more focused than those from previous methods.
Better Logic: When they visualized the results, CONSIGN produced smooth, realistic shapes (like a clean line between a sheep and a cow). The old methods often produced "noisy" results with random, scattered pixels that didn't make sense visually.
Efficiency: Even though the math is complex, the method is fast enough to be practical. It doesn't need to check every single possibility; it just checks the main "themes" of uncertainty.

In a Nutshell

Imagine you are trying to guess the shape of a cloud.

The Old Method looks at every drop of water individually. If it's unsure about one drop, it says, "This drop could be water or air." It does this for every drop, resulting in a giant, messy cloud of possibilities that covers the whole sky.
CONSIGN looks at the cloud as a whole. It realizes, "Ah, the edge of the cloud is fuzzy." It then says, "The cloud is definitely here, but the edge might wiggle a little bit." It gives you a much clearer, more accurate picture of the cloud's shape, with a mathematical guarantee that it's right.

The paper claims that by respecting the fact that pixels are neighbors (spatial groupings), we can get much more useful and reliable uncertainty estimates for AI image segmentation, especially in critical fields like medicine.

Technical Summary: CONSIGN - Conformal Segmentation Informed by Spatial Groupings via Decomposition

1. Problem Definition

Machine learning-based image segmentation models typically output pixel-wise confidence scores (e.g., softmax probabilities) that serve as heuristic indicators of uncertainty. While valuable in high-stakes domains like medical imaging, these scores lack rigorous statistical guarantees. Conformal Prediction (CP) offers a framework to transform these heuristics into statistically valid uncertainty estimates with user-specified error guarantees (Conformal Risk Control).

However, a critical limitation of existing CP applications in segmentation is the pixel-wise independence assumption. Standard methods construct prediction sets for each pixel separately, ignoring the fundamental spatial correlations inherent in image data. This disregard for spatial structure leads to:

Overly conservative estimates: The resulting prediction sets (sets of possible segmentation masks) are unnecessarily large.
Incoherent predictions: The sets may contain combinatorial label configurations that are spatially inconsistent or unrealistic (e.g., noisy, fragmented masks).
Reduced interpretability: The lack of structural constraints makes the uncertainty estimates less meaningful for decision-making.

2. Methodology: CONSIGN

The authors propose CONSIGN (Conformal Segmentation Informed by Spatial Groupings via Decomposition), a method that integrates spatial correlations into the conformal prediction framework. The approach is designed to be compatible with any pre-trained segmentation model capable of generating multiple sample outputs (e.g., via Monte Carlo dropout, Bayesian modeling, or ensembles).

The method consists of two primary stages:

A. Construction of Spatially-Aware Sets

Instead of treating pixels independently, CONSIGN leverages Principal Component Analysis (PCA) via Singular Value Decomposition (SVD) to capture the joint variations in uncertainty across the image.

Sample Generation: Given a pre-trained model $f$ , the method generates $N_s$ sample outputs $\hat{s}_1, \dots, \hat{s}_{N_s}$ for a given input image $X$ .
SVD Decomposition: The samples are centered around their mean $\mu(X)$ and decomposed:
$\hat{S} - \mu(X) \cdot \mathbf{1}^T = U \Sigma V^T$
The columns of $U$ (denoted $u_k$ ) represent the principal directions of uncertainty. The authors note that a truncated SVD using a small number of components ( $K \in \{2, 5\}$ ) is sufficient to capture the essential spatial structure, significantly reducing computational complexity compared to a full basis.
Coefficient Bounds: For each principal component $k$ , the method computes empirical quantiles ( $a_k, b_k$ ) of the projection coefficients over the calibration samples. These define a baseline interval.
Prediction Set Definition: The prediction set $C^*_\lambda(X)$ is defined as the set of all label maps $Y$ that can be reconstructed by a linear combination of the first $K$ principal components, where the coefficients $c_k$ lie within bounds scaled by a parameter $\lambda$ :
$c_k \in \left[ \frac{a_k+b_k}{2} - \lambda \Sigma_{k,k} \frac{b_k-a_k}{2}, \quad \frac{a_k+b_k}{2} + \lambda \Sigma_{k,k} \frac{b_k-a_k}{2} \right]$
The final label map is obtained via a non-linear projection $P(\cdot)$ (argmax) applied to the reconstructed score vector. This non-linearity allows the method to enforce label-wise accuracy constraints ( $\beta$ ) while maintaining a low-dimensional representation.

B. Calibration

The method employs a calibration procedure to determine the optimal $\hat{\lambda}$ that satisfies the Conformal Risk Control guarantee:
$\mathbb{E}[\ell(C(X_{test}), Y_{test})] \leq \alpha$
where $\ell$ is a bounded loss function measuring the failure of the true label to be in the prediction set.

Optimization Challenge: Unlike pixel-wise methods where checking if a ground truth is in the set is trivial, checking if $Y \in C^*_\lambda(X)$ in CONSIGN requires finding if there exists a coefficient vector $c$ within the bounds such that the reconstructed mask matches $Y$ (up to accuracy $\beta$ ).
Solution: The authors formulate this as a constrained minimization problem to find the optimal coefficients $c^*$ that minimize the loss between the ground truth and the reconstructed mask. If the numerical solver finds a solution within the bounds, the ground truth is considered covered.
Guarantee: Even if the solver fails to find the global optimum (leading to a potentially more conservative $\lambda$ ), the statistical coverage guarantee remains valid.

3. Key Contributions

Spatial-Aware Conformal Prediction: The paper introduces a novel framework that explicitly incorporates spatial correlations into conformal prediction sets for image segmentation, moving beyond the pixel-wise independence assumption.
SVD-Based Decomposition: By utilizing SVD to extract principal directions of uncertainty, the method constructs prediction sets that are structurally coherent and significantly smaller (lower uncertainty volume) than pixel-wise baselines.
Efficiency with Truncation: The authors demonstrate that meaningful prediction sets can be achieved with very few principal components ( $K=2$ or $5$), making the approach computationally feasible despite the non-linear projection step.
General Applicability: The method is model-agnostic, requiring only the ability to generate multiple samples from a pre-trained model, and is applicable to both medical and natural image datasets.

4. Experimental Results

The authors evaluated CONSIGN against two baselines:

Pixel-wise RAPS: A standard conformal risk control method treating pixels independently.
SACP (Spatial-Aware Conformal Prediction): A method that aggregates softmax scores over local neighborhoods.

Datasets: Three medical imaging datasets (M&Ms-2, MS-CMR19, LIDC) and two COCO subsets (animals, vehicles).
Metrics:

Chao Estimator: Estimates the number of unique segmentations in the prediction set (uncertainty volume).
Sample-based Estimated Coverage (sEC): Measures how quickly the empirical coverage converges to the target $1-\alpha$ .
Correlation: Measures the internal correlation of sampled segmentations.

Findings:

Reduced Uncertainty Volume: CONSIGN consistently produced prediction sets with significantly lower uncertainty volumes (fewer unique segmentations) compared to both RAPS and SACP. In the LIDC dataset, the difference was several orders of magnitude.
Improved Efficiency: CONSIGN achieved the required coverage guarantees with fewer samples than the baselines, indicating that the prediction sets occupy a lower-dimensional subspace.
Structural Coherence: The samples generated by CONSIGN exhibited higher internal correlation and smoother transitions between regions, avoiding the "noisy" or combinatorially inconsistent masks often seen in pixel-wise approaches.
Baseline Limitations: The SACP method, which aggregates scores post-hoc, performed comparably to the pixel-wise RAPS, suggesting that simple aggregation does not effectively capture the true spatial dependencies required for tight uncertainty bounds.

5. Significance and Limitations

Significance:
The paper claims that CONSIGN provides a robust tool for generating "meaningful" and "interpretable" uncertainty estimates in image segmentation. By acknowledging spatial correlations, it produces tighter prediction sets that are more useful for critical decision-making, particularly in medical imaging where false positives/negatives have high costs. The method transforms heuristic confidence scores into statistically valid bounds without requiring retraining of the underlying segmentation model.

Limitations:

Exchangeability Assumption: Like all standard conformal prediction methods, the guarantees hold only under the assumption that calibration and test data are exchangeable (e.g., i.i.d.). The method does not currently address distribution shifts or out-of-distribution data.
Computational Cost: The implicit form of the prediction set requires solving a numerical optimization problem during calibration, which increases offline computational time compared to pixel-wise methods. However, the authors note that online sampling remains fast.
Solver Optimality: The coverage guarantee relies on the numerical solver finding a suitable coefficient vector. While the statistical guarantee holds even if the solver is conservative (increasing $\lambda$ ), this may lead to wider bounds than theoretically necessary in pathological cases.