💻 computer science

Image Thresholding: Understanding Bias of Evaluation Metrics towards Specific Evaluation Functions

This paper reveals an inherent bias in image segmentation evaluation, demonstrating that Otsu's between-class variance objective function consistently correlates more strongly with standard quality metrics like SSIM and PSNR than Kapur's entropy, thereby challenging the assumption of metric neutrality.

Original authors: Eslam Hegazy, Mohamed Gabr

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eslam Hegazy, Mohamed Gabr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to cut a cake into perfect slices. In the world of computer vision, this "cake" is an image, and "cutting" it means separating different parts of the picture (like the sky from the ground, or a tumor from healthy tissue). This process is called image thresholding.

To do this, computers need a rulebook, or an Objective Function, to tell them where to cut. Two of the most famous rulebooks are:

Otsu's Method: This rulebook says, "Cut the cake where the difference between the light and dark parts is the biggest." It's like trying to maximize the contrast between the frosting and the sponge.
Kapur's Method: This rulebook says, "Cut the cake where the information content is the most balanced." It's a more complex rule based on how much "surprise" or variety is in each slice.

Once the computer makes a cut, we need to check if it did a good job. To do this, we use Evaluation Metrics, which are like judges scoring the cut. The two most popular judges in this field are:

SSIM (Structural Similarity Index): A judge that looks at how similar the shapes and structures of the cut image are to the original.
PSNR (Peak Signal-to-Noise Ratio): A judge that measures how much "static" or error was introduced during the cut.

The Big Problem: The Judge is Biased

The authors of this paper, Eslam Hegazy and Mohamed Gabr, noticed something suspicious. In almost every study comparing these methods, Otsu's Method seems to win when scored by SSIM and PSNR. Researchers assumed this meant Otsu was just a better rulebook.

But the authors asked a different question: "What if the judges (SSIM and PSNR) are secretly biased toward Otsu's rulebook?"

Think of it like a cooking competition.

Otsu is a chef who specializes in making dishes with high contrast (very salty, very sweet).
Kapur is a chef who makes balanced, complex dishes.
SSIM and PSNR are the judges.

The authors suspected that SSIM and PSNR are actually "flavor-biased" judges. They are programmed to love high-contrast dishes. So, even if Kapur makes a perfectly balanced dish, the judges might give it a lower score simply because it doesn't taste like what they prefer.

How They Tested It

To prove this, they didn't use a fancy AI to find the best cut. Instead, they did something very thorough: they tried every single possible cut on 500 different natural images (from a dataset called BSDS500).

For every single cut, they calculated:

How good the cut was according to Otsu's rule.
How good the cut was according to Kapur's rule.
How the judges (SSIM and PSNR) scored that cut.

Then, they looked at the relationship (correlation) between the rulebooks and the judges.

The Results: The Bias is Real

The findings were clear and striking:

Otsu and the Judges are Best Friends: When Otsu's rulebook said a cut was "good," the judges (SSIM and PSNR) almost always agreed. In fact, for 100% of the images, Otsu's scores moved in perfect lockstep with the PSNR scores. For 91% of the images, they moved in lockstep with SSIM.
- Analogy: It's like Otsu and the judges are speaking the same language. When Otsu raises its hand, the judges raise theirs immediately.
Kapur and the Judges are Strangers: Kapur's rulebook had a much weaker connection to the judges. Sometimes they agreed, sometimes they didn't. The relationship was messy and unpredictable.
- Analogy: Kapur is trying to speak a different dialect. The judges understand him sometimes, but often they just shrug and give a mediocre score, even if the cut was actually good.

What This Means

The paper concludes that the reason Otsu-based methods often "win" in research papers isn't necessarily because they are better at cutting the cake. It's because the scoring system (SSIM and PSNR) is rigged to favor Otsu.

If a researcher invents a new, super-smart AI algorithm to help Kapur find the best cuts, but they still use SSIM and PSNR to grade it, the new algorithm might look like a failure. Why? Because the judges are biased against Kapur's style, regardless of how good the AI is.

The Takeaway

The authors are warning the scientific community: Don't trust the judges blindly.

If you want to fairly compare different ways of cutting images, you can't just use SSIM and PSNR. You have to realize that these tools inherently prefer one specific method (Otsu) over others. To get a true picture of who is the best "chef," you might need new judges who don't have a favorite flavor.

Technical Summary: Image Thresholding and Evaluation Metric Bias

Problem Statement
Multilevel image thresholding is a fundamental technique in image segmentation, widely applied in fields ranging from medical imaging to remote sensing. Due to the combinatorial explosion of possible threshold combinations, researchers typically employ metaheuristic optimization algorithms (e.g., genetic algorithms, particle swarm optimization) to approximate optimal thresholds. These algorithms optimize predefined objective functions, most notably Otsu's between-class variance and Kapur's entropy.

A critical, yet often implicit, assumption in comparative studies within this domain is that full-reference image quality metrics—specifically Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR)—provide an objective and neutral basis for evaluating segmentation quality. However, empirical observations suggest that methods based on Otsu's criterion frequently outperform those based on Kapur's entropy when evaluated via SSIM and PSNR. This study investigates whether these performance disparities stem from genuine improvements in segmentation or from an inherent bias where evaluation metrics systematically favor specific objective functions.

Methodology
To decouple the effects of optimization algorithms from the inherent properties of objective functions and evaluation metrics, the authors conducted a systematic correlation analysis on the BSDS500 dataset, comprising 500 natural images.

Scope of Analysis: Instead of relying on the output of specific optimization algorithms, the study evaluated all possible single thresholds ( $0 < T \le L-1$ ) for each image.
Metrics and Functions: For every possible threshold, the authors computed:
- Objective Functions: Otsu's between-class variance and Kapur's entropy.
- Quality Metrics: SSIM and PSNR (calculated against the original image).
Correlation Analysis: The study calculated the Pearson correlation coefficient between the values of the objective functions and the quality metrics across the entire threshold space for each image.
Aggregation: The analysis aggregated these correlation coefficients across all 500 images to identify global trends and statistical distributions.

Key Results
The analysis reveals a strong, systematic alignment between Otsu's criterion and standard quality metrics, contrasting sharply with the weaker alignment of Kapur's entropy.

Correlation Strength: Otsu's function exhibits consistently high positive correlation with both SSIM and PSNR. In contrast, Kapur's entropy demonstrates significantly weaker and more variable correlations.
Quantitative Findings (BSDS500):
- Mean Correlation (Otsu vs. Metrics): Otsu achieved a mean correlation of $0.8959$ with SSIM and $0.9871$ with PSNR.
- Mean Correlation (Kapur vs. Metrics): Kapur achieved a mean correlation of $0.6743$ with SSIM and $0.6817$ with PSNR.
- Image-wise Dominance: For the PSNR metric, Otsu showed a higher correlation than Kapur in 100% of the 500 tested images. For the SSIM metric, Otsu showed higher correlation in 91.4% of the images.
Visual Evidence: Representative plots show that for many images, the curves of Otsu's values and the quality metrics (SSIM/PSNR) follow nearly identical trajectories, whereas Kapur's curves often diverge or exhibit low correlation coefficients (e.g., 0.14 in one sample case).

Significance and Contributions
The paper argues that the observed superiority of Otsu-based methods in literature is not solely due to the effectiveness of the optimization algorithm or the intrinsic quality of the segmentation, but is partially an artifact of metric–objective-function bias.

Re-evaluating Comparative Studies: The findings suggest that when an optimization algorithm is applied to an objective function already favored by the evaluation metric (e.g., optimizing Otsu and evaluating with PSNR), the method will naturally appear superior. Conversely, optimizing an disfavored function (e.g., Kapur) may be unjustly perceived as inferior, regardless of the optimizer's performance.
Methodological Caution: The authors highlight that relying on SSIM and PSNR as neutral benchmarks may lead to misleading conclusions in multilevel thresholding research. The dominance of Otsu in existing literature may be a reflection of this alignment rather than a universal truth about segmentation quality.
Future Directions: The study motivates the need for more neutral evaluation frameworks. It suggests extending this analysis to true multilevel thresholding (beyond bi-level), incorporating additional metrics like Feature Similarity Index (FSIM), testing other objective functions (e.g., Minimum Error Thresholding, Masi entropy), and validating these biases across different domains such as medical and infrared imaging.

In conclusion, this work provides empirical evidence that SSIM and PSNR are not unbiased indicators across different thresholding strategies. It calls for a more transparent interpretation of performance gains in the literature, acknowledging that reported improvements may stem from metric alignment rather than genuine algorithmic advancement.