Evaluation Metrics for Image Classification, Segmentation and Localization
This critical review systematically analyzes and organizes 72 evaluation metrics for image classification, segmentation, and localization to guide the selection of complementary, context-appropriate measures that ensure reproducible and clinically meaningful scientific assessments.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical imaging and computer vision, machines have become remarkably skilled at looking at pictures. They can identify a disease in an X-ray, trace the outline of a tumor in a scan, or pinpoint the exact location of a specific anatomical feature. But as these systems grow more powerful, a quiet problem has emerged: we often do not know how to measure them correctly. A computer might be excellent at finding the right answer 99 percent of the time, yet still miss the one critical case that matters most, or it might draw a boundary that is mathematically close but clinically useless. The core challenge is that the tools we use to grade these systems are often mismatched to the questions we are trying to answer. If you ask a machine to find a rare condition, a simple score of "accuracy" can be misleadingly high even if the machine finds nothing at all. If you ask it to draw a shape, a score based on how much the shapes overlap might hide the fact that the edges are in the wrong place.
Behnam Kiani Kalejahi, a researcher at Central Asian University and INTI International University, has written a comprehensive guide to untangle this confusion. His work is not a new invention of a smarter machine, but a critical map for how we should judge the machines we already have. He argues that the way we evaluate image analysis systems depends entirely on what the system is supposed to do. Whether the task is simply labeling an image, drawing a detailed mask around an object, or finding a specific point in space, the rules for success must change. The paper serves as a manual for scientists and doctors to stop using a single, one-size-fits-all number to describe performance and instead choose a small, specific set of measurements that actually reflect the real-world consequences of the system's errors.
The author begins by pointing out that the unit of measurement must match the unit of the claim. In a simple classification task, where a computer decides if an image contains a disease or not, the most important question is often how well the system finds the positive cases without raising too many false alarms. However, if the dataset is heavily skewed with healthy images, a system that simply says "no disease" to everything will look perfect on paper, even though it is useless. The paper demonstrates through constructed examples that a system can have high accuracy but zero ability to detect the rare condition it was built to find. To fix this, the author insists that researchers must report not just a single score, but a combination of metrics that tell the full story: how often the system finds the target, how often it is wrong when it says it found something, and how reliable its confidence levels are.
When the task moves from labeling to drawing, the rules change again. In segmentation, where a computer draws a mask around an organ or a tumor, the most common metric measures how much the computer's drawing overlaps with the expert's drawing. While this is useful, the paper shows that two drawings can have the same amount of overlap while being completely different in shape or location. One might be a perfect circle slightly shifted, while the other might be a scattered collection of pixels that happen to cover the same area. The author explains that relying solely on overlap scores can hide these critical failures. Instead, a complete evaluation must also measure the distance between the edges of the shapes and check if the system has accidentally merged two separate objects into one or split a single object into pieces. For tasks involving small lesions or complex structures like blood vessels, the paper suggests that the most important metric might be whether the system keeps the structure connected, rather than just how much area it covers.
For systems that locate specific points, such as finding a landmark on a face or a specific spot on a bone, the paper emphasizes the importance of physical units. A computer might be off by a few pixels, but if the image was taken with a specific type of scanner, those few pixels could represent a dangerous distance in the real world. The author warns against using normalized scores that hide the actual physical error. If a system is supposed to guide a surgeon, the measurement must be in millimeters, not in abstract ratios. The paper also highlights that the way a system handles uncertainty is just as important as its accuracy. A system that knows when it is unsure and asks a human for help is often more valuable than a system that confidently gives the wrong answer. The author provides examples showing how a system can improve its safety by refusing to make a decision on difficult cases, even if that means it processes fewer images overall.
Throughout the review, the author uses a series of ten clear, step-by-step examples to show how different ways of calculating a score can lead to completely different conclusions about which system is better. In one scenario, a system that performs poorly on large, easy cases but perfectly on small, hard ones might look worse than a system that is mediocre on everything, simply because of how the scores were averaged. In another, a system that makes a tiny mistake in a critical area might be ranked higher than one that makes a large mistake in a harmless area, depending on which metric is chosen. These examples serve as a warning that the choice of a measurement tool is not a neutral decision; it actively shapes what we consider a success or a failure. The paper argues that researchers must decide on their measurement rules before they even look at the results, ensuring that the metrics align with the actual clinical or practical goal.
The review also addresses the issue of how these systems are tested in the real world. It points out that a system might work well in a controlled lab setting but fail when faced with different types of scanners, different patient populations, or different lighting conditions. The author stresses that a single number cannot capture this complexity. Instead, researchers need to report how the system performs across different subgroups and under different conditions. The paper suggests that the most valuable comparisons are not about who has the highest score on a leaderboard, but about understanding exactly where and why a system fails. This understanding allows developers to fix specific problems rather than just chasing a higher number.
Ultimately, the paper concludes that there is no single "best" metric for image analysis. The right way to measure a system depends on what the system is supposed to do, who will use it, and what the consequences of an error might be. For a system that helps triage patients, finding every possible case is more important than avoiding false alarms. For a system that guides a surgery, precision and safety are paramount. The author provides a checklist and a set of guidelines to help researchers design their studies correctly, ensuring that they measure the right things in the right way. By following these principles, the field can move away from vague claims of high performance and toward a clearer, more honest understanding of what these powerful tools can and cannot do. The goal is not to find the perfect system, but to build systems that are understood well enough to be trusted when they are needed most.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.