Evaluation Metrics for Image Classification, Segmentation and Localization Mathematical Foundations, Comparative Analysis and Research Recommendations
This critical review synthesizes 150 references to systematically categorize and compare 72 evaluation metrics for image classification, segmentation, and localization, offering mathematical foundations, worked examples, and research recommendations to guide the selection of appropriate, complementary metrics for reproducible and clinically valid studies.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical imaging, computers have become remarkably skilled at looking at pictures of the human body. They can spot a tumor, trace the outline of an organ, or pinpoint a specific landmark on a bone. But there is a hidden trap in how we judge these machines. Imagine a student taking a test where the teacher only grades them on the final score, without looking at the specific questions they got wrong. If the student guessed the right answer on a few hard questions but missed dozens of easy ones, a simple score might hide the fact that they are actually failing the subject. In the same way, a computer model might appear brilliant because it gets a high number on a standard test, while secretly failing to find the rare diseases that doctors need to catch most urgently, or drawing the edges of a tumor so poorly that a surgeon cannot use the map. The core problem is that the tools we use to measure success often do not match the real-world job the computer is supposed to do.
Two researchers, Behnam Kiani Kalejahi and Mohammad Javad Rajabi, have written a comprehensive guide to fix this mismatch. They did not build a new computer program to find diseases. Instead, they built a new way of thinking about how to measure the programs that already exist. They reviewed hundreds of existing studies and mathematical methods to create a clear map of what different measurements actually tell us. Their work is a critical review, meaning they analyzed the rules of the game rather than playing the game themselves. They found that the field is full of confusion because scientists often use the same words to describe different things, or they pick a measurement that makes their model look good while hiding its failures. The paper argues that there is no single "best" score. Instead, a reliable evaluation requires a small, carefully chosen set of different measurements that answer specific questions about how the computer behaves.
The researchers organized their findings by the type of task the computer is performing. For the task of simply deciding if a disease is present or absent, they explain that a high overall accuracy can be misleading if the disease is rare. They show that a model could be right 99 percent of the time simply by saying "no disease" for every single patient, yet it would fail to find a single sick person. To avoid this, they insist that scientists must report how well the model finds the sick people and how often it falsely alarms healthy ones, rather than just giving a single average number. They also point out that a model's confidence score is not the same as its actual accuracy. A computer might be very confident about a wrong answer, and standard tests often fail to catch this. The authors suggest that researchers must check if the computer's confidence matches reality, ensuring that when it says it is 90 percent sure, it is actually right 90 percent of the time.
When the task moves from simple yes-or-no answers to drawing outlines, the rules change again. In this area, known as segmentation, the computer must draw a shape around a specific organ or lesion. The researchers demonstrate that two models can have the exact same score for how much their drawing overlaps with the correct shape, yet one model might be drawing a perfect circle in the wrong place, while the other is drawing a jagged line that touches the right spot. They show that a single number cannot tell the difference between a small error in the middle of a large organ and a massive error on a tiny, critical structure. They found that scientists often ignore the edges of these drawings, but in medicine, the edge is often the most important part. If a computer misses a tiny piece of a tumor boundary, it could lead to a patient receiving too little treatment. The paper urges researchers to measure the distance between the drawn line and the true line, not just how much the shapes overlap.
For the task of finding and counting specific objects, like individual cells or tumors, the researchers highlight how easily results can be manipulated by how the computer matches its findings to the truth. They explain that if a computer finds the same tumor twice, it should be counted as a mistake, but many standard tests do not catch this. They also show that the way researchers average their results can completely change the ranking of which computer is best. In one of their detailed examples, they created a scenario where one computer looked better when you averaged the scores of each patient, but a different computer looked better when you averaged the total number of pixels. Both computers were the same; only the math changed the story. This proves that the choice of how to combine numbers is just as important as the numbers themselves.
The paper also addresses the issue of uncertainty and the human element. In a real hospital, a computer does not work in a vacuum; it works alongside doctors. The researchers show that a model that refuses to answer when it is unsure can be more useful than one that guesses confidently and gets it wrong. They describe a method where a computer can say "I don't know" and refer the case to a human doctor. This improves the overall safety of the system, but standard tests often ignore these referrals and only look at the cases the computer tried to solve. The authors argue that a complete evaluation must count the referrals and the errors that happen even after a human checks the work. They also warn against using "fidelity" scores, which measure how realistic a generated image looks, as a substitute for checking if the computer actually found the disease. A picture can look beautiful and perfect while missing the critical medical detail entirely.
Throughout their analysis, the authors provide ten specific, worked-out examples to prove their points. These are not real patient data, but carefully constructed scenarios that act like controlled experiments. In one, they show how a model can have a high score for finding tumors but fail to find any of the rare, dangerous ones because the test was weighted toward common cases. In another, they demonstrate how a tiny shift in the physical size of a pixel can make a computer look like it is performing better or worse, even though its actual performance has not changed. These examples serve as a warning that the numbers reported in scientific papers can be fragile and depend entirely on the specific rules used to calculate them.
The researchers conclude that the field needs to stop chasing a single leaderboard score. Instead, they propose a checklist for anyone designing a study. This checklist forces scientists to define exactly what they are measuring, who they are measuring it on, and what the rules are for counting a success or a failure before they even look at the results. They insist that the unit of measurement must match the unit of the claim. If a doctor cares about the patient, the score must be about the patient, not just the individual image slices. If a surgeon needs to know the edge of a tumor, the score must measure the edge, not just the volume. The paper does not claim to have solved the problem of medical AI, but it provides the essential tools to stop measuring the wrong things. By aligning the measurement with the actual goal, researchers can finally see which models are truly helpful and which are just good at passing tests.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.