💻 computer science

CCS: A Continuous Spatial-Semantic Concordance Score for Robust Evaluation of Object Detection Models

This paper proposes CCS, a continuous spatial-semantic concordance score that replaces unstable hard-threshold metrics with Gaussian-based spatial similarity and taxonomy-driven semantic similarity to provide robust, threshold-independent evaluation of object detection models, particularly in class-imbalanced and semantically structured domains like medical tongue diagnosis.

Original authors: Quoc Thai Mai

Published 2026-07-31

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Quoc Thai Mai

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show where contestants are trying to find hidden objects in a giant, messy room. In the world of computer vision, these "contestants" are AI models, and the "objects" are things like cats, cars, or in this specific case, different types of tongue symptoms for Traditional Chinese Medicine. For years, the judges have used a very strict, all-or-nothing rulebook: if the AI draws a box around an object that overlaps with the real object by at least 50%, it gets a perfect "correct" stamp. If the box is just 49% overlapping, the AI gets a "wrong" stamp, and the score is exactly the same as if the AI had missed the object entirely. It's like a dance competition where a dancer who misses a step by a millimeter gets the same score as someone who forgot the whole routine. This "hard-threshold" rule is the standard way we measure how good these AI detectives are, but it has a flaw: it ignores the fact that some mistakes are much closer to being right than others.

Now, imagine a new kind of judge who doesn't just look at the box, but also listens to the intent of the guess. This new judge understands that if an AI guesses "red tongue" when the real answer was "red-spotted tongue," it's not a total failure; it's a near-miss that shows the AI actually understands the general idea. This paper introduces a new scoring system called CCS (Continuous Spatial-Semantic Concordance Score) that acts like this smarter judge. Instead of a simple "pass/fail" switch, CCS gives partial credit. It uses math to measure how close the AI's box is to the real one (even if they don't quite touch) and how close the AI's label is to the real label in a family tree of medical terms. The authors tested this on six different AI models trying to spot tongue symptoms and found that while the old strict rules made the rankings jump around wildly depending on tiny changes in the rules, this new CCS score stayed steady and fair, recognizing that a "close call" is actually a sign of progress, not just a failure.

The Problem with "All-or-Nothing" Scoring

For a long time, the standard way to grade AI object detectors has been like a game of "hot or cold" with a very sharp cutoff. If your guess is "hot" enough (meaning the box overlaps the real object by at least 50%), you get a point. If it's "cold" (49.9% overlap), you get zero. This is the IoU (Intersection over Union) threshold. The problem is that this creates a cliff. A box that is 99% perfect and a box that is 51% perfect get the exact same score, while a box that is 49% perfect gets nothing. It's like grading a student's essay where a paper with one tiny typo gets an A, but a paper with that same typo plus a missing comma gets an F.

Furthermore, this old system treats all wrong labels as equally bad. If the AI sees a "red tongue" but calls it a "blue tongue," that's a huge error. But if it calls a "red-spotted tongue" a "red tongue," the old system treats that just as harshly as calling a "red tongue" a "blue tongue." In the real world of medicine, confusing two similar red tongues is a much smaller, more forgivable mistake than confusing red with blue. The old scoring system ignores this nuance completely.

Enter CCS: The "Partial Credit" Judge

The authors of this paper, Quoc Thai Mai, propose a new score called CCS to fix these issues. They built it with two main parts, like a two-ingredient smoothie that tastes better than either ingredient alone.

1. The Spatial Part (Csp): The "Soft" Box
Instead of treating the AI's box as a rigid rectangle with hard edges, CCS imagines the box as a soft, fuzzy cloud (mathematically, a 2D Gaussian distribution). Think of it like a heat map: the center of the box is the hottest (most confident), and it gets cooler as you move toward the edges. When the AI's fuzzy cloud overlaps with the real object's fuzzy cloud, the score doesn't drop to zero just because they don't perfectly align. Instead, it smoothly decreases based on how far apart their centers are and how different their sizes are. This means a box that is slightly off-center still gets a high score, rather than being punished as harshly as a total miss. It's the difference between a judge saying, "You missed the bullseye, but you were close, so here's a B," versus "You missed the bullseye, so here's an F."

2. The Semantic Part (Csem): The "Family Tree" of Mistakes
This is the clever second ingredient. The authors organized the eight different tongue symptoms into a family tree (a taxonomy). For example, "Red Tongue" and "Red-Spotted Tongue" are siblings in the "Color" branch, while "Enlarged Tongue" is in the "Shape" branch. If the AI guesses "Red Tongue" but the ground truth is "Red-Spotted Tongue," the old system says "Wrong." The CCS system looks at the family tree, sees they are siblings, and says, "Okay, that's a close guess. You get partial credit." It uses a mathematical tool called Wu-Palmer similarity to calculate exactly how close the guess is to the truth based on how far apart they are in the family tree. If the AI guesses a "Red Tongue" for a "Shape" problem, that's a total miss (zero credit). But if it guesses a "Red Tongue" for a "Red-Spotted Tongue," it gets a high partial score.

What They Found: Stability and Fairness

The team tested this new CCS score against six different versions of a popular AI model (YOLOv8, v10, and v11) on a dataset of tongue symptoms. Here is what happened:

The Old Rules Were Unstable: When they used the old "hard threshold" rules, the ranking of the AI models changed wildly depending on whether they set the threshold at 0.3, 0.5, or 0.7. In fact, for 5 out of 15 pairs of models, the "winner" flipped depending on which rule they used. It was like a race where the winner changed just because the finish line moved a few inches.
CCS Was Rock Solid: The CCS score stayed remarkably stable. No matter how they tweaked the settings, the ranking of the models didn't flip. The score for the models hovered consistently between 0.62 and 0.65, whereas the old F1 scores dropped significantly (by up to 16.9%) as the rules got stricter.
The "Near-Miss" Rescue: The study found that a significant chunk of what the old system called "errors" were actually just "near-misses." Specifically, 39.5% of the mistakes made on the hardest classes were semantically close (like confusing "Red" with "Red-Spotted"). The old system counted these as total failures, but CCS gave them partial credit, revealing that the AI was actually doing a better job than the old scores suggested.
The "Best" Model Changed: Under the old strict rules (mAP), one model (YOLOv10n) looked like the winner. But under the new CCS rules, a different model (YOLOv8m) took the top spot. This suggests that the old rules were favoring models that were just good at hitting the exact pixel-perfect center, while CCS favored models that understood the general concept and the shape of the symptoms better.

The Limits: What CCS Doesn't Do Yet

The authors are careful to point out that CCS isn't a magic wand that solves everything.

It's a "Class-Level" Score: Currently, CCS counts a class (like "Red Tongue") as present if the AI finds at least one good box for it. It doesn't penalize the AI if there are 20 red spots on the tongue and the AI only finds 1. The paper notes that for some symptoms, like "Red-Spotted Tongue," there can be dozens of spots in one image, and the current CCS method ignores the ones it misses.
The Family Tree is Temporary: The "family tree" of tongue symptoms they built was created by the authors to make the math work, not by a panel of medical experts. It's a "provisional" tree. If a real doctor says, "No, those two symptoms aren't actually related," the score would need to be recalculated.
One Dataset Only: They tested this on just one dataset of tongue images. While the results are promising, we don't know yet if it works perfectly for detecting cars, tumors, or other objects without more testing.

The Bottom Line

This paper suggests that we need a new way to grade AI detectives. The old "pass/fail" system is too harsh and too unstable, punishing AI for being slightly off and ignoring the fact that some mistakes are smarter than others. The CCS score offers a smoother, fairer way to evaluate AI by giving partial credit for being close in space and close in meaning. While it's not a finished product ready for every medical application yet, it shows great promise for making AI evaluation more human-friendly and clinically relevant, especially in fields like Traditional Chinese Medicine where the lines between symptoms are often blurry.

Technical Summary: CCS – A Continuous Spatial-Semantic Concordance Score

Problem Statement
Standard object detection evaluation relies on hard-threshold metrics (e.g., mAP, F1) that require Intersection over Union (IoU) to exceed a fixed value (typically 0.5) to count a detection as a true positive. The authors argue this binary decision-making process creates two critical issues in clinical and semantically structured domains:

Instability and Harsh Penalties: Continuous localization quality is collapsed into a yes/no decision. A box slightly below the threshold is penalized as harshly as a complete miss, leading to unstable model rankings as thresholds vary.
Semantic Blindness: All incorrect labels are treated as equally wrong. In domains like Traditional Chinese Medicine (TCM) tongue diagnosis, a prediction of "red-spotted tongue" for a "red tongue" ground truth is clinically distinct from a random error, yet standard metrics treat both as total failures.

Existing continuous metrics like GIoU, DIoU, and Normalized Wasserstein Distance (NWD) address spatial sensitivity but remain purely spatial, ignoring label semantics. Conversely, semantic similarity metrics exist in classification but have not been integrated into object detection evaluation.

Methodology: The CCS Framework
The paper proposes CCS (Continuous Spatial-Semantic Concordance Score), a single metric combining two complementary terms:
$CCS = \alpha \cdot C_{sp} + \beta \cdot C_{sem}$
where $\alpha + \beta = 1$ . The default setting is $\alpha = \beta = 0.5$ .

Spatial Concordance ( $C_{sp}$ ):
- Mechanism: Instead of hard IoU, bounding boxes are modeled as 2D axis-aligned Gaussians centered at the box centroid. The standard deviation ( $\sigma$ ) is derived from the box dimensions ( $\sigma = \text{width}/4$ ).
- Calculation: The score is the normalized inner product (cosine similarity) between the predicted Gaussian and the ground-truth Gaussian.
- Properties: This yields a closed-form solution that smoothly penalizes spatial deviation based on centroid distance and scale differences. It avoids the discontinuity of IoU thresholds, awarding partial credit for "near-miss" localizations.
Semantic Concordance ( $C_{sem}$ ):
- Mechanism: This term utilizes a provisional taxonomy of the 8 tongue-symptom classes (grouped into Color, Coating, and Shape branches).
- Calculation: It employs Wu-Palmer similarity, which calculates the proximity of two labels based on their distance in the taxonomy tree (specifically the depth of their Lowest Common Subsumer).
- Application: If a predicted label does not match the ground truth exactly, it receives partial credit proportional to its semantic closeness. For example, a prediction of "red-spotted tongue" against "red tongue" ground truth receives a score > 0, whereas a prediction of "enlarged tongue" would receive 0.

Experimental Setup

Dataset: An eight-class tongue-symptom subset from the TCM-Tongue benchmark, characterized by severe per-class difficulty imbalance (e.g., "hongdianshe" has high instance density, while others are sparse).
Models: Six YOLO variants spanning three generations (YOLOv8, v10, v11; n/m sizes).
Baselines: Compared against IoU, GIoU, DIoU, NWD, and standard COCO mAP@[0.5:0.95].
Metrics: Evaluated F1 scores at varying IoU thresholds (0.3, 0.5, 0.7) and the proposed CCS.

Key Results

Stability vs. Threshold Sensitivity:
- Standard F1 scores degraded significantly (10.8%–16.9% relative drop) as the IoU threshold tightened from 0.3 to 0.7.
- CCS remained stable (ranging 0.62–0.65) regardless of threshold changes, as it is computed without a matching cutoff.
Ranking Consistency:
- Hard-threshold metrics produced unstable rankings; 5 out of 15 model pairs reversed their relative ranking depending on the IoU threshold used.
- CCS provided a consistent ranking that strongly correlated with F1@0.5 (Spearman $\rho = 0.94$ ).
- Notably, the model with the highest mAP (YOLOv10n) had the lowest CCS, indicating mAP rewards pixel-precise localization on easy classes while CCS captures broader semantic and spatial robustness.
Semantic Rescue of Errors:
- The semantic component ( $C_{sem}$ ) consistently outperformed the spatial component ( $C_{sp}$ ) (0.67–0.70 vs. 0.57–0.60).
- Analysis revealed that 39.5% of errors in the hardest classes were "near-miss" confusions (semantically adjacent) rather than random failures. In specific classes like botaishe, 94.5% of errors were near-misses.
Weight Sensitivity:
- The ranking remained stable for $\alpha \in [0.5, 0.7]$ .
- Data-driven entropy weighting yielded $\alpha \approx 0.64$ , which reproduced the default equal-weight ranking, validating the robustness of the 0.5/0.5 split.

Significance and Claims
The paper positions CCS not as a finished, general-purpose replacement for mAP, but as a viable candidate for evaluation in class-imbalanced, semantically structured label spaces.

Clinical Relevance: In TCM tongue diagnosis, CCS distinguishes between clinically meaningful near-misses and genuine failures, a distinction hard-threshold metrics cannot express.
Robustness: It eliminates the instability caused by arbitrary threshold selection.
Limitations Acknowledged: The authors explicitly note that CCS currently uses class-level scoring (one pair per class per image), which fails to penalize missed instances in dense, multi-instance classes (e.g., hongdianshe). The taxonomy used is provisional and not validated by TCM experts. The evaluation is limited to YOLO variants on a single dataset.

The authors conclude that CCS should be used alongside existing metrics: mAP/F1 for pixel-precise localization, and CCS to assess whether errors are clinically benign or severe, particularly when detector outputs feed into diagnostic workflows.