💻 computer science

Who Gets Missed in the Tail? Thresholded Subgroup Underdiagnosis in Long-Tailed Chest X-ray Classification

This paper demonstrates that in long-tailed chest X-ray classification, acceptable ranking metrics can mask severe underdiagnosis of rare positive cases within specific subgroups, necessitating a combined approach of subgroup-aware weighting and tail-specific threshold selection to significantly reduce false negative rates across diverse patient demographics.

Original authors: Ha-Hieu Pham, Hai-Dang Nguyen, Dang P. M. Cao, Thanh-Huy Nguyen, Min Xu, Trung-Nghia Le, Ulas Bagci, Huy-Hieu Pham

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ha-Hieu Pham, Hai-Dang Nguyen, Dang P. M. Cao, Thanh-Huy Nguyen, Min Xu, Trung-Nghia Le, Ulas Bagci, Huy-Hieu Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, super-smart robot doctor named "CXR-Bot" that looks at chest X-rays to spot diseases. It's really good at its job, but it has a weird blind spot: it's great at spotting common colds but often misses rare, tricky diseases. Even worse, it seems to miss these rare diseases more often in specific groups of people, like older adults or certain genders.

The big question this paper asks isn't just "Is the robot smart?" but "Who does the robot miss when it has to make a final 'Yes' or 'No' decision?"

The Score vs. The Decision

Think of the robot's brain as a scoreboard. When it looks at an X-ray, it gives every possible disease a "score" (like a grade from 0 to 100). A high score means "I think this disease is here."

But the robot doesn't just shout out scores; it has to make a decision. It uses a threshold, which is like a finish line. If a disease's score crosses the line, the robot says, "Alert! Check this!" If it doesn't cross, the robot says, "All clear," and the patient goes home.

The paper found that even if the robot's scores are pretty good, the way it sets that finish line can cause it to miss rare diseases in specific groups of people. It's like having a race where the finish line is moved so high that the slowest runners (the rare diseases) never get a medal, even if they were running just fine.

The "Diagnostic Ladder" Experiment

The researchers didn't just guess; they built a "diagnostic ladder" to test different ways of training the robot and setting that finish line. They used two different groups of X-rays:

VinDr-CXR: A smaller group with about 1,500 test images.
MIMIC-CXR: A massive group with nearly 79,000 test images.

They tried different training tricks (like giving extra points for rare diseases) and different ways to set the finish line.

The Big Surprise: Moving the Finish Line Works Best

Here is the most important part: The researchers found that simply changing how the robot sets its finish line for rare diseases worked better than changing how the robot was trained.

On the smaller group (VinDr), they tried a special "tail-aware" finish line.

Before: The robot missed 66.5% of the rare disease cases. For the worst group (older patients), it missed 82.2% of them!
After: By just adjusting the finish line, the robot missed only 26.9% of the rare cases. For the older patients, it missed only 13.3%!

That is a huge drop in missed patients. The paper suggests that simply moving the finish line lower for rare diseases helps the robot catch more of them without needing to rebuild the robot's brain.

However, on the massive group (MIMIC), the problem was much harder. Even with the new finish line, the robot still missed a lot of rare cases (dropping from missing 86.6% to 74.1%). This suggests that while moving the finish line helps, it doesn't magically fix everything, especially when the data is huge and complex.

What the Paper Says is NOT the Answer

The paper is very clear about what doesn't solve the problem:

Just making the robot "fair" in general isn't enough. They tested a method called "GroupDRO" that tries to make the robot fair to all groups at once. It didn't work as well as simply adjusting the finish line for rare diseases.
Good "ranking" scores don't tell the whole story. The robot could have a high "macro-mAP" score (a fancy way of saying it ranks diseases well overall), but still miss the specific rare cases in specific groups once it has to make a decision. The paper argues that you can't just look at the ranking score and assume everyone is safe.

The Catch: More Alerts Mean More Work

There is a trade-off. When the robot catches more rare diseases by lowering the finish line, it also sends out more "Alerts."

On the small group, the number of alerts jumped from about 5 per 100 patients to 39 per 100 patients.
The paper notes that this is a "clinical policy choice." Doctors have to decide: "Do we want to catch more rare diseases and look at more X-rays, or do we want to look at fewer X-rays and risk missing some?"

The Bottom Line

The paper concludes that "rare-label fairness" isn't just about the robot's brain or how common a disease is. It depends on three things working together:

The Disease: Is it rare?
The Group: Is the patient in a subgroup that gets missed?
The Threshold: Where did we set the finish line?

The authors suggest that we need to measure "missed positives per 100 true positives" (how many sick people we send home) rather than just looking at general accuracy scores. They don't claim to have "solved" the problem, especially for the massive dataset where misses are still high. Instead, they offer a new way to measure the problem so doctors and engineers know exactly who is getting missed and why, before they deploy these robots in real hospitals.

Technical Summary: Thresholded Subgroup Underdiagnosis in Long-Tailed Chest X-ray Classification

1. Problem Definition

The paper addresses a specific fairness gap in long-tailed multi-label Chest X-ray (CXR) classification: the discrepancy between acceptable ranking performance (e.g., macro-mAP) and the actual deployment outcome where rare-positive patients, particularly within specific subgroups, remain undetected below the decision threshold.

While existing long-tail methods focus on class-level reweighting and fairness methods often aggregate over groups, this work argues that the critical failure mode is thresholded class-subgroup underdiagnosis. A model may achieve high aggregate metrics while systematically missing rare findings (the "tail") for specific demographic groups (e.g., specific age or sex combinations) once scores are converted into binary clinical decisions. The core audit question is: After a long-tailed model is converted from scores to decisions, who is missed?

2. Methodology

The authors employ a "diagnostic ladder" approach using model variants as probes to isolate the source of underdiagnosis. The study utilizes two cohorts: VinDr-CXR (limited to sex and age metadata) and MIMIC-CXR/CXR-LT (richer metadata including race and insurance). All models share a ConvNeXt-Tiny backbone with a dataset-specific linear head.

The diagnostic ladder consists of six model configurations (M1–M6):

M1 (Baseline): Binary Cross-Entropy (BCE) loss.
M2: Asymmetric Loss (ASL).
M3: ASL with Effective-Number class weighting (CB-ASL).
M4: Group Distributionally Robust Optimization (GroupDRO) as a reference for aggregate group robustness.
M5: ASL multiplied by a clipped class-subgroup weight based on training counts (subgroup-aware training).
M6 (Thresholding Probe): Reuses M5 scores but alters the thresholding rule. While head/medium classes use F1-optimal thresholds, tail-class thresholds are set to maximize worst-group recall subject to a minimum precision floor ( $\tau = 0.05$ ).

Key Experimental Design:

Separation of Variables: By comparing M5 and M6, the study isolates the effect of threshold selection from score-model training, as both share identical ranked scores, ECE, and Brier scores.
Metrics: The study moves beyond ranking metrics (macro-mAP, tail-mAP) to focus on Thresholded False Negative Rate (FNR), Worst-Group FNR per subgroup axis, and Missed Positives per 100 True Positives.
Uncertainty Quantification: Due to small sample sizes in tail classes (e.g., 3–19 positives per class in VinDr), the authors report independent bootstrap 95% confidence intervals and paired bootstrap contrasts to ensure statistical rigor.

3. Key Results

VinDr-CXR Findings:

Thresholding Impact: Moving from standard F1 thresholds (M5) to tail-aware thresholds (M6) drastically reduced the tail FNR from 0.471 to 0.269.
Subgroup Improvements: The worst-group FNR for sex dropped from 0.507 to 0.157, and for age from 0.633 to 0.133.
Macro Metrics: Macro-mAP remained stable (0.635 for both M5 and M6), confirming that the improvement was driven by threshold selection, not score re-ranking.
Specific Disease Impact: Improvements were driven by specific tail classes with sufficient support (e.g., Calcification, ILD, Consolidation), while classes with extremely low support (e.g., Atelectasis with $n=3$ ) showed unstable results.

MIMIC-CXR/CXR-LT Findings:

Consistent Direction: The same thresholding strategy (M6) reduced tail FNR from 0.866 to 0.741 and lowered worst-group FNR across sex, age, race, and insurance.
Residual Risk: Despite reductions, absolute missed-positive rates remained high (e.g., 0.741 tail FNR), indicating that thresholding alone does not solve the tail problem but significantly mitigates the worst-case subgroup misses.
GroupDRO Limitations: The GroupDRO baseline (M4) did not outperform M6 on thresholded tail FNR, suggesting that aggregate group robustness alone is insufficient to address rare subgroup misses in this setting.

Trade-offs:

Reducing misses via lower thresholds increases the "alert burden" (predicted tail-positive labels per 100 studies). For VinDr, this increased from 4.9 (F1 threshold) to 38.9 (tail-aware threshold), highlighting a clinical policy trade-off between sensitivity and workflow load.

4. Contributions

The paper makes four primary contributions:

Audit Unit Definition: It defines thresholded class-subgroup underdiagnosis as the fundamental unit for auditing long-tailed CXR fairness, shifting focus from rare labels or groups in isolation to the intersection of finding, subgroup, and operating threshold.
Diagnostic Ladder: It introduces a methodology to separate class-level losses, subgroup-aware weighting, group robustness, and threshold selection to localize where missed positives enter the pipeline.
Interpretable Metrics: It reports subgroup-conditioned FNR, missed positives per 100 true positives, and bootstrap contrasts to make small-cell evidence interpretable.
Empirical Evidence: It demonstrates across two cohorts that ranking metrics and aggregate group robustness do not determine who is missed after thresholding; the operating point materially changes observed tail harm.

5. Significance and Claims

The paper positions itself as a measurement instrument rather than a deployable solution or a general fairness remedy. Its primary claims are modest and methodological:

Threshold Dependency: Rare-label fairness in CXR depends jointly on the finding, the subgroup, and the operating threshold. It is not determined by label frequency or ranking metrics alone.
Limitation of Aggregate Metrics: A model can have acceptable macro-mAP while leaving rare-positive cases in specific subgroups below the threshold.
Observability Bounds: Fairness claims are strictly bounded by the available metadata (e.g., VinDr lacks race/insurance data) and the positive support in the test set.
Clinical Policy: The choice of threshold ( $\tau$ ) is a clinical policy decision involving a trade-off between missed positives and review burden, not a universal parameter.

The study concludes that threshold-aware auditing is necessary to reduce observed subgroup misses, but high residual FNRs and metadata limitations mean this approach should be treated as a fairness measurement tool to guide policy, not as a standalone deployment-ready fix.