Discrimination Transfers but Thresholds Do Not: External Validation of Convolutional and Transformer Pneumonia Classifiers on Adult and Paediatric Chest Radiographs

When a doctor suspects pneumonia, the first step is often a chest X-ray, a simple image that reveals the hidden landscape of the lungs. For decades, the interpretation of these images has relied on human eyes, a skill that varies even among specialists. In recent years, computer programs trained on vast libraries of medical images have promised to automate this task, acting as a second pair of eyes that never tires. These artificial intelligence systems learn to spot patterns associated with infection, aiming to provide quick, consistent diagnoses. However, a critical question has lingered in the background of this technological rush: if a computer learns to recognize pneumonia on X-rays from one hospital, can it truly understand the disease when it sees X-rays from a completely different hospital, or from a child instead of an adult? The ability of these programs to distinguish between a sick and a healthy lung is only part of the story; the real test lies in whether the specific rules they use to make a diagnosis and the confidence they express in their answers hold up when the world around them changes.

A team of researchers set out to answer this question by building and testing a wide variety of these computer models. They started with a massive collection of over 18,000 chest X-rays from adults, carefully splitting the data so that the models could learn on some images and be tested on others they had never seen before. They trained fifteen different types of neural networks, which are the digital brains behind the systems, using various methods to see which ones performed best. Some of these networks were built from scratch, while others were adapted from systems that had already learned to recognize objects in everyday photos. The researchers were particularly interested in a newer type of network called a transformer, which processes images in a way that mimics how humans might scan a scene, looking at the whole picture and its parts simultaneously. After training, they selected the single best-performing model and also created a team of five top models that voted together, hoping that a group decision would be more reliable than a single opinion.

The true test came when these frozen models, which could no longer learn or change, were shown X-rays from two completely different groups of people. The first group consisted of thousands of images from children, a population with smaller lungs and different disease patterns. The second group was a collection of adult X-rays from a different country, representing a general hospital population with a much lower rate of pneumonia than the original training data. The results revealed a fascinating split in how the technology behaves. The models proved remarkably good at simply telling the difference between a healthy lung and a sick one, regardless of whether the patient was a child or an adult. In the children's group, the models achieved an AUC of 0.925, and in the adult group, they achieved an AUC of 0.888. This ability to distinguish between classes, known as discrimination, transferred well across different populations.

However, the story changed dramatically when the researchers looked at how the models actually made their decisions. A computer model does not just say "sick" or "healthy"; it calculates a probability, a number representing how likely it thinks the patient is sick. To turn this number into a diagnosis, a human must set a threshold, a cutoff point where anything above it is called pneumonia. The researchers found that the threshold which worked perfectly for the original adult data failed miserably when applied to the new groups. The same rule that caught 84% of pneumonia cases in the original group caught only 59% of cases in children and a mere 39% in the new adult group. The models were still seeing the disease, but they were too cautious to call it out. When the researchers took a small sample of the new data to reset the cutoff point, the models immediately became accurate again, achieving an AUC of 0.83 in children and 0.79 in adults. This showed that the models' internal logic was sound, but their decision-making rules were rigid and tied to the specific environment where they were first trained.

Another layer of complexity emerged when the researchers examined the confidence the models expressed in their answers. In the original training data, the models were calibrated to give probability scores that matched the real-world frequency of the disease. But when faced with the new groups, this calibration broke down. In the group of children, where pneumonia was extremely common, the models drastically underestimated the risk, predicting a much lower chance of sickness than was actually present. In the new adult group, where the disease was rare, the models' probability estimates remained surprisingly accurate. This suggests that while the models can learn the visual features of pneumonia, their understanding of how common the disease is in a specific setting is fragile and easily disrupted by changes in the patient population.

The study also uncovered that the models were sensitive to how the X-rays were taken. In the original data, images taken with the X-ray machine in front of the patient, known as anteroposterior views, were much more likely to show pneumonia than images taken from behind, known as posteroanterior views. The models learned to use this clue, performing worse on the front-facing images because they were harder to read, not because the models were confused. When the researchers tried to force the models to ignore this clue during training, the models actually became worse at their job, suggesting that the difficulty was real and inherent to the image type, not just a method the models were exploiting. Furthermore, the researchers found that the best model for adults was not necessarily the best for children. The ranking of which computer brain was superior changed depending on the age of the patient, meaning that a model chosen for adult hospitals might not be the right choice for a pediatric ward.

Ultimately, the research demonstrates that while artificial intelligence can learn to see pneumonia with a high degree of skill that travels across different populations, the practical tools needed to use that skill in a hospital do not travel as easily. The ability to distinguish disease is robust, but the specific rules for making a diagnosis and the way the system expresses its confidence are deeply tied to the local context. For these systems to be useful in the real world, they cannot simply be downloaded and used as-is. They require a local tuning process, where the decision rules are adjusted to match the specific mix of patients and the prevalence of the disease at each new hospital. The study concludes that for these technologies to be trusted, future reports must go beyond simply stating how well a model can tell sick from healthy; they must also show how well the model performs at the specific decision points used in practice, and how its confidence holds up when the world changes.