Feature leakage inflates machine-learning estimates of antiretroviral non-adherence risk in children living with HIV: a tiered sensitivity analysis of the EARTH cohort

In the world of modern medicine, doctors increasingly rely on computers to help predict which patients might struggle to take their daily medications. This is especially critical for children living with HIV, where missing doses of antiretroviral therapy can have severe, life-threatening consequences. To make these predictions, researchers gather vast amounts of data from hospital visits, such as how many pills a child has left, whether a parent reported a missed dose, or how long it took to refill a prescription. They feed this information into machine learning models, which are sophisticated computer programs designed to find patterns and forecast future behavior. The hope is that these tools will identify at-risk children early, allowing clinics to intervene before a child's health deteriorates. However, a hidden trap exists in how this data is collected and used. If the information used to make the prediction is too closely tied to the definition of the problem itself, the computer may appear to be a genius when it is actually just reading the answer key. This phenomenon, known as data leakage, creates an illusion of perfect accuracy that vanishes the moment the model is tested in the real world.

A team of researchers from Uganda recently investigated this exact problem using data from the EARTH cohort, a long-term study tracking children living with HIV. They wanted to understand how much the apparent success of a machine learning model depended on a specific type of methodological issue: using the very clues that define the problem to solve the problem. In their companion study, the team had built a model to predict "suboptimal adherence," which they defined as a child missing doses, running out of medicine, or delaying a pharmacy refill. To be rigorous, they had previously stripped away the specific data points that directly created this definition before training their computer. That earlier work showed modest, realistic results. But for this new analysis, the researchers asked a different question: what happens if we leave those clues in? They wanted to see just how much the model's performance would inflate if it was allowed to peek at the answer.

To find out, the researchers reconstructed the dataset from the same group of 208 children and 590 clinic visits, but they organized the data into three distinct layers of strictness. The first layer was the "uncontrolled" version, where the computer was given every single piece of information available, including the direct reports of missed doses and the exact counts of pills remaining. These are the specific facts that the researchers used to decide whether a child was adhering poorly or not. When the computer was trained on this layer, the results were startlingly perfect. Six different types of machine learning algorithms, including complex tree-based models, achieved a score of 1.000, meaning they made zero mistakes in their predictions. The computer was not predicting the future; it was simply repeating the rules it had been given. It found that if a child had missed a dose, the model predicted non-adherence, and if they had not, it predicted adherence. It was a perfect loop, but a useless one for real-world prediction.

The researchers then moved to the second layer, the "loosely controlled" version. Here, they removed the five specific fields that directly defined the outcome: the report of a missed dose, the delay in pharmacy collection, the report of running out of medicine, the calculation of the delay, and the amount of medicine remaining. Without these direct clues, the computer's performance dropped significantly. The best model, a program called CatBoost, saw its accuracy score fall from a perfect 1.000 to 0.735. While this was still a decent result, it was a far cry from the perfection of the first layer. This drop proved that the initial success was largely an illusion created by the model reading the definition of the problem rather than learning a genuine pattern.

However, the story did not end there. The researchers noticed that even in this second layer, one variable was still dominating the model's decisions: a field indicating whether the child had interrupted their treatment since the last visit. Although this variable did not technically define the outcome, it was so closely related to the concept of adherence that it acted as a near-perfect proxy. When the researchers removed this final variable to create the third, "strictly controlled" layer, the model's performance fell again. The CatBoost model's score dropped to 0.680, and its ability to correctly identify the most critical cases fell to 0.407. This final result closely matched the modest performance seen in their earlier, more rigorous study, confirming that the true predictive signal in the data was much weaker than the initial, inflated numbers suggested.

The study highlights a crucial lesson for anyone using computers to predict medical outcomes. The researchers found that tree-based models, which are popular for their ability to handle complex data, were particularly susceptible to this kind of inflation because they are excellent at finding the exact rules that define a category. When the target outcome is built from a simple "or" combination of process variables—such as "missed dose OR running out of medicine"—the model can easily memorize these rules and achieve perfect scores that do not reflect reality. The researchers emphasize that variables like pharmacy refill records are not inherently bad; they are useful for predicting different outcomes, such as whether a child's virus will be suppressed. The danger arises only when the goal is to predict the adherence itself, and the model is allowed to use the very measurements that define adherence as its primary clues.

Ultimately, this analysis serves as a warning against taking high-performance numbers at face value. The researchers demonstrated that including outcome-defining fields can artificially boost a model's discrimination by as much as 0.41 points, a massive difference that could lead clinicians to trust a tool that is not actually ready for use. They suggest that future studies should adopt a tiered approach, systematically removing direct components and close proxies to audit for this kind of leakage. By doing so, scientists can ensure that the models they build are detecting genuine, transportable signals that will hold up in the real world, rather than just reflecting the structure of the data they were fed. The work confirms that while machine learning holds great promise for improving care in low-resource settings, its power must be tempered with a rigorous understanding of how the data is constructed.