📊 statistics

A New Trained Supervised Method for Calculating Patient Similarity

This paper proposes a new supervised patient similarity metric using weighted cosine similarity with relaxed adaptive group lasso weights, which significantly improves the discrimination and overall predictive accuracy of personalized models for binary health outcomes despite a minor trade-off in calibration.

Original authors: Minzee Kim, Joel A. Dubin

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minzee Kim, Joel A. Dubin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of medicine, vast digital records of patient history are becoming available, offering a treasure trove of information for predicting health outcomes. Traditionally, doctors and data scientists have built a single, universal model to predict how a specific illness might progress for anyone. This approach treats every patient as part of a massive, mixed group, assuming that the average experience of the entire population applies to the individual. However, a growing school of thought in personalized medicine suggests that this "one-size-fits-all" strategy misses the mark. Instead, it proposes that the best prediction for a specific person comes from studying only those other people who are most like them. The challenge, then, is not just having the data, but figuring out exactly how to measure "likeness" between two people when they differ in dozens of ways, from their age and gender to their complex, fluctuating vital signs.

A team of researchers at the University of Waterloo tackled this problem by developing a new way to calculate how similar two patients are. Their work focuses on a method called personalized predictive modeling, where a unique prediction model is built for each individual using a small, custom group of similar patients from the training data. The core of their innovation is a new mathematical tool for measuring similarity. While previous methods often treated every piece of patient information as equally important, or relied on experts to decide which factors mattered most, this new approach lets the data itself decide. The researchers created a system that assigns a specific importance score to every piece of information a patient has, such as a blood pressure reading or a diagnosis, based on how strongly that factor actually predicts the outcome in question. They used a sophisticated statistical technique to filter out the noise and highlight the signals, effectively teaching the computer which details to weigh heavily and which to ignore when comparing two people.

The researchers tested this new method by creating thousands of simulated patient records with known outcomes, ranging from low-risk scenarios to high-risk ones, and then applied their algorithm to see if it could find the right matches. They compared their new weighted method against older, standard ways of measuring similarity, such as simple distance calculations that treat all data points the same. The results showed that their new approach was significantly better at distinguishing between patients who would have different outcomes, a quality known as discrimination. In many of their tests, the new method correctly identified the most similar patients, allowing the model to make sharper, more accurate predictions. Interestingly, while the new method sometimes made the predictions slightly less perfectly calibrated to the exact probabilities, the massive improvement in its ability to tell patients apart meant that the overall accuracy of the prediction was higher. In one specific simulation with moderate signals, the best results came from training the model on just the top twenty percent of the most similar patients, rather than using the entire dataset.

To see if this held up in the real world, the team applied their method to a massive, real-world database of intensive care unit records from over 200,000 patient stays across the United States. They focused on predicting whether a patient would survive their hospital stay, using a mix of demographic data and complex, continuous vital sign measurements like heart rate and oxygen levels over the first twenty-four hours. Just as in the simulations, the new weighted method outperformed the standard approach. When the researchers used their new tool to select the most similar twenty percent of patients to train the model for a specific individual, the resulting predictions were more accurate than those made using the standard method or the entire dataset. The study suggests that by letting the data determine which features matter most, rather than relying on fixed rules or expert guesses, medical models can become far more precise. The researchers note that while their method shows great promise, it is most effective when the underlying signals in the data are strong, and they plan to explore how these techniques might work with even more complex machine learning models in the future.

Technical Summary: A New Trained Supervised Method for Calculating Patient Similarity

Problem Statement
Personalized predictive modelling (PPM) aims to improve prediction accuracy by fitting unique models to individuals using subsets of training data that are most similar to the target patient. While previous studies demonstrate that PPM outperforms global models trained on heterogeneous data, a critical bottleneck remains: accurately quantifying patient similarity. Existing methods often rely on unsupervised distance metrics (e.g., Euclidean, Mahalanobis) or weighted cosine similarity where weights are assigned based on feature frequency, hierarchical structure, or expert opinion. These approaches fail to leverage the relationship between predictors and the specific outcome of interest, potentially leading to suboptimal similarity measures and reduced predictive performance.

Methodology
The authors propose a new supervised approach to calculate patient similarity, specifically for binary response data, by developing a Weighted Cosine Similarity Metric (Weighted CSM). The core innovation lies in deriving predictor-specific weights ( $w_p$ ) using a relaxed adaptive group lasso regularization technique.

The methodology proceeds as follows:

Augmented Design Matrix: The original predictor matrix is augmented to include non-linear transformations (e.g., squared values, exponentials) to account for potential curvilinear relationships without prior knowledge of the true outcome model.
Grouping: Predictors and their associated non-linear transformations are grouped together.
Weight Estimation: A relaxed adaptive group lasso is fitted to the training data to select relevant predictor groups and estimate coefficients. This process yields two specific weighting schemes:
- Weight01 CSM: Assigns a weight of 1 to selected predictor groups and 0 to unselected ones.
- Weightbeta CSM: Assigns weights proportional to the magnitude of the estimated group coefficients ( $\|\hat{\beta}_{Gp}\|_2$ ) for selected groups, and 0 otherwise.
Similarity Calculation: These weights are applied to the standard cosine similarity formula to compute the similarity between a target patient and the training population.
PPM Implementation: For each test patient, the training data is sorted by similarity. A generalized linear model is fitted on the top $M_p$ proportion of the most similar patients to predict the target's outcome. The optimal subpopulation proportion ( $M_p$ ) is tuned via cross-validation to minimize the Brier Score.

Key Contributions

Supervised Weighting: Unlike existing weighted similarity methods that rely on unsupervised feature properties or expert-defined clusters, this method learns weights directly from the data's relationship with the outcome using regularization.
Handling Non-linearity: The approach incorporates non-linear transformations of predictors into the similarity calculation without requiring prior specification of the functional form.
Feature Selection Integration: The method effectively assigns zero weights to noise variables, ensuring that similarity is driven only by predictors correlated with the outcome.
Two Distinct Schemes: The paper introduces and compares two specific implementations: a binary selection scheme (Weight01) and a magnitude-based scheme (Weightbeta).

Results
The authors evaluated the method through extensive simulation studies (six cases varying in prevalence and signal strength) and an analysis of the eICU Collaborative Research Database (ICU mortality prediction).

Discrimination: Both proposed methods (Weight01 and Weightbeta) consistently demonstrated substantial improvements in discrimination (measured by AUROC/AUPRC) compared to standard CSM, Euclidean distance, Mahalanobis distance, and Wald-statistic-based weighting. The Weightbeta CSM consistently achieved the highest discrimination, particularly when training on smaller, highly targeted subpopulations (e.g., top 10–20% of similar patients).
Calibration: The results regarding calibration (measured by Integrated Calibration Index, ICI) were mixed. In simulation studies with low prevalence, the proposed methods sometimes showed slightly worse calibration than standard CSM, though the difference was often negligible. In the eICU data analysis, Weightbeta CSM showed improved calibration over standard CSM across most subpopulation proportions.
Overall Performance (Brier Score): Despite occasional minor trade-offs in calibration, the Brier Score (which balances discrimination and calibration) improved significantly with the proposed methods. The gains in discrimination provided by Weightbeta CSM outweighed the losses in calibration, resulting in superior overall predictive accuracy.
Signal Strength: The relative improvement of the proposed methods over standard CSM increased as the signal strength (association between predictors and outcome) became stronger.

Significance and Claims
The paper claims that the proposed supervised weighted cosine similarity metric more effectively identifies similar participants than existing unsupervised or expert-driven methods. By leveraging the relaxed adaptive group lasso, the method automatically handles feature selection and non-linearity, leading to models with better discrimination and, consequently, better overall predictive performance (lower Brier Score).

The authors note that while the method improves discrimination, calibration can be sensitive to dataset characteristics (e.g., prevalence, number of predictors), and further investigation is needed to understand these nuances. They also acknowledge computational limitations when scaling to complex machine learning models (like random forests) due to the need to refit models for every test patient and cross-validation fold, though they demonstrated feasibility with generalized linear models. The work suggests that future research should explore calibration adjustments and more flexible non-linear modeling approaches.

More like this