Identical results, hidden data problems: running CMS's digital quality measures on real hospital data
This study demonstrates that while CMS's digital quality measures for severe hypoglycemia and hyperglycemia can be executed efficiently and consistently on real hospital data using FHIR, the resulting metrics are highly sensitive to underlying data quality issues—such as missing units and admission status classifications—rather than execution engine errors, underscoring the critical need for data readiness testing before mandatory FHIR reporting.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Hospitals are increasingly relying on computers to track how well they care for patients. Instead of doctors and nurses manually reviewing charts to count errors or complications, software now scans electronic health records to generate quality scores automatically. These scores determine how well a hospital performs and can even affect how much money it receives from government programs. To make this process faster and more consistent, health officials are shifting toward a new system where the data is organized in a standard digital format, allowing different computer systems to speak the same language. The goal is to have these digital measures run automatically, providing a clear, real-time picture of patient safety. However, a critical question remains: if the computer reads the data exactly as it is written, will the results reflect the true reality of patient care, or will hidden flaws in how the data is recorded lead to misleading numbers?
A researcher named Sai Sharan Kasa set out to answer this question by testing two specific safety measures on a massive, real-world database of hospital records. These measures are designed to catch two dangerous conditions: severe low blood sugar and severe high blood sugar, both of which can occur in hospitalized patients. The researcher used a complete set of de-identified records from a major teaching hospital in Boston, covering hundreds of thousands of patient stays over more than a decade. The records had been converted into the new standard digital format, but they still contained the original, messy details of how the hospital staff had entered the information. The researcher built a system to run the official government software on this data, but with a crucial twist: the system was designed to process the data in small, manageable chunks for each patient stay, rather than trying to read a patient's entire year of history at once. This approach allowed the researcher to run the complex calculations on all 431,231 hospital stays in just a few hours, a task that would have taken weeks or months using older methods.
The first major discovery was that the new, chunked system produced results that were perfectly identical to the original, unmodified software when run on the same data. This was a vital finding because it proved that the new way of processing the data did not introduce any errors or changes to the logic itself. The computer was doing exactly what it was supposed to do. However, when the researcher looked deeper at what the computer was actually finding, they discovered that the results were entirely dependent on the quality of the data feeding into it. The software was a faithful mirror, but the reflection it showed was distorted by how the data had been recorded in the first place.
The most significant problem found was related to how blood sugar levels were recorded. In many cases, bedside glucose meters produced a number but failed to include the unit of measurement, such as milligrams per deciliter. Because the safety measure requires a specific unit to determine if a number is dangerously low or high, the computer could not compare these numbers to the safety thresholds. As a result, nearly half of the severe low blood sugar events were completely invisible to the system. When the researcher manually assigned the correct unit to these missing entries, the number of detected severe low blood sugar events jumped significantly, revealing that the original data had hidden a large portion of these dangerous incidents. Similarly, the system missed many severe high blood sugar days because it could not link a patient's diabetes diagnosis to their hospital stay, causing the computer to ignore the context needed to count those days correctly.
The study also examined how the computer decided which medications to count. The software relies on a list of drug names to identify patients who received glucose-lowering medications. The researcher tested different ways of matching the hospital's local drug codes to the standard list. While changing these matching rules shifted which specific patients were included in the count, it did not drastically change the overall percentage of patients who experienced a bad outcome. This suggests that while the specific list of patients might vary depending on how the data is mapped, the overall rate of harm remained relatively stable in this specific dataset. However, the researcher found that if the system was told to count observation stays—short-term visits that are not full hospital admissions—as full hospital stays, the number of patients counted in the safety measures roughly doubled. This highlighted that a simple classification error in the data could dramatically alter the size of the group being measured, even if the rate of harm within that group stayed the same.
Perhaps the most surprising finding was that standard checks designed to ensure data quality failed to catch these issues. The computer systems that verify whether data follows the correct format reported that everything looked fine. The records passed all the technical checks for structure and completeness, yet they still contained the hidden errors that caused the safety measures to miss dangerous events. A record could have a missing unit or a broken link to a diagnosis and still pass every validation test, only to be ignored by the safety software when it tried to calculate a result. This means that hospitals cannot rely on standard data checks to ensure their safety scores are accurate; they must look at the specific logic of the safety measures themselves to find these gaps.
The researcher concluded that while the new digital system works technically and can process vast amounts of data quickly and cheaply, the numbers it produces are only as good as the data fed into it. The study showed that hidden data problems, such as missing units or broken links, can hide nearly half of the severe low blood sugar events and significantly alter the count of high blood sugar days. These errors are not caused by the software failing to work, but by the data not being ready for the software's strict requirements. Before hospitals are required to report these digital safety scores, they need to test their own data against the specific rules of the measures to ensure they are not reporting a false sense of security. The technology is ready, but the data must be cleaned and prepared to reveal the true picture of patient safety.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.