Repeat variation and definition dependence of ECG/VCG markers: an exploratory audit of one implementation (LoopLens 10.1)

The human heart beats with a rhythm that is as unique as a fingerprint, yet it speaks a language written in electricity. When doctors place sticky pads on a patient's chest, they are capturing this electrical story on a strip of paper or a digital screen, creating a tracing known as an electrocardiogram. For over a century, this tool has been the standard for checking heart health, allowing physicians to spot irregularities by looking at the shape and timing of the waves. However, the story the heart tells is not always the same story that the machine writes down. The numbers a computer spits out depend heavily on how the software is built, which mathematical formulas it uses to translate raw signals, and exactly where the human or machine decides a wave begins and ends. If two different computer programs look at the same heartbeat, they might measure the duration of a pulse differently, or calculate the angle of the electrical force in a different direction. This creates a problem: if the numbers change based on the tool used rather than the patient's actual health, it becomes difficult to know if a change in the reading means the patient is getting better, worse, or if the computer simply changed its mind.

A researcher at Johannes Kepler University of Linz set out to investigate just how much these measurements depend on the specific software and rules used to create them. The study focused on a specific piece of software called LoopLens, version 10.1, which is designed to automatically find the start and end points of heart waves and calculate various measurements from them. The researcher did not ask if the software could diagnose a disease or predict a heart attack. Instead, the question was simpler and more fundamental: if you take the same heart recording and run it through the software twice, or if you change the mathematical map the software uses to translate the signal, how much do the numbers change? To find the answer, the researcher gathered a massive collection of heart recordings from a public database containing tens of thousands of patients. They looked at pairs of recordings taken from the same person within a few hours of each other to see how much the numbers naturally fluctuate. They also tested what happened when they swapped out the mathematical formulas used to reconstruct the heart's 3D electrical shape from the standard 12-lead recording. Finally, they compared the software's automatic markings of where a wave starts against the markings made by human experts.

The results revealed that the numbers produced by the software are far less stable than one might hope. When looking at thousands of pairs of recordings taken from the same person just hours apart, the measurements varied significantly. For the duration of the main electrical pulse, known as the QRS complex, the difference between two measurements of the same heart could be as much as 27 milliseconds. For the time it takes for the heart to recharge, known as the QT interval, the variation was even larger, reaching nearly 62 milliseconds. In the world of heart measurements, where doctors often look for changes of just a few milliseconds to make critical decisions, these swings are substantial. The study found that for many of the measurements, the software could not reliably tell the difference between a real change in the heart and the natural noise of the measurement process. Only a small fraction of the measurements showed a high level of consistency, while the majority showed moderate to low agreement. This means that if a doctor sees a number change slightly between two visits, it might be a sign of a real health issue, or it might simply be the result of the software measuring the same heart slightly differently each time.

The investigation also showed that the choice of mathematical map used to reconstruct the heart's 3D shape has a profound effect on the results. The software can use different sets of rules to translate the flat 12-lead recording into a three-dimensional view. When the researcher swapped one set of rules for another, more than 60 percent of the measurements changed enough to be considered dependent on the specific rules used. In fact, not a single measurement remained perfectly stable when the rules were changed. This suggests that the number a doctor sees is not just a property of the heart, but a property of the heart combined with the specific mathematical choices made by the software. Even small adjustments to where the software decides a wave begins or ends could shift the results. When the researcher tested the software's ability to find the start of the P-wave, which represents the electrical signal that triggers the heart to squeeze, the error was 14 milliseconds when compared to human experts using one set of rules. When the reference point was changed to a different standard, the error dropped to just 1 millisecond. This demonstrates that the definition of a boundary is just as important as the measurement itself.

The study also looked at how these measurements held up in other datasets and compared the software to other tools. In recordings taken up to a day apart, the variation was similar to or slightly better than what was seen in the first set of data, but the researcher could not definitively say whether the software was better or worse than other tools because the conditions were not identical. When the software was tested on recordings from a clinical trial where patients took drugs known to affect heart timing, the software correctly identified the direction of the change for all three drugs that were expected to lengthen the heart's recharge time. However, the size of the change measured by the software did not always match the size measured by the trial's original annotators. In some cases, the software and the human annotators disagreed on the magnitude of the effect, and in one instance, the software detected a change in the width of the pulse that the human annotators did not see. This indicates that while the software can spot that something is happening, the exact numbers it produces depend on how it was built and how it was told to look at the data.

Ultimately, the research concludes that the numbers generated by automated heart analysis tools are not absolute truths but are interpretations that depend heavily on the software's internal rules. The variation seen between measurements taken hours apart, the sensitivity to the mathematical map used, and the dependence on how boundaries are defined all mean that a single number cannot be taken in isolation. The study does not claim that these tools are useless, but it does show that their reliability is limited. For a doctor to trust a measurement, they must understand that the value is tied to the specific software version and the specific rules used to generate it. The findings suggest that before these measurements can be used to make decisions about a patient's health, the sources of variation must be understood and separated from the actual condition of the heart. The work serves as a reminder that in the digital age, the tools we use to measure the human body are just as much a part of the story as the body itself.