From Video to Clinical Insight: Computer Vision-Based Detection of Upper Limb Compensation Strategies Following Stroke Utilizing SAM3D Body

Recovery from a stroke is often a story of adaptation. When the brain's command center for the arm is damaged, the body frequently finds a workaround to get the job done. If a person cannot reach forward with their hand because the shoulder muscles are weak, they might lean their entire torso forward or lift their shoulder high to bring their hand closer to a cup. These "compensatory movements" allow the person to complete the task, but they can become bad habits that prevent the arm from ever learning to move correctly again. For decades, therapists have relied on their own eyes to spot these shortcuts during clinic visits, watching a patient reach for an object and noting if their spine bends or their shoulder lifts. However, this method is limited to the few hours a patient spends in a hospital, making it difficult to track progress or catch subtle changes when the patient is back at home.

A team of researchers has now developed a way to see these hidden movements using nothing more than a standard video camera and a computer program. By teaching a machine to recognize the specific angles of the body, they created a system that can automatically flag when a stroke survivor is using these compensatory strategies. The goal was not to replace the therapist, but to give them a tool that works around the clock, turning a simple video recording into a detailed report on how the body is moving. This approach offers a path toward monitoring recovery in the real world, where patients live and practice, rather than just in the controlled environment of a laboratory.

The researchers focused on three common ways the body compensates during a simple task: reaching out to pick up a water bottle and bringing it to the mouth. The first is lifting the shoulder away from the side of the body. The second is bending the upper body forward. The third is leaning the upper body to the side. To teach the computer to spot these, the team recruited twenty-six people who had suffered a stroke between one and six months prior. Each participant sat at a table and performed the reaching task ten times with their stronger arm and ten times with their weaker arm. A tablet camera recorded every movement from the front.

Instead of attaching sensors to the skin or asking patients to wear special suits, the team used a sophisticated computer vision model called SAM3D Body. This software analyzes the video frame by frame to build a three-dimensional skeleton of the person's body, tracking the position of the spine, shoulders, elbows, and wrists in space. The researchers then calculated specific angles for each movement. For example, to measure shoulder lifting, the computer calculated the angle between the upper arm and the spine. To measure leaning, it measured how far the spine tilted away from a straight vertical line. They looked at the maximum angle reached, the average angle held, and how much the angle changed from the starting position.

To ensure the computer was learning the right things, three human experts—doctors and therapists—watched the videos and labeled each attempt as either showing a compensatory movement or not. They agreed on the labels for the vast majority of the trials. The researchers then trained a machine learning algorithm, a type of computer program that learns patterns from data, to predict these labels based solely on the angles calculated by the camera. They tested the system using a rigorous method where the computer was trained on data from twenty-five people and then asked to predict the movements of the twenty-sixth person, a process repeated until every participant had been tested. This ensured the system could recognize these movements in a new person it had never seen before.

The results showed that the computer could identify these movements with high reliability. For the shoulder-lifting strategy, the system was correct about 95.5% of the time. For leaning to the side, it was correct 87.5% of the time. The forward-bending movement was the hardest to detect, with an accuracy of 75.6%, but the overall success rate across all three types was strong. The researchers found that the system worked best for people with more severe impairments. In these patients, the compensatory movements were large and obvious, making them easy for the computer to spot. For those with milder impairments, the movements were subtler and closer to normal movement patterns, which made them slightly harder to distinguish. This suggests the tool is particularly useful for the patients who need the most help, as their larger movements are the ones the system catches most reliably.

The study also revealed why some movements were harder to detect than others. The forward bend of the torso relies on depth perception—how far the body moves toward or away from the camera. Since the system used a single camera, it had to estimate this depth, which is inherently more difficult than measuring side-to-side or up-and-down movements on a flat screen. This technical limitation explains why the forward-bending detection was less accurate than the shoulder or side-leaning detection. Despite this, the system successfully distinguished between the affected arm and the healthy arm, showing that the weaker arm consistently triggered larger compensatory angles.

The researchers emphasize that this technology is designed to assist, not replace, human judgment. In a practical setting, the system would act as a filter, reviewing hours of video and highlighting only the specific attempts where a compensatory movement likely occurred. A therapist could then review just those flagged moments, perhaps seeing a 3D reconstruction of the body overlaid on the video to confirm the finding. This would save therapists time and allow them to focus on the most relevant data. The system could also be used for remote monitoring, allowing patients to perform the task at home with a tablet and sending the data to their care team. This would provide a continuous picture of recovery, showing whether a patient is improving or if they are slipping back into old habits between clinic visits.

While the current system requires powerful computers to process the video, the researchers note that future improvements could make it fast enough to run on standard mobile devices. They also acknowledge that real-world conditions, such as poor lighting or cluttered backgrounds, might challenge the system, and that more data is needed to refine its accuracy for people with mild impairments. However, the core finding stands: it is possible to use a single video camera and advanced software to objectively measure how a stroke survivor moves. By turning a simple video into a precise measurement of body angles, this work opens the door to a future where rehabilitation is not just a series of clinic visits, but a continuous, data-driven journey toward recovery.