🤖 machine learning

VSMP-IMU: Video-Grounded Semantic Motion Programs for Sensor-Aware Synthetic IMU Generation

The paper introduces VSMP-IMU, a video-grounded framework that generates controllable synthetic IMU data using structured Semantic Motion Programs to overcome data scarcity and significantly improve wearable human activity recognition performance across low-resource, imbalanced, and subject-generalization scenarios.

Original authors: Lala Shakti Swarup Ray, Vitor Fortes Rey, Mengxi Liu, Paul Lukowicz, Bo Zhou

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lala Shakti Swarup Ray, Vitor Fortes Rey, Mengxi Liu, Paul Lukowicz, Bo Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize when you are dancing, running, or doing yoga. To do this, the robot needs to "feel" your movements, usually through a smartwatch or a sensor strapped to your body that records how you shake and spin. This field is called Human Activity Recognition. The problem is that teaching a robot requires thousands of examples of real people moving, and getting those examples is a nightmare. You have to strap sensors on real humans, ask them to repeat the same move over and over, and hope they don't get tired or change how they do it. It's expensive, slow, and often leaves the robot confused when it meets a new person who moves slightly differently.

To solve this, scientists have tried to "fake" the sensor data. Some try to watch a video of a person and guess what the sensor would feel (like looking at a dance video and guessing the rhythm). Others try to write a text description like "jump up and down" and have a computer imagine the movement. But these methods have flaws: the video method gets confused if the camera angle is weird, and the text method is too vague, often creating movements that sound right but feel wrong to a sensor. The big question is: can we create fake sensor data that is both realistic enough to teach the robot and flexible enough to cover all the different ways humans actually move?

Enter VSMP-IMU, a new system that acts like a super-smart director for a movie about movement. Instead of just guessing from a video or a text prompt, this system uses a "Semantic Motion Program" (SMP). Think of an SMP as a detailed recipe card for a movement. It doesn't just say "jumping jacks"; it breaks the move down into specific steps: "stand, jump open, close, repeat," and notes exactly which body parts are involved, how fast it should go, and how wide the arms should swing.

Here is how the magic happens: The system watches a video of a real person doing an activity and extracts this "recipe card." Then, it gets creative. It can tweak the recipe—maybe ask for the jumping jacks to be done super fast or with huge arm swings—while keeping the core identity of the move (it's still a jumping jack). Once the recipe is tweaked, a computer generates a 3D animation of a person doing exactly that. Finally, the system simulates what a sensor strapped to that animated person's body would feel, turning the 3D motion into fake sensor data.

The researchers tested this by feeding this new, fake data into a robot's brain to see if it got better at recognizing real human movements. The results were impressive. When the robot was trained only on real data, it got a score of about 68.6 out of 100. When they added the VSMP-IMU fake data, the score jumped to 78.3. That's a huge improvement, especially when there isn't much real data to begin with. In situations where the robot had very little real data to learn from (like only 1% of the usual amount), the system helped the robot improve its score by nearly 19 points compared to learning from real data alone.

The system also proved it could follow instructions. When the researchers asked the system to generate movements with a specific speed, the resulting motion matched the request 92% of the time. When they asked for a specific number of repetitions, the system got it right within one count 90.6% of the time. However, the paper notes that while the system is great at big, whole-body movements like jumping or squatting, it sometimes struggles with tiny, precise hand movements (like catching a ball) because the underlying 3D animation doesn't show every single finger twitch.

In short, VSMP-IMU suggests that by using a structured "recipe" to guide the creation of fake sensor data, we can teach robots to understand human movement much better than before. It bridges the gap between watching a video and feeling a vibration, creating a library of synthetic movements that are diverse, controllable, and surprisingly realistic. While it's not perfect for every tiny detail of human motion, it offers a powerful new way to train the next generation of wearable technology without needing to strap sensors on thousands of real people.

Technical Summary: VSMP-IMU

1. Problem Statement

Wearable Human Activity Recognition (HAR) is significantly hindered by the scarcity of labeled Inertial Measurement Unit (IMU) data, particularly in low-resource, class-imbalanced, and subject-generalization scenarios. While synthetic data generation offers a potential solution to reduce dependency on real-world collection, existing methods face a fundamental trade-off:

Video-driven methods (e.g., IMUTube) are visually grounded but suffer from sensitivity to pose-estimation errors, occlusion, and viewpoint ambiguity. They often lack control over label-preserving execution variations (e.g., tempo, amplitude).
Text-driven methods (e.g., IMUGPT 2.0) offer high controllability but are often weakly grounded in the specific spatial-temporal patterns of how activities are actually performed, potentially generating motions that match a label but lack wearable-relevant details.
Classical signal augmentation (e.g., jittering, scaling) improves robustness but does not introduce new semantic motion patterns or physically plausible variations of an activity.

The core challenge is to generate synthetic IMU data that preserves the activity-defining invariants while enabling controllable, label-preserving variations in motion execution and sensor realization, without relying on brittle frame-level pose estimates or ungrounded text prompts.

2. Methodology: VSMP-IMU

The authors propose VSMP-IMU (Video-Grounded Semantic Motion Programs), a framework that introduces a structured intermediate representation called the Semantic Motion Program (SMP) to bridge video understanding and inertial synthesis.

2.1 The Semantic Motion Program (SMP)

The SMP is a structured schema that separates activity-defining semantics from execution attributes. It is defined as a set of fields:
$S = \{y, P, D, B, T, A, O, C, U\}$
Where:

Invariants (Fixed): Activity label ( $y$ ), primitive sequence ( $P$ ), body-part involvement ( $B$ ), and object/contact information ( $O$ ). These define the activity identity.
Variants (Augmentable): Primitive durations ( $D$ ), temporal organization ( $T$ ), execution attributes ( $A$ ) such as tempo, amplitude, dominant side, and symmetry. These allow for controlled variation without changing the class label.
Constraints & Uncertainty: Augmentation constraints ( $C$ ) and field-level uncertainty ( $U$ ) guide safe perturbations.

2.2 The VSMP-IMU Pipeline

The framework operates through the following stages:

Video-Grounded SMP Extraction: A pretrained Vision-Language Model (VLM, specifically GPT-5.3 in the implementation) extracts an initial SMP from an input activity video. The VLM is prompted to output structured fields (label, primitives, body parts, attributes, uncertainty) rather than free-form text.
Semantic Refinement: The extracted SMP undergoes consistency checks (label-primitive compatibility, body-part consistency, temporal coherence, object/contact consistency). Fields failing checks are corrected or marked with high uncertainty.
Constrained Augmentation: The refined SMP is augmented to generate $K$ variants. Augmentation is uncertainty-conditioned: fields with low uncertainty (high confidence) are perturbed more aggressively, while high-uncertainty fields are changed minimally or left fixed. Crucially, invariant fields remain unchanged.
Verbalization & Motion Synthesis: Each augmented SMP is converted into a natural-language prompt compatible with HY-Motion, a frozen text-to-motion generator. HY-Motion synthesizes a 3D human motion sequence ( $M$ ) based on the prompt.
IMU Synthesis: The 3D motion is converted into virtual IMU signals using IMUSim. Virtual sensors are attached to specific body segments with nominal configurations (bias, noise, mounting).
Sensor-Aware Grounding: To bridge the gap between synthetic and real wearable data, two adaptation steps are applied:
- Simulation-in-the-loop Optimization: Bounded physical residuals (e.g., mounting orientation, bias, noise scale) are optimized to match temporal and multivariate statistics ( $D_{stat}$ ) of the real training data.
- Rank-Based Grounding: The signal is aligned to the target dataset's conventions (axis order, sign, gravity, units) and then mapped to the target domain's marginal distributions using rank-based quantile matching ( $D_{rank}$ ).
Quality Filtering: Generated samples are filtered based on motion plausibility, class faithfulness (using a lightweight HAR validator), and redundancy (embedding-based nearest neighbor check).

3. Key Contributions

Semantic Motion Programs (SMP): The introduction of a structured intermediate representation that explicitly separates activity-defining invariants from label-preserving execution attributes, enabling controlled variation.
VSMP-IMU Framework: A complete pipeline for video-grounded synthetic IMU generation that integrates SMP extraction, constrained augmentation, text-to-motion synthesis, and sensor-aware grounding.
Comprehensive Evaluation: Extensive evaluation across five public datasets (MM-Fit, UTD-MHAD, MMAct, PAMAP2, HAD-AW) under three settings: Standard LOPO (Leave-One-Person-Out), Low-Resource, and Long-Tail (class-imbalanced).
Controllability Analysis: A rigorous end-to-end study evaluating whether requested SMP attributes (tempo, amplitude, repetition count, primitive duration, symmetry) are realized in generated motion and observable in grounded IMU signals.

4. Results

4.1 Performance Improvements

Standard LOPO: VSMP-IMU achieved an average Macro-F1 of 78.33%, outperforming Real-only training (68.56%) by 9.77% and the strongest prior synthetic baseline (IMUTube, 74.30%) by 4.04%.
Low-Resource: In settings with reduced training data (e.g., 1% label budget), VSMP-IMU improved over Real-only training by 18.54% and over the strongest prior baseline by 6.76%.
Long-Tail: For underrepresented (tail) classes, VSMP-IMU improved Macro-F1 by 19.86% over Real-only training and by 4.76% over SOTA, demonstrating superior coverage for scarce activities.

4.2 Controllability Findings

The study evaluated the realization of requested controls across the pipeline ( $S \to M \to \text{Raw IMU} \to \text{Grounded IMU}$ ):

Tempo & Repetition Count: Showed strong agreement at the motion level ( $\rho=0.92$ and $\rho=0.93$ ) but experienced significant attenuation after IMU synthesis and grounding ( $\rho=0.61$ ).
Amplitude: Showed moderate motion-level control ( $\rho=0.86$ ) but was better preserved in grounded IMU ( $\rho=0.75$ ) compared to other attributes.
Primitive Duration: Exhibited increasing error across the pipeline (MAPE 9.8% in motion $\to$ 14.1% in grounded IMU).
Symmetry: Evaluated only at the motion level due to lack of bilateral sensors in the dataset; achieved 86% accuracy.
Label Retention: High retention rates (93.5%–97.1%) were observed, indicating that variations rarely altered the core activity identity.

4.3 Ablation Studies

Structured vs. Unstructured: The structured SMP representation provided a larger performance gain (+2.90 Macro-F1) than simply adding attribute instructions to generic captions.
Component Contributions: Sensor-domain adaptation (simulation-in-the-loop + rank-based grounding) was the most critical component, contributing the largest performance drop when removed.

5. Significance and Claims

The paper claims that structured video-grounded semantics provide a practical foundation for generating controllable, wearable-relevant synthetic sensor data. By separating activity identity from execution variation, VSMP-IMU addresses the limitations of both purely visual and purely textual approaches.

The authors emphasize that:

The framework is most beneficial in low-resource and long-tail settings, where semantically meaningful diversity is more valuable than simple signal perturbation.
The approach successfully extends beyond periodic exercises to wrist-dominant gestures and non-periodic actions, though gains are smaller for activities relying on fine-grained hand articulation or object contact (e.g., catching, knocking) due to the limitations of the full-body motion backend.
The method does not claim to generate perfect articulated finger motion or object-state transitions but serves as a structured supplement for body- and wrist-level wearable data.

The work suggests that combining structured intermediate representations with calibrated simulation pipelines is a promising direction for overcoming data scarcity in wearable computing.