🤖 AI

Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts

This paper introduces TRI-HAR, a rotation-invariant framework that structurally ensures robustness to independent per-location IMU orientation shifts in multi-sensor human activity recognition by employing an SO(3)-equivariant backbone and invariant feature fusion, thereby outperforming traditional rotation-augmented methods without requiring explicit calibration or augmentation.

Original authors: Seungyeol Baek, Yoonbyung Chai, Yonghyeon Lee, Sungjoon Choi, Sungho Suh

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seungyeol Baek, Yoonbyung Chai, Yonghyeon Lee, Sungjoon Choi, Sungho Suh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human activity recognition is the science of teaching computers to understand what a person is doing by listening to the tiny movements of their body. This technology relies on small sensors, often called inertial measurement units, which are strapped to the wrist, chest, or ankle to record acceleration and rotation. These devices are the silent observers behind at-home rehabilitation programs, fitness tracking, and medical monitoring, translating raw physical motion into digital labels like "walking," "running," or "sitting." However, a persistent problem has long plagued these systems: the sensors are rarely attached in the exact same way twice. When a patient puts a sensor on their wrist for a second time, or when a different person wears the same device, the orientation shifts. The sensor might be tilted slightly left or rotated forward, causing the same physical movement to generate a completely different set of numbers. For years, computers struggled to recognize that a "run" recorded with a tilted sensor was the same activity as a "run" recorded with a straight one, often failing when the device was moved even slightly.

A team of researchers at Korea University and the Massachusetts Institute of Technology has developed a new approach to solve this specific problem, creating a system that remains accurate even when sensors are reattached in random orientations. Their work, presented as the TRI-HAR framework, addresses a scenario common in self-administered health monitoring where multiple sensors are used simultaneously on different parts of the body. In these settings, each sensor might be twisted or turned independently of the others, a situation that confuses standard computer models. The researchers found that by treating the data from each sensor location as a distinct three-dimensional vector and processing them through a specialized mathematical structure, they could strip away the confusion caused by rotation. This allows the system to focus purely on the shape of the movement itself, regardless of how the device was held.

The core of their discovery lies in how they handle the data before combining it. Traditional methods often try to guess the correct orientation of a sensor or rely on training the computer with thousands of artificially rotated examples, hoping it learns to recognize the pattern. The researchers argued that these methods are fragile; if the computer sees a rotation it was not trained on, it fails. Instead, they built a system where the mathematical rules themselves prevent the rotation from changing the final answer. They designed the software to first look at the data from each body location—such as the chest or the thigh—individually. For each location, the system converts the raw sensor signals into a form that rotates along with the sensor, and then immediately collapses that information into a stable, rotation-proof summary. Only after each sensor's data has been made immune to its own specific tilt does the system combine the summaries from all the body parts to decide what the person is doing.

This approach proved remarkably effective when tested against four different public datasets containing recordings from people performing various activities. In experiments where the researchers artificially rotated the data from each sensor location by a fixed, random amount to simulate a reattachment, the new system maintained its accuracy almost perfectly. In contrast, existing models that did not use this special structure saw their performance drop drastically, often failing to recognize basic activities. Even when compared to models that were trained with extra data designed to mimic these rotations, the new system performed better without needing any of that extra training. The researchers also tested the system on a real-world dataset where sensors were physically moved and reoriented by a human instructor, a scenario that introduces both rotation and position changes. In this challenging test, the new system again outperformed the best existing methods, confirming that its ability to ignore rotation holds up even when the sensors are moved in unpredictable ways.

The researchers also examined how fast this new system runs on standard computer hardware, as speed is critical for real-time health monitoring. They found that while the system is more complex than simpler models, it processes data quickly enough to keep up with live sensor streams. For a setup with three sensors, the system takes less than a hundred milliseconds to analyze a window of data, which is far faster than the time it takes for the sensors to collect the next batch of information. This means the technology could realistically be used on a person's computer or phone to monitor activity as it happens, without needing to send data to a distant server. The work demonstrates that by changing the fundamental architecture of how the computer sees the data, rather than just feeding it more examples, it is possible to build activity recognition systems that are robust to the messy reality of how people actually wear their devices.

Technical Summary: Rotation-Invariant Multi-IMU Activity Recognition under Independent Per-Location Orientation Shifts

Problem Statement

Human Activity Recognition (HAR) using self-administered wearable inertial measurement units (IMUs) faces a critical distribution shift: the independent orientation offsets that occur when sensors are reattached across different sessions. In multi-IMU settings (e.g., chest, wrist, thigh, ankle), each sensor may acquire a distinct, arbitrary rotation relative to the body segment it monitors.

Conventional scalar HAR models lack structural robustness to these rotations, leading to significant performance degradation. Existing solutions rely on two main approaches, both with limitations:

Rotation Augmentation: Training models on synthetic rotated data. This robustness is sampling-dependent and does not guarantee invariance to arbitrary, unseen $SO(3)$ rotations.
Calibration Pipelines: Preprocessing steps to normalize orientation using reference frames, static intervals, or magnetic data. These introduce engineering overhead and assumptions (e.g., reliable magnetic fields) that often fail in uncontrolled, self-administered environments.

The core challenge is to achieve structural invariance to independent, per-location $SO(3)$ orientation shifts without requiring rotation augmentation or explicit test-time calibration.

Methodology: TRI-HAR

The authors propose Truly Rotation-Invariant HAR (TRI-HAR), a framework that embeds rotation invariance directly into the model architecture. The method processes accelerometer and gyroscope streams as triaxial vectors rather than scalar channels.

1. Location-Grouped Vector Representation

Input data is reshaped from flattened scalar channels ( $X_{raw} \in \mathbb{R}^{T \times 3S}$ ) into triaxial vector streams ( $X \in \mathbb{R}^{T \times S \times 3}$ ). Crucially, these streams are grouped by physical IMU location ( $L$ locations). Each group $X^{(l)}$ is treated as a unit subject to an independent rotation $R_l \in SO(3)$ . This formulation explicitly models the scenario where sensors at different body locations are reattached with distinct offsets.

2. Shared $SO(3)$-Equivariant Backbone

TRI-HAR employs a shared backbone, instantiated as FER-VN-DGCNN (Frequency-based Equivariant Feature Representation with Vector Neurons and Dynamic Graph Convolutional Neural Networks).

Vector Neurons (VN): Scalar hidden features are lifted to 3D vector-valued channels that transform predictably under 3D rotations.
Multi-Frequency Equivariance: The backbone lifts input vectors into a multi-frequency equivariant space. The direction of the input vector determines the angular component, while a learnable radial function modulates magnitude-dependent frequency coefficients.
Graph Processing: Within each location group, the backbone constructs a dynamic graph over time-indexed IMU states, aggregating features from kinematically similar motion states while preserving equivariance.

3. Invariant Projection and Fusion

The key innovation lies in the order of operations: Invariant Projection precedes Cross-Location Fusion.

Invariant Projection: For each location group, the equivariant features are projected into rotation-invariant scalar representations. This is achieved by computing inner products between the latent features and a learned, co-rotating frame. Mathematically, if $V_t$ is the feature and $F_t$ is the frame, the invariant feature $z_t = V_t F_t^\top$ cancels out the rotation $R_l$ .
Per-Location Aggregation: Invariant time-step features are pooled (global max/average) to produce a single invariant feature vector $h_l$ for each location.
Fixed-Order Fusion: The invariant vectors $\{h_1, \dots, h_L\}$ are concatenated in a fixed body-location order and passed to a final MLP classifier.

This design ensures that the model is invariant to independent rotations $R_l$ at each location because the rotation is canceled locally before the features from different locations are combined.

Key Contributions

TRI-HAR Framework: A multi-IMU HAR architecture that learns equivariant-to-invariant representations per physical location, providing structural robustness to independent orientation offsets without rotation augmentation or test-time calibration.
Independent Per-Location Shift Protocol: The formulation of a specific evaluation protocol where each IMU location receives a distinct, fixed $SO(3)$ rotation, accurately simulating self-administered reattachment scenarios.
Empirical Validation: Demonstration across four public benchmarks (PAMAP2, DSADS, Opportunity, RealDISP) that TRI-HAR preserves macro-F1 scores under fixed independent per-location rotations, outperforming rotation-augmented baselines that rely on sampling.

Experimental Results

The authors evaluated TRI-HAR against five supervised baselines (MC-CNN, DeepConvLSTM, MLP-HAR, TinyHAR, SA-HAR) under two conditions: original test data ( $I$ ) and fixed independent per-location $SO(3)$ offsets ( $SO(3)_{loc-fix}$ ).

Robustness to Rotation: Under $SO(3)_{loc-fix}$ , non-equivariant baselines suffered catastrophic performance drops (e.g., macro-F1 dropping from ~80% to ~30% on PAMAP2). In contrast, TRI-HAR maintained its performance on the original test data, showing negligible degradation (relative invariance error $< 10^{-10}$ ).
Comparison with Augmentation: Even when baselines were trained with matched per-location rotation augmentation (loc-sample), TRI-HAR achieved higher macro-F1 scores on all four datasets without using any rotation augmentation.
RealDISP Mutual-Displacement: On the challenging RealDISP "Mutual-displacement" fold (where sensors are physically repositioned and reoriented by an instructor), TRI-HAR achieved the highest macro-F1 (85.29%), outperforming the strongest rotation-augmented baseline (TinyHAR at 78.49%) by 6.8 points.
Ablation of Fusion Strategy: A control model, TRI-HAR-joint, which applies invariant projection to the concatenated features of all locations (assuming a shared global rotation), failed significantly under independent per-location shifts (losing ~37–50 macro-F1 points). This confirms that per-location invariant projection is necessary for the target deployment setting.
Latency: Host-side latency measurements on 3-IMU and 5-IMU streams showed that while TRI-HAR is slower than scalar baselines due to the per-location equivariant processing, its p99 window-to-label latency (67.61 ms and 85.78 ms, respectively) remains well below the window update periods (~2.5s), supporting real-time streaming inference.

Significance and Claims

The paper claims that TRI-HAR offers a structural solution to the orientation variability problem in self-administered multi-IMU HAR. By treating rotation invariance as a model property rather than a data augmentation strategy, the framework eliminates the need for:

Synthetic rotation augmentation during training.
Explicit orientation calibration or reference-frame assumptions during inference.

The authors position TRI-HAR as a "calibration-light" approach suitable for at-home rehabilitation and exercise monitoring, where sensor reattachment is frequent and user-performed calibration is impractical. The work highlights that while current implementations incur higher computational costs than scalar models, the architectural trade-off is justified by the robustness gained in realistic, uncontrolled deployment scenarios. Future work is directed toward optimizing the encoder's internal cost for edge deployment.