💻 computer science

Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs

This paper introduces IMU-to-4D, a novel framework that leverages large language models to reconstruct detailed 4D human motion and 3D scene layouts purely from wearable inertial sensors, offering a privacy-preserving and energy-efficient alternative to vision-based perception.

Original authors: Hao-Yu Hsu, Tianhang Cheng, Jing Wen, Alexander G. Schwing, Shenlong Wang

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao-Yu Hsu, Tianhang Cheng, Jing Wen, Alexander G. Schwing, Shenlong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart assistant that knows exactly what you are doing, where you are, and what objects you are touching—but it has never seen a single photo or video of you.

That is the core idea behind the paper "Seeing Without Eyes: 4D Human–Scene Understanding from Wearable IMUs."

Here is the breakdown of how they did it, using simple analogies.

1. The Problem: The "Privacy vs. Vision" Dilemma

Usually, to understand human movement, computers need cameras. They watch you to know if you are walking, sitting, or picking up a cup.

The Downside: Cameras are invasive (privacy issues), drain batteries, and can't work in the dark or if you turn your back.
The Goal: The researchers wanted to build a system that understands your life without ever taking a picture.

2. The Solution: The "Invisible Detective" (IMUs)

Instead of cameras, the system uses IMUs (Inertial Measurement Units). These are the tiny motion sensors already inside your smartwatch, earbuds, and smartphone.

The Analogy: Think of these sensors as a detective who can't see the crime scene but can feel every vibration, shake, and step. If your watch jiggles a certain way, the detective knows you picked up a heavy box. If your earbuds drop suddenly, the detective knows you sat down on a chair.

3. The Magic Ingredient: The "Language Brain" (LLMs)

This is the cleverest part. The researchers didn't build a new robot from scratch. Instead, they took a Large Language Model (LLM)—like the AI brain behind ChatGPT—and gave it a new job.

The Metaphor: Imagine teaching a language expert (who usually reads books) to read music.
- Normally, an LLM reads words like "The cat sat on the mat."
- In this paper, they taught the LLM to read motion data as if it were a language.
- They turned the raw "jiggles" from your watch into "words" (tokens) that the AI could understand.

4. What Does the System Actually Do? (The "4D" Part)

The system takes the raw data from your devices and reconstructs a 4D world (3D space + Time). It predicts three things simultaneously:

Your Body's Dance (Motion): It reconstructs your exact skeleton movement (SMPL-X) in 3D.
- Example: It knows you are flipping a pancake, even if it only saw the shake of your wrist.
The Story (Text): It writes a sentence describing what you are doing.
- Example: "The subject lifts a floor lamp with their right hand."
The Room (Scene): It guesses the layout of the room and the objects in it.
- Example: It infers there is a "lamp" and a "table" nearby because of how you moved around them.

5. Why is this better than the old way?

The Old Way (Cascaded Pipeline):
Imagine a relay race where three different people run the race:

Person A guesses your motion from the sensor.
Person B takes Person A's guess and tries to write a story.
Person C takes Person B's story and tries to draw the room.

The Flaw: If Person A makes a tiny mistake, Person B gets confused, and Person C draws a completely wrong room. The errors pile up like a snowball.

The New Way (IMU-to-4D):
Imagine a single genius who listens to the sensor, thinks about the motion, the story, and the room all at once.

Because the AI "thinks" about everything together, if the motion looks like "flipping a pancake," it automatically knows there must be a "pan" and a "stove" in the room. It corrects itself using context, just like a human would.

6. The Result

The paper shows that this "Language Brain" approach is much more accurate and stable than previous methods.

Privacy: No cameras, no photos.
Battery: Uses tiny sensors, not heavy video processing.
Understanding: It doesn't just track your arm; it understands what you are doing and where you are doing it.

Summary Analogy

Think of the old way as trying to guess a movie plot by looking at a single, blurry frame every few seconds.
IMU-to-4D is like listening to the soundtrack and the actors' footsteps in the dark. Even without seeing the screen, the AI can tell you: "The hero is running up the stairs, knocking over a vase, and shouting for help."

It proves that we don't need eyes to understand the world; we just need to listen to the rhythm of our movement.

1. Problem Statement

Current AI systems for understanding human activities and environments rely heavily on visual sensors (cameras, LiDAR). While informative, these modalities face significant challenges regarding privacy, safety, energy efficiency, and scalability.
The authors propose a solution to reconstruct 4D human-scene dynamics (3D motion + time + scene layout) purely from wearable Inertial Measurement Units (IMUs) (e.g., earbuds, smartwatches, smartphones). The core challenge is that IMU data is sparse, noisy, and lacks direct visual information, making the inference of complex 3D scenes and semantic activities an ill-posed problem.

2. Methodology: IMU-to-4D

The paper introduces IMU-to-4D, a framework that repurposes Large Language Models (LLMs) to perform non-visual spatiotemporal reasoning. Instead of treating IMU-to-motion, motion-to-text, and motion-to-scene as separate tasks, it unifies them into a single generative model.

A. Architecture Overview

Backbone: The model uses Phi-1.5 (a 1.5B parameter LLM) as the unified transformer backbone.
Input: Sparse IMU signals from up to 5 devices (ear, wrists, thighs).
Output: A sequential generation of three modalities:
1. 4D Human Motion: SMPL-X body parameters (global trajectory, orientation, joint rotations).
2. Text Description: Natural language captions of the activity.
3. 3D Scene Layout: A set of objects with class labels, 6-DoF poses (orientation + translation), and bounding boxes.

B. Key Technical Components

Novel Tokenization Scheme:
- IMU Encoding: Raw IMU signals are divided into sliding windows, normalized, and projected into continuous embeddings via an MLP.
- Motion Tokenization (Critical Innovation): Unlike prior works that tokenize motion globally or in the frequency domain, IMU-to-4D uses a 1D VQ-VAE with per-window normalization.
  - Problem Solved: Global normalization (used in previous methods like Motion-VQ) fails to capture fine-grained details due to high variance across the dataset.
  - Solution: Poses are normalized using statistics (mean/variance) computed only within the specific time window. The mean and variance are quantized separately and predicted alongside the discrete tokens. This significantly improves reconstruction quality and temporal coherence.
- Scene/Text Tokenization: Object classes are tokenized as discrete indices; continuous pose parameters are regressed via MLP heads. Text uses the pre-trained Phi-1.5 tokenizer.
Unified Sequential Prediction:
The model treats the generation of motion, text, and scene as a next-token prediction task within the LLM.
- Order: IMU Input $\rightarrow$ Motion (all timesteps) $\rightarrow$ Text (autoregressive) $\rightarrow$ Scene Objects (autoregressive: Class ID $\rightarrow$ Pose).
- Attention Mechanism:
  - Bidirectional Attention: Used for motion tokens to allow the model to reason over the entire sequence simultaneously, reducing error accumulation.
  - Causal (Autoregressive) Attention: Used for text and scene generation to enable streaming inference.
Two-Stage Training Strategy:
Due to the scarcity of datasets containing IMU, motion, text, and scene simultaneously:
- Stage 1 (Pretraining): Trained on large-scale IMU-to-Motion-Text data (MotionMillion + LINGO) using synthesized IMU signals. This establishes strong alignment between inertial signals, body motion, and semantics.
- Stage 2 (Finetuning): Finetuned on smaller Human-Scene Interaction datasets (HUMOTO, ParaHome) to learn the coupling between motion and 3D scene geometry.

3. Key Contributions

First Vision-Free 4D Framework: The first system to recover detailed human motion, textual activity descriptions, and coarse 3D scene layouts using only 2-3 everyday wearable IMUs.
LLM Repurposing for Spatiotemporal Reasoning: Demonstrates that LLMs can effectively model the coupled distribution of inertial signals, physical motion, and semantic context without visual input.
Novel Tokenization: Introduced a per-window normalization tokenization scheme for motion that significantly outperforms existing time-domain and frequency-domain tokenizers in reconstruction quality.
Joint Reasoning vs. Cascaded Pipelines: Proves that a unified model outperforms state-of-the-art (SoTA) cascaded pipelines (IMU $\to$ Motion $\to$ Text $\to$ Scene) by preventing error propagation and leveraging cross-modal consistency.

4. Experimental Results

The model was evaluated on diverse datasets (LINGO, HumanML3D, HUMOTO, ParaHome, DIP-IMU) across three tasks:

IMU-to-Motion:
- Outperformed SoTA baselines (IMUPoser, MobilePoser, BoDiffusion) in all metrics (MPJPE, PA-MPJPE, MTE).
- On the LINGO dataset (5 sensors), the bidirectional variant achieved 30.49 MPJPE vs. 58.96 for the best baseline.
- Demonstrated superior long-term temporal stability and reduced drift compared to recurrent baselines.
IMU-to-MotionText:
- Significantly outperformed the modular pipeline (MobilePoser + MotionGPT3) in text generation metrics (BLEU, ROUGE-L, CIDEr, BERT-Score).
- The unified model achieved a CIDEr score of 252.87 vs. 28.78 for the pipeline, highlighting the benefit of joint semantic reasoning.
IMU-to-Scene (3D Layout):
- Achieved a 3D-IoU of 47.78, vastly outperforming the modular baseline (0.00) and a scene-only variant (2.86).
- Showed that joint reasoning with motion and text provides the necessary context to anchor objects in 3D space, which is impossible with motion alone.
Real-World Application (Re-localization):
- On the ParaHome dataset, the model successfully inferred global position and orientation within a fixed 3D scene, correcting the drift inherent in pure IMU integration by reasoning about scene affordances (e.g., "sitting on a chair" implies proximity to a chair).

5. Significance and Limitations

Significance:

Privacy-Preserving AI: Enables rich environmental understanding without cameras, addressing critical privacy concerns in homes and public spaces.
Scalability: Leverages ubiquitous consumer devices (phones, watches, earbuds) rather than specialized motion capture suits.
Embodied AI: Paves the way for AI agents that perceive the world through motion and interaction rather than sight, suitable for low-power, always-on applications.

Limitations:

Ambiguity: Certain activities (e.g., holding a bowl vs. a cutting board) are indistinguishable from IMU signals alone, leading to "hallucinations" in scene prediction.
Drift: While improved, long-horizon motion still accumulates drift without loop-closure mechanisms.
Data Scarcity: High-fidelity human-scene interaction datasets with IMU ground truth are limited, requiring heavy reliance on pretraining and synthetic data.

In conclusion, IMU-to-4D demonstrates that wearable sensors, when combined with large foundation models and novel tokenization strategies, can provide a robust, privacy-preserving alternative to vision-based perception for understanding human activities and their surrounding environments.