💻 computer science

Teaching Foundation Models to Read mmWave: Pose-Guided Kinematic Representation for Human Behavior Understanding

The paper introduces mmMind, a radar-language model that leverages pose-guided pretraining to align mmWave radar data with large language models for privacy-friendly human behavior understanding, validated by a new real-world benchmark and superior performance in captioning and question-answering tasks.

Original authors: Duo Zhang, Zhehui Yin, Zhiyun Yao, Haotong Qin, Xusheng Zhang, Hongliu Yang, Jianyu Sun, Junzhe Wang, Zizhou Fan, Michele Magno, Daqing Zhang

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Duo Zhang, Zhehui Yin, Zhiyun Yao, Haotong Qin, Xusheng Zhang, Hongliu Yang, Jianyu Sun, Junzhe Wang, Zizhou Fan, Michele Magno, Daqing Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot assistant how to understand what people are doing in a room, but with a catch: you can't let the robot use its eyes. Cameras are great, but they feel like a violation of privacy in places like bedrooms or bathrooms, and wearing special sensors is annoying. So, scientists turn to a different kind of "sight" called millimeter-wave (mmWave) radar. Think of this radar like a bat using echolocation; it sends out invisible radio waves that bounce off people and return, creating a cloud of dots that shows where a body is and how fast it's moving. It's private, contactless, and doesn't capture any recognizable images.

However, there's a big problem: these radar dots are messy, sparse, and hard for a robot's brain (a Large Language Model, or LLM) to understand. An LLM is like a brilliant student who can read books and write stories, but if you hand it a handful of scattered, noisy dots, it has no idea what they mean. It's like trying to explain a dance routine to someone by just giving them a list of random coordinates without any rhythm or flow. The big question is: How do we turn these confusing, invisible radar dots into a clear story that a language-savvy robot can read and understand, so it can tell us exactly what a person is doing without ever seeing them?

This is where a new project called mmMind comes in. The researchers at Peking University and ETH Zurich have built a clever system that teaches a radar-sensing robot to "read" human movement by using a secret training trick. Imagine you are teaching a student to describe a dance. Usually, you might just show them the messy video and ask them to guess. But with mmMind, the teachers first show the student the dance along with a perfect, synchronized skeleton drawing of the dancer's joints. The student learns to connect the messy radar dots to this clear skeleton structure. Once the student has learned the rhythm and the body's shape from these "skeleton" lessons, the teachers take the skeleton drawings away. Now, when the student sees only the messy radar dots, they can still "see" the skeleton in their mind and describe the dance perfectly.

The paper introduces mmMind, a model that uses this "pose-guided" training method. During the training phase, the system looks at radar data and the corresponding 3D pose (the skeleton) of a person moving. It learns to translate the sparse, noisy radar points into a smooth, continuous language of movement that preserves the body's structure and how it changes over time. Once this learning is done, the system discards the skeleton data. From then on, it only needs the raw radar signals to understand human behavior. The researchers then taught this radar-savvy model to talk to a large language model, allowing it to answer questions, write descriptions, and even hold conversations about what it "sees."

To prove this works in the real world, the team didn't just use computer simulations; they built mmMind-Bench, a massive new dataset. They recorded 17.9 hours of real people moving in seven different indoor environments, like living rooms and offices. They captured 23 participants doing everything from walking and sitting to jumping jacks and falling. Crucially, they didn't just label these actions with simple words like "jumping"; they created detailed, bilingual (English and Chinese) descriptions, questions, and multi-turn dialogues that describe exactly how the body moved, the direction of the motion, and the sequence of events.

The results suggest that this approach is a significant step forward. When tested on tasks like describing an action, answering questions about movement, and recognizing actions the model had never seen before, mmMind consistently outperformed previous methods. For example, in describing actions, it achieved a score of 86.8, beating the next best method by a wide margin. In recognizing new, unseen actions, it was correct 91.2% of the time. The researchers found that removing the "skeleton" training step caused the model's performance to drop dramatically, suggesting that learning the body's structure is essential for understanding the movement. They also discovered that trying to compress the radar data into simple, discrete codes (like turning the movement into a list of numbers) made the model less accurate, confirming that keeping the movement as a smooth, continuous flow is better.

While the system is impressive, the authors are careful to note its limits. It isn't perfect at counting exact numbers, like how many times someone jumped or the precise speed of a turn, especially if the radar signal is weak or the person is only partially visible. They also point out that currently, the system needs help to separate multiple people in a room; it relies on an external tool to sort the mixed-up radar dots into individual people before mmMind can analyze them. However, this work suggests that by teaching AI to understand the "kinematics" (the physics of motion) of the human body through radar, we can build privacy-friendly assistants that know what we are doing without ever needing to take a picture of us.

Technical Summary: mmMind – Pose-Guided Kinematic Representation for Human Behavior Understanding

Problem Statement

Large Language Model (LLM) agents require the ability to perceive human behavior in physical environments to provide timely, behavior-aware assistance. While Millimeter-wave (mmWave) radar offers a privacy-friendly, contactless, and low-cost sensing modality, its observations are difficult to align with language. Existing radar-language methods face two primary challenges:

Kinematic Representation: Conventional approaches often map radar sequences to closed-set labels or use discrete tokenization (e.g., VQ-VAE) that may lose fine-grained 3D body structure and motion dynamics. Generic point-cloud LLMs are ill-suited for the sparse, noisy, and temporally irregular nature of radar data.
Real-World Alignment: Aligning radar with language typically relies on synthetic data or lacks explicit supervision for human body structure. Real-world datasets with fine-grained behavior annotations and synchronized poses are scarce, limiting the ability to train models that generalize to unseen actions and complex reasoning tasks.

The core question addressed is: How can sparse radar sequences be transformed into kinematically grounded representations and aligned with an LLM for behavior understanding?

Methodology: mmMind

The authors propose mmMind, a radar-language model that learns continuous kinematic tokens from sparse mmWave observations using training-only 3D pose supervision. The system operates without 3D pose or visual data during inference.

1. Pose-Guided Kinematic Tokenization

The core innovation is a spatio-temporal radar encoder pretrained to recover synchronized 3D poses, shaping its latent space to preserve kinematic structure.

Input: A sequence of radar point clouds $X = \{X_1, \dots, X_T\}$ , where each point contains $(x, y, z, v)$ .
Architecture:
- Spatial Encoder: A Spatial Transformer aggregates variable-sized point clouds into frame-level representations, modeling relationships among simultaneous reflections.
- Temporal Encoder: A Temporal Transformer with rotary position embeddings integrates information across frames to capture motion continuity.
- Pose Head (Training Only): A lightweight head maps radar tokens to 3D joint coordinates. The encoder is optimized with a kinematic loss ( $L_{kin}$ ) comprising joint-position, global root motion, local joint articulation, and skeletal consistency terms.
Inference: The pose head is removed after pretraining. The encoder produces continuous kinematic tokens directly from radar data, preserving body configuration and motion evolution without explicit pose input.

2. Radar-Language Alignment

The continuous tokens are aligned with an LLM through a two-stage process:

Stage 1 (Alignment): A lightweight projector maps radar tokens to the LLM embedding space. The model is trained on captioning data (English/Chinese) to establish a correspondence between radar kinematics and behavioral language.
Stage 2 (Instruction Tuning): The radar encoder is frozen. The projector and LLM (via LoRA adapters) are instruction-tuned on behavior captioning, spatio-temporal question answering (single-turn and multi-turn), and dialogue. This stage incorporates kinematic rationales to support reasoning.

Key Contributions

Pose-Guided Continuous Representation: The paper redefines radar tokenization by learning kinematically grounded continuous representations using 3D pose supervision only during training. This enables radar tokens to preserve body structure and motion evolution, avoiding the information bottlenecks of early discrete quantization.
mmMind-Bench: The authors introduce the first large-scale real-world mmWave-language dataset and benchmark. It contains 17.9 hours of recordings from 23 participants across seven indoor environments, covering 27 behaviors. Crucially, it provides synchronized 3D poses, fine-grained language annotations (captions, QA, dialogues), and is split into participant-, environment-, and session-disjoint sets to ensure rigorous evaluation of generalization.
Foundation Model for Radar: mmMind is presented as the first mmWave-enabled foundation model supporting multi-turn reasoning and interaction for human behavior understanding, achieving state-of-the-art performance on captioning, QA, and unseen-action generalization.

Experimental Results

Evaluated on mmMind-Bench, mmMind consistently outperforms existing baselines (including mmExpert, RadarLLM, mmCLIP, and general-purpose VLMs like Qwen3-VL) across all tasks:

Radar Captioning: Achieved a BERTScore of 86.8 and METEOR of 46.5, significantly outperforming the strongest baselines (e.g., +11.6 BERTScore over RadarLLM).
Spatio-Temporal QA: Achieved 89.2% accuracy on single-turn QA and 84.7% on multi-turn dialogue, surpassing the Qwen3-VL-32B baseline by 23.3 points in multi-turn scenarios.
Unseen-Action Generalization: Achieved 91.2% accuracy on actions excluded from all training stages, outperforming the contrastive mmCLIP baseline by 11.5 points.

Ablation Studies confirmed the necessity of each component:

Removing pose-guided pretraining caused the largest performance drop (e.g., -21.8 points in unseen-action accuracy), proving language supervision alone is insufficient for organizing sparse radar data.
Removing auxiliary kinematic losses (velocity, bone length) reduced performance, highlighting the need for dynamic and structural constraints.
Replacing continuous tokens with discrete VQ tokens degraded performance, validating the benefit of continuous kinematic representation.

Significance and Claims

The paper claims that mmMind demonstrates the potential of mmWave sensing as a privacy-friendly perceptual interface for human-centered agents. By decoupling the need for visual imagery or wearable devices, the system enables agents to understand "what people are doing and how their activities unfold" in sensitive environments like bedrooms and bathrooms.

The authors emphasize that their approach addresses the fundamental gap between sparse radar observations and language understanding by explicitly modeling human kinematics during the representation learning phase. While the system currently relies on external tracking for multi-person scenarios and has limitations in estimating precise quantitative metrics (e.g., exact speed or repetition counts), it establishes a new paradigm for radar-language foundation models that prioritize structural and dynamic fidelity over discrete classification.