Technical Summary: AnyMo

Problem Statement

Wearable and mobile devices offer a practical means to continuously sense human motion in the wild. However, inertial measurement unit (IMU) signals are highly dependent on the sensing setup, including body location, mounting position, sensor orientation, device hardware, and sampling protocols. This "setup dependence" creates a significant barrier: models trained on one configuration often fail to transfer to others, limiting the broader utility of wearable IMUs beyond closed-set recognition tasks.

Furthermore, bridging the gap between continuous, multi-sensor inertial signals and discrete textual concepts (language) remains challenging. Existing approaches face three coupled limitations:

1. Data and Supervision Scarcity: Real IMU data is fragmented across placements and devices, often limited to coarse, closed-set activity labels rather than rich descriptions.
2. Limited Realism of Synthetic Augmentation: Existing synthetic generation pipelines often rely on specific labels or sparse sensor placements, failing to capture the physical realism of dense body-surface variations.
3. Modality Gap: Directly connecting raw IMU streams to Large Language Models (LLMs) is inefficient due to token scaling issues, and simple contrastive alignment fails to resolve the gap as sensor complexity increases.

Methodology

The authors propose AnyMo, a geometry-aware framework for setup-agnostic human motion modeling. The framework operates through three primary stages:

1. Physics-Grounded Geometry-Aware IMU Simulation

Instead of joint-centric simulation, AnyMo synthesizes IMU signals over dense body-surface placements using the Nymeria body model (23 anatomical segments).

• Surface Modeling: For each anatomical segment, the system selects candidate surface vertices on the body mesh. It computes a segment centroid and defines a local sensor frame based on the surface normal, an anatomical axis, and a tangent direction.
• Physics-Based Synthesis: The system simulates virtual sensor trajectories by transforming global body motion into local sensor frames, accounting for mounting orientation variations (in-plane rotation and tilt) and device noise priors estimated from real data.
• Outcome: This generates a diverse distribution of plausible synthetic IMU signals across the entire body surface, bridging the gap between synthetic pre-training and real-world sparse observations.

2. Geometry-Aware Setup-Agnostic Pre-Training

The framework learns robust representations by treating setup variation as structured body-surface variation rather than arbitrary noise.

• Graph Encoder: A spatio-temporal graph encoder models the full body as a graph where nodes represent anatomical segments and edges follow the kinematic tree.
• Masked Cross-View Predictive Contrastive Learning: The model is trained to reconstruct the full-body motion latent from a masked, sparse view of the graph. Specifically, it constructs two full graph views (A and B) by sampling different surface placements and orientations. It then masks nodes in one view and predicts the latent of the full view of the other. This forces the encoder to learn setup-agnostic motion representations that are stable across different sensor placements.

3. Full-Body IMU Tokenization and Motion-Language Alignment

To connect sparse wearable inputs with LLMs, AnyMo introduces a compact tokenization strategy.

• Tokenization: A product-quantized VAE tokenizer discretizes the continuous, setup-stable sequence latents from the encoder into compact IMU tokens. This avoids the inefficiency of feeding raw IMU streams into an LLM.
• Motion-Language Modeling: The LLM (Qwen2.5-0.5B) is extended with the IMU vocabulary. The tokenizer's codebook vectors are projected into the LLM embedding space to provide motion-aware embeddings.
• Multi-Task Contrastive Instruction Tuning: The model is fine-tuned using a combination of:
  • Instruction Generation: For motion captioning and multiple-choice recognition.
  • Contrastive Alignment: A symmetric IMU-text contrastive loss aligns IMU tokens with motion narrations and activity labels. Learnable soft prompts are used to optimize the textual context without manual template engineering.

Key Contributions

1. Geometry-Aware Simulation: The authors propose a physics-grounded simulation method over dense body-surface placements, generating diverse synthetic signals that bridge the synthetic-to-real gap more effectively than joint-centric methods.
2. Setup-Agnostic Representation Learning: They develop a pre-training method using masked cross-view predictive contrastive learning and full-body tokenization, enabling the model to learn motion representations that generalize across different sensor placements and orientations.
3. IMU-Language Generalist Model: AnyMo introduces a unified model supporting zero-shot activity recognition, cross-modal retrieval, and motion captioning, effectively grounding wearable sensing in language.

Experimental Results

AnyMo was evaluated on three complementary tasks across 14 completely unseen downstream datasets, including in-the-wild benchmarks (EgoExo4D, Ego4D) and standard HAR datasets (UCI-HAR, Opportunity, etc.).

• Zero-Shot Activity Recognition: AnyMo achieved an average Accuracy of 35.7%, macro-F1 of 29.5%, and Recall@2 of 57.5%. This represents an improvement of 11.7% in Accuracy, 11.6% in F1, and 22.6% in Recall@2 over the strongest baseline (UniMTS).
• Cross-Modal Retrieval: On the Nymeria held-out set (sim-to-real transfer), AnyMo improved the IMU-to-Text Mean Reciprocal Rank (MRR) from 10.0 to 44.6 and Text-to-IMU MRR from 6.7 to 46.7.
• Motion Captioning: In zero-shot captioning on EgoExo4D, AnyMo achieved a BERT-F1 score of 67.1, significantly outperforming prompting-based baselines like GPT-5.4 Mini and Gemma 4 26B.

Ablation studies confirmed that removing geometry-aware simulation or the masked cross-view contrastive objective caused severe performance degradation, validating the necessity of both dense surface modeling and the specific pre-training objective.

Significance and Claims

The paper claims that AnyMo establishes a new path toward generalist wearable motion understanding. By treating wearable setup variation as structured body-surface variation rather than arbitrary noise, and by connecting these variations to language through compact, full-body tokens, AnyMo demonstrates that it is possible to learn robust representations that transfer across diverse devices, placements, and datasets without requiring task-specific fine-tuning.

The authors position AnyMo not merely as a better classifier, but as a foundational step toward proactive, context-aware AI systems that can interpret human motion in the wild using the rich, open-ended semantics of language. The results suggest that combining geometry-aware simulation with motion-language modeling is a promising direction for overcoming the data scarcity and setup-dependence challenges that have historically limited wearable IMU applications.