💻 computer science

T-MOR: Learning Motion-Aware Skeleton Representations for Human Action Recognition

This paper proposes T-MOR, a motion-aware framework that leverages a new large-scale PoseCap-1M dataset and multi-modal contrastive learning to align skeleton sequences with video and language representations, thereby achieving superior and generalizable human action recognition performance through lightweight skeleton-only inference.

Original authors: Di Yang, Mahmoud Ali, Quan Kong, Gianpiero Francesca, Francois Bremond

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Di Yang, Mahmoud Ali, Quan Kong, Gianpiero Francesca, Francois Bremond

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human actions, like "playing tennis" or "dusting furniture."

Most current AI models are like tourists with a camera. They look at a video and try to guess what's happening based on what things look like: the color of the shirt, the background scenery, or the shape of the tennis racket. If the lighting changes or the person wears a different shirt, the robot gets confused.

Other models are like stick-figure artists. They only look at the skeleton (the joints and bones) to see how the body moves. This is great because it ignores the messy background, but these models usually need to be taught every single action from scratch. If you show them a new action they've never seen before, they are lost.

T-MOR is a new framework proposed in this paper that tries to get the best of both worlds. Think of it as a dance instructor who has watched millions of videos and read millions of books, but only needs to see the dancer's skeleton to understand the move.

Here is how it works, broken down into simple concepts:

1. The "Super-Teacher" (Multi-Modal Learning)

During its training phase, T-MOR is like a student sitting in a classroom with three different teachers:

The Visual Teacher: Shows it videos of people doing actions.
The Language Teacher: Reads descriptions of those actions (e.g., "A person is dusting furniture").
The Motion Teacher: Shows it the skeleton data (the actual movement of the joints).

The goal is to teach the model that the movement of the skeleton, the look of the video, and the words describing the action all mean the same thing. It uses a technique called "contrastive learning," which is like a game of "match the pairs." It learns to say, "This skeleton movement matches this video clip and this text description."

2. The "Grouping Game" (Cluster-Guided Learning)

One of the paper's clever tricks is how it handles mistakes. Imagine you are sorting a huge pile of mixed-up photos. If you just grab random photos to compare, you might accidentally pick two photos of the same action and think they are different (a "false negative").

T-MOR uses a "grouping" strategy. Before comparing, it organizes the movements into clusters (like sorting photos into folders). It then compares movements within the same folder as "friends" (positive pairs) and movements in different folders as "strangers" (negative pairs). This helps the model understand the true structure of human motion without getting confused by random noise.

3. The "New Library" (PoseCap-1M)

To teach this model, the authors realized they needed a massive library of data that combined videos, text, and skeletons all at once. They couldn't find one big enough, so they built their own: PoseCap-1M.

Think of this as a library with over one million entries.
Every entry has a video clip, the corresponding skeleton data, and a text description.
This massive dataset allows the model to learn general rules about how humans move, rather than just memorizing specific examples.

4. The "Lightweight Performance" (Inference)

This is the most important part for real-world use. After the model has learned from all those videos and texts, it doesn't need them anymore.

Training: It uses heavy video and text data to learn.
Testing (Inference): It only needs the skeleton.

It's like a chef who learned to cook by tasting thousands of dishes and reading recipes, but now can cook a perfect meal just by looking at the ingredients list (the skeleton) without needing the original recipe book or the tasting menu. This makes it very fast and efficient, perfect for devices with limited power.

What Did They Find?

The authors tested T-MOR on several real-world challenges:

Recognizing Actions: It got better at identifying actions like "playing tennis" or "dusting" compared to previous methods, even when the background was messy or the lighting was bad.
Detecting Timing: It could tell exactly when an action started and stopped in a long video.
The "Magic" of Zero-Shot: This is the most impressive part. They asked the model to recognize actions it had never seen before (like "playing a specific game" it wasn't trained on). Because it learned the concept of movement through language and video, it could guess the action just by reading the name of the action and looking at the skeleton. It performed as well as or better than models that had been trained specifically on those actions.

Summary

In short, T-MOR is a system that learns to understand human movement by studying videos and reading text, but it only needs to see a stick-figure skeleton to do its job. By using a massive new dataset and a smart "grouping" strategy, it creates a model that is accurate, fast, and capable of understanding new actions it has never encountered before.

Technical Summary: T-MOR

Problem Statement

While Vision-Language Models (VLMs) like CLIP have achieved success in visual understanding, they primarily rely on appearance-level supervision and lack explicit modeling of human motion. This limitation hinders their effectiveness in fine-grained, human-centric action recognition tasks, where actions are defined by temporally structured and physically grounded body movements. Conversely, existing skeleton-based methods often operate in closed-set settings, making them difficult to transfer to new actions or tasks without extensive re-training. The core challenge addressed is how to leverage large-scale semantic supervision (from video and text) to learn transferable, physically grounded, and temporally coherent human motion representations that remain robust and lightweight for inference.

Methodology

The authors propose T-MOR (Transferable skeleton MOtion Representation), a motion-aware framework designed to learn transferable action representations from skeleton sequences using video and language supervision during training.

1. Architecture and Feature Extraction

Core Representation: The framework uses skeleton sequences ( $s_k$ ) as the primary input for inference. The motion encoder ( $E_M$ ) is based on UNIK, a hybrid architecture combining Transformers and TCNs to capture spatial joint relations and temporal evolution.
Multi-Modal Alignment: During pre-training, T-MOR aligns the motion embeddings ( $E_M(s_k)$ $E_{M} (s_{k})$ ) with:
- Visual Embeddings ( $E_V(v)$ ): Extracted from RGB video frames using a frozen ViCLIP video encoder.
- Textual Embeddings ( $E_T(a)$ ): Extracted from action descriptions using a frozen ViCLIP text encoder.
Inference Efficiency: The video and text encoders are frozen and discarded after pre-training. At inference time, the model utilizes only the lightweight skeleton encoder, making it suitable for resource-constrained scenarios.

2. Cluster-Guided Multi-modal Contrastive Learning

T-MOR employs a two-stage training strategy enhanced by a cluster-guided contrastive mechanism to refine positive and negative sample selection, mitigating false negatives caused by random sampling in memory banks.

Clustering: After each epoch, motion embeddings are clustered using DBSCAN. Samples within the same cluster are treated as positive pairs, while those in other clusters serve as negatives.
Self-Supervised Pre-training (Motion-Video): The model optimizes a loss function ( $L_{self} = L_{mm} + L_{mv}$ ) to align motion-motion pairs and motion-video pairs without using action annotations. This learns a transferable skeleton encoder aligned with visual motion cues.
Supervised Alignment (Motion-Text): A second stage aligns motion representations with language semantics ( $L_{mt}$ ) to enable zero-shot transfer. The total supervised objective is $L_{sup} = L_{mm} + L_{mv} + L_{mt}$ .
Loss Formulation: The contrastive loss utilizes cosine similarity with temperature scaling, encouraging motion embeddings to form semantically compact yet discriminative manifolds across modalities.

3. PoseCap-1M Dataset

To support large-scale pre-training, the authors constructed PoseCap-1M, a new dataset containing over one million synchronized triplets of video, skeleton, and text. This dataset covers diverse human activities and addresses the lack of large-scale, multi-modal skeleton resources in existing public datasets.

Key Contributions

T-MOR Framework: A novel framework that learns transferable skeleton representations by aligning motion embeddings with visual and language embeddings via cluster-guided multi-modal contrastive learning.
PoseCap-1M Dataset: A large-scale action dataset providing motion-video-text triplets to facilitate the multi-modal pre-training of skeleton-based models.
Performance and Generalization: Extensive experiments demonstrating that T-MOR improves performance on action classification and frame-wise temporal detection tasks, while highlighting the importance of embodied motion modeling for transferable action understanding.

Experimental Results

The authors evaluated T-MOR on multiple benchmarks, including Toyota Smarthome, UAV-Human, Penn Action, TSU, and Charades, under linear probing and fine-tuning settings.

Action Classification: T-MOR consistently outperformed previous state-of-the-art (SoTA) skeleton-based methods. For instance, on the Penn Action dataset, T-MOR achieved 97.8% Top-1 accuracy (fine-tuned with V+M+T pre-training), compared to 97.3% for the previous SoTA and 29.8% for the UNIK backbone trained from scratch.
Action Detection: On the TSU and Charades datasets, T-MOR showed significant gains. In fine-tuning on TSU, it achieved 38.3% CS accuracy, surpassing the previous SoTA (SD-TCN) which reached 31.2%.
Zero-Shot Transfer: T-MOR demonstrated strong generalization in zero-shot settings. On the Smarthome dataset, it achieved 21.9% accuracy using motion-only inputs, outperforming existing skeleton models trained with visual-textual features and competing with video-language foundation models.
3D Transfer: The framework also showed strong transferability to 3D action detection on the PKU-MMD dataset, achieving 94.3% mAP at IoU 0.1.

Significance and Claims

The paper claims that T-MOR successfully bridges the gap between large-scale semantic supervision and physically grounded motion modeling. By leveraging video and language data during training but relying solely on skeleton inputs during inference, the framework offers a practical solution for real-world scenarios where computational resources are limited.

The authors emphasize that motion-centric and embodied representations are crucial for transferable action understanding. The results suggest that large-scale video data can significantly benefit downstream skeleton-based tasks even without explicit action labels during pre-training. Furthermore, the proposed approach demonstrates that combining motion features with visual-language features (or using motion alone in zero-shot settings) leads to stronger generalization in challenging, fine-grained action recognition scenarios. The work positions motion-aware, embodied representations as a key direction for future human action understanding.