🤖 AI

DynaPURLS: Dynamic Refinement of Part-Aware Representations for Skeleton-Based Zero-Shot Action Recognition

DynaPURLS is a novel framework for zero-shot skeleton-based action recognition that overcomes the limitations of static semantic alignment by dynamically refining multi-scale visual-semantic correspondences at inference time through hierarchical text generation, adaptive joint partitioning, and a confidence-aware memory bank, thereby achieving state-of-the-art performance on major benchmarks.

Original authors: Jingmin Zhu, Anqi Zhu, James Bailey, Jun Liu, Hossein Rahmani, Mohammed Bennamoun, Farid Boussaid, Qiuhong Ke

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingmin Zhu, Anqi Zhu, James Bailey, Jun Liu, Hossein Rahmani, Mohammed Bennamoun, Farid Boussaid, Qiuhong Ke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Static Map" vs. The "Moving Target"

Imagine you are trying to teach a robot to recognize human actions (like "dancing," "running," or "throwing a ball") just by looking at a stick-figure skeleton moving on a screen.

The robot has already learned to recognize 60 specific actions (the "Seen" classes). Now, you want it to recognize 20 new actions it has never seen before (the "Unseen" classes), like "shooting a basket" or "hitting someone with a stick."

The Old Way (The Problem):
Previous methods tried to teach the robot by giving it a static map. They would write a single, fixed description for each action (e.g., "Shooting a basket: A person throws a ball"). They would then try to match the robot's view of the skeleton to this fixed description.

Why it failed:

Too Broad: Sometimes, two very different actions look similar from a distance. For example, "hitting someone with a stick" and "shooting a basket" both involve swinging an arm. A static map sees the whole swing and gets confused. It misses the tiny details (like how the hand grips the object).
Too Rigid: The real world is messy. People move differently, cameras are at different angles, and sometimes parts of the body are hidden. A fixed description can't adapt to these changes. It's like trying to use a paper map to navigate a city where the roads are constantly changing; the map becomes useless the moment you step outside.

The Solution: DynaPURLS (The "Smart, Adapting Guide")

The authors created a new system called DynaPURLS. Think of it as upgrading from a paper map to a GPS that updates in real-time.

Here is how it works in three simple steps:

1. The "Zoom-In" Dictionary (Multi-Granularity)

Instead of just giving the robot a one-sentence description, the system uses a super-smart AI (a Large Language Model, like GPT-3) to write a detailed, multi-part story for every action.

Global View: "A person is shooting a basketball."
Local View (The Zoom-In):
- Head: "Looking up at the hoop."
- Hands: "Gripping the ball and releasing it."
- Torso: "Twisting for power."
- Legs: "Bending knees to jump."

The Analogy: Imagine trying to identify a song. The old way was just saying "It's a rock song." DynaPURLS says, "It's a rock song, but specifically the guitar solo in the middle, the drummer's fast beat, and the singer's high note." This helps the robot spot the unique details that make "shooting a basket" different from "hitting someone," even if the arm swing looks similar.

2. The "Flexible Net" (Adaptive Partitioning)

In the past, researchers would force the robot to look at specific body parts in a rigid way (e.g., "Always look at the left arm first"). But what if the person is facing away, or their arm is blocked?

DynaPURLS uses a smart, flexible net. Instead of forcing the robot to look at pre-defined body parts, it asks the AI: "Based on the story we just wrote, which parts of the skeleton are actually moving right now?"

The Analogy: Imagine a security guard watching a crowd. A rigid guard only looks at the left side of the room. A flexible guard (DynaPURLS) looks wherever the action is happening. If the person is waving their right hand, the guard focuses there. If they are kicking with their left leg, the guard shifts focus. This ensures the robot sees the most important details, even if the person is partially hidden.

3. The "Live Update" (Test-Time Adaptation)

This is the paper's biggest innovation. Usually, once a robot is trained, it stays the same. If the lighting changes or the camera angle shifts, the robot gets confused.

DynaPURLS has a "Live Update" feature. When the robot sees a new action it hasn't seen before, it doesn't just guess. It takes a quick "self-check" while it's working:

Confidence Check: It looks at the action and asks, "Am I pretty sure about this?"
The Memory Bank: If it feels confident, it saves a tiny "note" about this specific movement in a special memory bank. Crucially, this bank is balanced. It makes sure it doesn't just save notes about common actions (like "walking") and ignore rare ones.
The Adjustment: Using these notes, the robot slightly tweaks its own internal "dictionary" (the text descriptions) to better match what it is actually seeing right now.

The Analogy: Imagine a chef tasting a soup.

Old Way: The chef follows the recipe exactly. If the ingredients are slightly different this time, the soup tastes off, and the chef doesn't know why.
DynaPURLS: The chef tastes the soup, realizes it needs a pinch more salt, and adjusts the recipe while cooking. Then, they remember that adjustment for the next batch. The system learns on the fly to handle the specific "flavor" of the new action.

The Results: Why It Matters

The authors tested this on three huge datasets of human movements (NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD).

The Outcome: DynaPURLS beat all previous methods. It set new "world records" for accuracy.
The Key Win: It was especially good at recognizing the "Unseen" actions that other robots failed at. By refining its understanding in real-time, it bridged the gap between what it learned in the classroom (training) and what it saw in the real world (testing).

Summary

DynaPURLS is a new way for computers to understand human movement. Instead of using a rigid, one-size-fits-all description, it:

Breaks actions down into tiny, detailed parts (hands, legs, timing).
Uses a flexible focus to find the most important moving parts.
Crucially: It updates its own understanding while it is watching the video, using a smart memory system to correct its mistakes instantly.

This allows the computer to recognize new, tricky actions much better than ever before, without needing to be retrained from scratch.

Technical Summary: DynaPURLS

Problem Definition

Zero-Shot Skeleton-based Action Recognition (ZS-SAR) aims to recognize action classes without direct training examples by transferring knowledge from seen to unseen categories via a shared semantic space. However, existing approaches face two primary limitations:

Coarse-Grained Alignment: Prevailing methods rely on aligning global skeleton features with static, class-level semantic embeddings. This fails to capture fine-grained, local body-part dynamics (e.g., hand movements) that are often shared across actions with different global patterns, leading to a failure in bridging the domain shift between seen and unseen classes.
Static Semantics: Current fine-grained methods generate textual descriptions offline using Large Language Models (LLMs) without access to the specific visual data. These static descriptions lack instance-specific distinctions and fail to adapt to the "Semantic-Visual Gap" caused by distribution shifts during inference, resulting in brittle generalization.

Methodology: DynaPURLS

The authors propose DynaPURLS (Dynamic Refinement of Part-aware Representations), a unified framework that shifts from static matching to dynamic, adaptive alignment. The framework consists of three core components:

1. Multi-Granularity Semantic Representation Generation

Instead of using single global labels, the framework leverages a Large Language Model (GPT-3) to decompose action labels into hierarchical textual descriptions:

Spatial Decomposition: Descriptions for four specific body parts (Head, Hands, Torso, Legs).
Temporal Decomposition: Descriptions for three phases (Start, Middle, End).
Global Description: A holistic description augmenting the original label.
These descriptions are encoded using a pre-trained CLIP text encoder to form a static multi-granularity semantic anchor matrix.

2. Adaptive Partitioning and Visual-Semantic Alignment

To align visual features with the multi-granularity text, the framework employs an adaptive partitioning module based on cross-modal attention:

Visual Encoding: A pre-trained Shift-GCN extracts spatio-temporal features from skeleton sequences.
Dynamic Aggregation: Unlike static partitioning (which rigidly groups joints), this module treats text embeddings as queries and visual features as keys/values. It learns to dynamically aggregate visual information relevant to each semantic concept, allowing the model to focus on the most informative spatio-temporal nodes regardless of pre-defined boundaries. This provides robustness to occlusions and viewpoint variations.
Training Objective: A symmetric InfoNCE contrastive loss is used to maximize similarity between matched visual-text pairs across all granularities.

3. Dynamic Test-Time Feature Refinement (TTA)

The core innovation is a Test-Time Adaptation (TTA) mechanism that refines semantic representations online during inference to address domain shifts:

Lightweight Affine Transformation: Instead of updating the heavy backbone encoders, the framework introduces learnable scaling ( $S$ ) and bias ( $\Delta F$ ) parameters to transform the static text embeddings ( $F$ ) into refined embeddings ( $F'$ ) based on the incoming visual stream.
Confidence-Guided Optimization: The refinement is driven by a self-supervised signal derived from the model's own high-confidence predictions.
Class-Balanced Memory Bank: To prevent error propagation from noisy pseudo-labels and bias toward frequent classes, a memory bank stores a balanced set of high-confidence samples per class. The refinement parameters are updated via gradient descent on a balanced batch sampled from this bank.
GZSL Extension: For Generalized Zero-Shot Learning, an entropy-based gating mechanism separates seen and unseen classes to prevent the model from defaulting to seen-class predictions for unseen samples.

Key Contributions

Novel Framework: Introduction of DynaPURLS, which establishes a dynamic link between visual and textual features to overcome coarse alignment and static semantics.
Fine-Grained Adaptive Alignment: A strategy matching multi-granularity LLM-generated semantics with visual features via an adaptive partitioning module, enabling effective part-level knowledge transfer.
Test-Time Refinement Mechanism: The first work to employ test-time adaptation for skeleton-based ZSL. It uses a lightweight, confidence-guided optimization with a class-balanced memory bank to adapt semantic features at inference, mitigating domain shift with minimal overhead.
State-of-the-Art Performance: Extensive experiments demonstrating superior performance in both ZSL and GZSL settings across three benchmarks.

Experimental Results

The framework was evaluated on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD under fixed and random splits.

ZSL Performance: DynaPURLS significantly outperforms prior art. On NTU RGB+D 60 (55/5 split), it achieves 88.52% accuracy, surpassing the previous best (Neuron) by 1.62%. On NTU RGB+D 120 (110/10 split), it reaches 89.06%, outperforming SCoPLe by 14.53%.
GZSL Performance: The method achieves high harmonic mean (H) scores, indicating balanced performance on seen and unseen classes. On NTU RGB+D 120 (random splits), it achieves an H-score of 74.72%, and on PKU-MMD, 60.11%.
Efficiency: The dynamic refinement module is computationally efficient, requiring only ~8,192 parameters per class and adding minimal inference time (e.g., 1.0s for NTU60 55/5 split on an A100 GPU).
Ablation Studies: Results confirm that the combination of multi-granularity alignment and dynamic refinement yields the best results. The memory bank is critical for stabilizing adaptation, and the method shows robustness across different visual backbones (Shift-GCN, PoseC3D, etc.) and initialization strategies.

Significance and Claims

The paper claims that DynaPURLS sets a new state of the art by fundamentally addressing the "Semantic-Visual Gap" inherent in static zero-shot learning. By moving from static matching to dynamic, adaptive alignment, the framework effectively bridges the domain shift between training and testing distributions. The authors emphasize that this is the first work to apply test-time adaptation specifically for skeleton-based zero-shot action recognition. The results suggest that dynamically refining semantic anchors at inference time is a more effective strategy for handling the nuances of unseen actions than relying solely on improved training-time alignment or static semantic descriptions. The work highlights the potential of combining LLMs for semantic enrichment with online adaptation mechanisms to achieve robust generalization in complex, real-world scenarios.