Explicit Time-Frequency Dynamics for Skeleton-Based Gait Recognition

1. Problem Statement

Gait recognition aims to identify individuals based on their walking patterns. While appearance-based methods (using silhouettes or pixel intensities) currently achieve state-of-the-art (SOTA) results on datasets like CASIA-B, they suffer significant performance degradation under covariate shifts such as carrying bags (BG) or wearing coats (CL).

Skeleton-based methods, which rely on joint coordinates, are naturally robust to appearance changes (texture/clothing) but have historically lagged behind appearance-based models. A key limitation of existing skeleton models (e.g., GaitMixer, GaitFormer) is that they model motion dynamics implicitly. They often fail to explicitly capture the time-frequency characteristics of joint movements, which are crucial for distinguishing gait when visual appearance is obscured or altered.

2. Methodology

The authors propose a plug-and-play "Wavelet Feature Stream" that can be attached to any existing skeleton-based backbone without modifying its architecture or requiring additional supervision. The framework consists of two parallel streams:

A. Backbone Stream

Utilizes a standard state-of-the-art skeleton model (e.g., GaitMixer) to encode global spatio-temporal patterns from joint sequences.

B. Wavelet Feature Stream (The Core Contribution)

This stream explicitly models motion dynamics through the following pipeline:

1. Velocity Computation: Per-joint velocities ($v_x, v_y$) are calculated from the 2D skeleton sequence ($S \in \mathbb{R}^{2 \times T \times V}$) by computing the difference between consecutive frames.
2. Continuous Wavelet Transform (CWT): The 1D velocity signals are transformed into time-frequency maps (scalograms) using the CWT. This decomposes the non-stationary motion signals into multiple scales, separating stable periodic gait rhythms from abrupt artifacts (e.g., pose jitter or bag swinging).
   • Formula: $W_\psi v(s, \tau) = \int v(t) \psi^*(\frac{t-\tau}{s}) dt$
3. Multi-Scale CNN: A lightweight Convolutional Neural Network processes the scalograms to learn discriminative dynamic cues.
   • Stem: Refines local patterns using $5\times5$ and $3\times3$ convolutions.
   • Parallel Multi-Scale Block: Uses concurrent $3\times3$ (fine-grained changes) and $7\times7$ (coarse rhythm/periodicity) branches.
   • Pooling: Global average pooling generates a fixed-size vector per joint.
4. Fusion: The resulting wavelet descriptor ($F_{wavelet}$) is concatenated with the backbone feature ($F_{backbone}$) and projected via a fully connected layer to form the final embedding.
5. Loss Function: The model is trained using Triplet Loss with hard mining to minimize intra-class variance and maximize inter-class separation.

3. Key Contributions

• Explicit Time-Frequency Modeling: The paper introduces the first plug-and-play module that injects explicit time-frequency dynamics of joint velocities into skeleton-based gait recognition using CWT.
• Architecture Agnostic: The method requires no changes to the backbone architecture and can be attached to various existing models (GaitGraph, GaitFormer, GaitMixer).
• New State-of-the-Art (SOTA): By attaching the stream to GaitMixer, the authors establish a new SOTA for skeleton-based gait recognition on the CASIA-B dataset.
• Superiority in Challenging Conditions: The model is the first skeleton-based approach to outperform existing appearance-based SOTA models under the most challenging "Coat" (CL) condition, proving the efficacy of skeleton information when appearance cues are lost.

4. Experimental Results

Experiments were conducted on the CASIA-B dataset (124 subjects, 11 viewpoints, 3 conditions: NM, BG, CL).

• Overall Performance:
  • GaitMixer + Ours achieved a mean Rank-1 accuracy of 89.7%, surpassing the previous skeleton SOTA (GaitMixer at 88.3%) and narrowing the gap with the appearance-based SOTA (3DLocal at 92.8%).
• Performance under Covariates:
  • Coat (CL): The proposed model achieved 86.7% accuracy with GaitMixer, a +2.2% improvement. Crucially, this score (86.7%) exceeded the best appearance-based method (3DLocal: 84.5%) in the CL condition.
  • Bag (BG): Consistent gains were observed, with GaitMixer + Ours reaching 87.2% (vs. 85.6% baseline).
• Ablation Studies:
  • Wavelet Choice: The Morlet wavelet function yielded the best results (89.7% mean accuracy) due to its ability to capture both amplitude and phase of quasi-periodic gait oscillations.
  • Backbone Compatibility: The stream provided consistent improvements across GaitGraph, GaitFormer, and GaitMixer, with the most significant gains observed in the CL condition.

5. Significance

This work addresses a critical gap in skeleton-based biometrics: the underutilization of explicit motion dynamics. By leveraging time-frequency analysis (CWT), the authors demonstrate that skeleton data contains rich, discriminative dynamic cues that are robust to appearance changes.

The significance lies in:

1. Robustness: Proving that skeleton-based systems can outperform appearance-based systems in scenarios where clothing or carried objects obscure visual features.
2. Efficiency: The "plug-and-play" nature allows for immediate performance boosts in existing pipelines without retraining the entire backbone from scratch.
3. Future Direction: It opens new avenues for analyzing non-stationary signals in biometrics using wavelet transforms, suggesting potential applications in cross-sensor scenarios (e.g., IMU data) and adaptive fusion strategies.