TokenSTFormer: A Tokenized Spatial-temporal Attention Model for Holistic Motion Analysis in Adolescent Idiopathic Scoliosis Screening

Technical Summary: TokenSTFormer for Adolescent Idiopathic Scoliosis Screening

Problem Statement
Adolescent Idiopathic Scoliosis (AIS) is a prevalent spinal deformity affecting approximately 5% of children globally. If left untreated, it can lead to severe health outcomes, including chronic back pain and psychosocial distress. The current gold standard for diagnosis relies on radiographic imaging to measure the coronal Cobb Angle (CA) >10°, but repeated X-ray exposure raises concerns regarding cumulative radiation risk. Traditional non-invasive screening methods, such as the Adams' Forward Bending Test and Scoliometer Measurement, are limited by subjective interpretation, reliance on professional expertise, and poor scalability for large-scale programs. Furthermore, existing computer vision approaches often rely on static information or require complex preprocessing pipelines (e.g., synthetic silhouette or skeleton maps), failing to fully incorporate kinematic features that provide bio-mechanical insights.

Methodology
To address these limitations, the authors propose ScoliDetect™, a system for AIS screening based on gait video analysis. The methodology is built upon three core components:

1. ScoliGait Dataset: The authors introduce a novel dataset comprising 1,516 gait video clips paired with corresponding spinal X-ray records and CA measurements. The data was collected using mobile phone cameras at 30Hz (1080p resolution) from 758 participants. Unlike traditional screening labels, the ground truth is derived from radiographic measurements, validated by senior medical doctors.
2. Kinematic Knowledge Map: Instead of using raw video or complex synthetic maps, the system constructs a "de-identified kinematic knowledge map" from pose estimation data (using YOLOv8). This map represents holistic motion features through 238 normalized features categorized into three domains:
   • Motion Space (140 features): Representing overall gait patterns.
   • Self-Skeleton Space (32 features): Capturing the subject's skeletal structure.
   • Signal Correlation (66 features): Derived from motion lagging and signal relationships using cross-correlation.
3. TokenSTFormer Architecture: The core model is a novel Transformer-based architecture designed to tokenize spatial and temporal semantics.
   • Spatial-Temporal Tokenization (STT): The kinematic knowledge map is processed through 2D convolutional layers with specific column and row kernels to separate the data into spatial and temporal tokens.
   • Feature Enhancement: These tokens undergo a Dense layer transformation, LayerNorm, and are concatenated. The architecture incorporates Multiheaded Self-Attention (MSA) and Multi-Layer Perceptron (MLP) residual blocks, along with LayerScale and Stochastic Depth modules.
   • Training Objective: The model uses a binary cross-entropy loss for the main classification task, supplemented by an auxiliary Mean Squared Error loss calculated from the CLS tokens of both temporal and spatial embeddings to enhance convergence.

Key Contributions

• ScoliGait Dataset: The creation of the first dataset containing paired gait videos and corresponding spinal X-ray labels, enabling supervised learning with radiographic ground truth.
• TokenSTFormer Model: The introduction of a model that utilizes Spatial-Temporal Tokenization to enhance feature representation and model convergence, specifically tailored for holistic motion analysis.
• Privacy-Conscious Workflow: The system processes data into a de-identified kinematic knowledge map, facilitating scalable screening on mobile devices without exposing raw video or sensitive patient data.

Experimental Results
The TokenSTFormer model was evaluated against a vanilla Vision Transformer encoder and traditional screening methods using a stratified dataset (1,216 training, 150 validation, 150 testing samples).

• Performance: TokenSTFormer achieved a state-of-the-art accuracy of 0.787, outperforming the vanilla Vision Transformer encoder (0.740) and traditional methods (average accuracy ~0.447).
• Metrics: The model demonstrated a sensitivity of 0.845 and a specificity of 0.660. It also achieved a Positive Predictive Value (PV+) of 0.845 and a Negative Predictive Value (PV−) of 0.660.
• Ablation Studies: Experiments confirmed that the Spatial-Temporal Tokenization module, specifically the inclusion of LayerNorm and independent positional encoding, is critical for model accuracy and robustness. The study found that 5 attention blocks provided the optimal balance between complexity and performance.

Significance and Claims
The paper claims that leveraging holistic motion features derived from gait video, combined with attention-based models like TokenSTFormer, offers a viable path toward scalable and cost-effective AIS screening. By moving away from subjective physical exams and radiation-heavy X-rays, this approach aims to enable early detection and timely intervention. The authors position this work as a promising step toward the deployment of accessible, accurate, and efficient diagnostic tools for scoliosis screening and monitoring, particularly in settings where professional expertise and specialized equipment are limited.