Technical Summary: Clinical-Prior Guided Multi-Modal Learning for Gait-Based Scoliosis Screening

Problem Statement

Adolescent Idiopathic Scoliosis (AIS) is a prevalent spinal deformity requiring early detection to mitigate progression. Current screening methods, such as the Adam's Forward Bending Test, suffer from subjectivity, inconsistency, and a reliance on specialized clinical expertise, making them difficult to scale. While computer vision offers alternatives, existing video-based approaches face three critical limitations:

1. Data Leakage: Many benchmark datasets contain repeated clips from the same individuals in both training and testing sets, leading to inflated performance metrics where models memorize subject-specific traits rather than learning generalizable pathological patterns.
2. Lack of Interpretability: Deep learning models for gait analysis often function as "black boxes," failing to provide the clinical transparency necessary for practitioner adoption.
3. Oversimplified Fusion: Current multi-modal methods often employ simplistic fusion mechanisms that fail to capture nuanced relationships between kinematic patterns and clinical priors.

Additionally, existing datasets frequently rely on physical screening labels prone to error, rather than the radiographic gold standard (Cobb Angle).

Methodology

1. The ScoliGait Dataset

To address data quality and leakage issues, the authors introduce ScoliGait, a new benchmark dataset constructed with rigorous protocols:

• Composition: 1,572 training clips derived from 550 unique adolescents and a held-out test set of 300 clips from 300 different individuals unseen during training.
• Ground Truth: Labels are based on radiographic Cobb Angle measurements (the clinical gold standard), with a threshold of 10 degrees for positive diagnosis.
• Multi-Modal Annotations: Each clip includes:
  • Video: 1080p, 30Hz, capturing complete walking cycles.
  • Kinematic Knowledge Map: A structured representation of 238 features across motion space, self-skeleton space, and signal cross-correlation.
  • Clinical Text: Automatically generated textual descriptions of gait patterns based on group-level statistical analysis and clinical priors (e.g., "asymmetric arm swing in thoracic curve types").

2. Kinematic Knowledge Map

The authors propose a clinical-prior-guided kinematic knowledge map to encode holistic gait dynamics.

• Feature Extraction: 238 features are computed, including paired joint distances (Euclidean), inter-segmental angles (trigonometric), and temporal coordination (signal cross-correlation).
• Interpretability: This map serves as a "human-readable dictionary." During inference, the model's attention scores are mapped back to this structure, allowing clinicians to identify specific kinematic features (e.g., asymmetry or joint dynamics) driving the decision, rather than relying on raw pixel attention.

3. Model Architecture: Latent Attention Pooling

The proposed framework integrates three modalities (Video, Knowledge Map, Text) using a specialized fusion mechanism:

• Encoders:
  • Text: Sentence-Transformers (all-MiniLM-L6-v2).
  • Video & Knowledge Map: Vision Transformer (ViT) backbones with distinct patch embedding methods.
• Latent Attention Pooling: Inspired by advanced language model embeddings (e.g., NV-Embed), the authors replace standard average pooling with a latent attention pooling layer. This utilizes a learnable latent dictionary to query the input sequence via cross-attention.
• Fusion: This mechanism generates highly expressive, holistic embeddings that better capture clinically salient representations than standard pooling, facilitating superior multi-modal fusion.
• Alignment: Positional embeddings are explicitly aligned between the knowledge map and video modalities to ensure temporal correspondence.

Key Results

Performance on Binary Screening

Evaluated on the subject-independent test set, the proposed multi-modal approach established a new state-of-the-art:

• Best Configuration (Knowledge Map + Video + Text): Achieved 70.0% Accuracy and a 61.9% F1-score.
• Modality Comparison: The Knowledge Map alone outperformed the raw Video model (58.0% vs. 56.3% Accuracy), demonstrating that structured kinematic representations are more discriminative for this task than raw visual features.
• Fusion Impact: Integrating the Knowledge Map with Video significantly improved performance over single modalities. Adding clinical text further enhanced the F1-score for the positive class.
• Baseline Comparison: Existing models like ScoNet-MT showed comparable overall accuracy but significantly lower sensitivity (Recall) for the positive class (e.g., 5.4% recall in one configuration), indicating a failure to correctly identify scoliosis cases.

Explainability

The system provides granular, clinically interpretable insights:

• Unlike conventional attention heatmaps on skeletons, the Knowledge Map allows for the extraction of specific kinematic features with high temporal resolution.
• The model can highlight specific time periods and clinical variables (e.g., limb synchronization) that contribute to the diagnosis, bridging algorithmic reasoning with clinical judgment.

Significance and Claims

The paper claims to provide a robust, interpretable, and clinically grounded foundation for scalable, non-invasive AIS assessment. Its primary contributions are:

1. Reliable Evaluation: By eliminating data leakage through a strictly subject-independent test set and using radiographic gold-standard labels, ScoliGait offers a realistic benchmark for evaluating generalization.
2. Interpretability: The clinical-prior-guided knowledge map moves beyond "black-box" features, offering a structured framework that translates model decisions into actionable, clinician-understandable motion analysis.
3. Advanced Fusion: The latent attention pooling mechanism demonstrates that advanced embedding techniques, typically used in language models, can significantly enhance the fusion of heterogeneous medical data (video, structured kinematics, and text) for complex movement analysis.

The authors conclude that this work addresses the core limitations of current video-based screening—data reliability, model opacity, and integration complexity—paving the way for more effective clinical deployment. Code is scheduled for release upon publication.