Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

Technical Summary: Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

Problem Statement
Traditional Chinese Medicine (TCM) rehabilitation therapies, such as acupuncture and Tuina, require precise hand movements and complex hand–object interactions. Existing automatic Action Quality Assessment (AQA) frameworks for physical therapy typically rely on single-view skeletal data. This approach presents three critical limitations for TCM: (1) it fails to capture the environmental context necessary for modeling hand–object interactions (e.g., needle manipulation on practice pads); (2) it is highly susceptible to self-occlusion, where hands or tools obscure one another, compromising evaluation continuity; and (3) there is a lack of publicly available, standardized multi-view datasets for TCM rehabilitation, hindering the development of robust benchmarks.

Methodology
The authors propose CME-AQA (Cross-view Multimodality Enhanced Action Quality Assessment), a framework designed to integrate visual and pose information from multiple viewpoints to address occlusion and contextual gaps. The system comprises two primary technical components:

1. Attention-based Visual–Pose Fusion (AVPF): This module addresses the insufficiency of pose-only representations in fine-grained procedures. It employs a pose-conditioned cross-attention mechanism where hand pose features act as fixed keys and values, while visual features serve as queries. This asymmetric update refines visual representations using geometric cues from the pose, enhancing interaction-aware fusion. The module is followed by temporal self-attention to model long-range dependencies within the sequence.
2. Multiscale View Alignment (MVA): To leverage multi-view data during training while maintaining single-view inference capability, the framework enforces hierarchical cross-view consistency. During training, synchronized first-person (egocentric) and third-person (exocentric) videos are processed through shared cross-attention modules. The strategy aligns latent features at three scales: early cross-attention features, AVPF output features, and self-attention features. This encourages view-invariant representations, allowing the model to deploy using only a single third-person view while benefiting from multi-view knowledge learned during training.

Key Contributions

• Framework Proposal: The introduction of CME-AQA, a cross-view multimodal framework specifically tailored for TCM rehabilitation, combining visual–pose fusion with multi-view learning.
• Dataset Creation: The release of two synchronized dual-view datasets, TCM-AQA61-A (Acupuncture) and TCM-AQA61-T (Tuina). Each contains recordings from 61 subjects with expert annotations for both categorical (binary) and continuous (metrological) skill indicators, captured via forehead-mounted (first-person) and external (third-person) cameras.
• Technical Innovation: The design of the AVPF module for pose-conditioned visual refinement and the MVA training strategy for enforcing cross-view consistency without requiring multi-view inputs at inference.

Experimental Results
The framework was evaluated on the proposed TCM datasets and an external Cardiopulmonary Resuscitation (CPR) dataset.

• TCM Performance: On the TCM-AQA61 datasets, CME-AQA achieved superior or comparable performance against competitive baselines (including STGCN, FineParser, PGT, and PHI).
  • Classification: The method achieved over a 10% relative improvement in weighted F1 scores on key tasks such as Needle Depth (0.60 vs. 0.52 for the second-best) and Quick Needle Insertion (0.63 vs. 0.57).
  • Metrological Assessment: The framework reduced Mean Absolute Error (MAE) in continuous measures, such as insertion time (reducing error from 3.26s to 3.12s) and manipulation frequency.
• Generalization: When tested on a CPR dataset (using only skeletal data due to privacy blurring), the framework achieved performance comparable to human experts and multi-view baselines on posture-based criteria (e.g., Hand Position, Arm Position), though it showed higher errors in metrics dependent on deformable object interaction (e.g., Depth, Rate) where visual cues were absent.
• Ablation Studies: Experiments confirmed that removing either the first-person view or pose features degraded performance, validating the necessity of multimodal and multi-view integration. Sensitivity analysis indicated the model remains relatively stable under moderate pose estimation noise.

Significance and Claims
The paper claims that CME-AQA enhances assessment accuracy for structured TCM rehabilitation training by providing a cost-effective, portable, and vision-based solution that overcomes the limitations of single-view skeletal analysis. By integrating complementary viewpoints and multimodal cues, the framework facilitates more precise and reliable skill evaluation, particularly for fine-grained hand-centric procedures. The authors position the work as a step toward standardized evaluation in TCM, noting that while the current validation is based on a structured teaching cohort (medical students), the framework demonstrates potential for broader clinical skill assessment. The authors modestly acknowledge limitations, including dependency on initial pose estimation accuracy and the need for future validation across a wider spectrum of expertise levels.