DeepGaitLab: Accurate and Flexible Markerless Motion Tracking Powered by Synthetic Data

Problem Statement
Scalable, accurate human motion tracking is essential for modernizing the diagnosis of gait pathologies, optimizing sports performance, and advancing movement research. While traditional marker-based motion capture systems provide high accuracy, they are restricted to research laboratories due to their reliance on dedicated infrastructure, specialized personnel, and time-consuming calibration. Markerless alternatives offer a pathway to scalability but currently face a trade-off between accuracy and flexibility. Single-view methods are accessible but suffer from depth ambiguity and occlusion, limiting their clinical utility. Existing multi-view markerless systems either fail to improve accuracy with additional cameras (e.g., OpenCap) or require dense camera arrays and proprietary software that hinder deployment in resource-constrained settings (e.g., Theia3D). Furthermore, many current tools lack generalizability to individuals with mobility limitations, as they are often trained on data from healthy populations.

Methodology
The authors present DeepGaitLab, an open-source framework designed to deliver accurate 3D kinematics across flexible camera configurations (from two to ten cameras) without requiring inter-camera spatial calibration. The system operates through a three-stage workflow:

1. 2D Dense Landmark Detection: Unlike standard computer vision approaches that detect sparse joint centers, DeepGaitLab estimates a high density of surface landmarks (57 anatomical points). This is achieved using a Transformer-based network. To overcome the scarcity of labeled multi-view data, the model is trained on large-scale synthetic data derived from the BEDLAM dataset. This dataset utilizes 271 real human bodies (from the AMASS dataset) performing 9.3 hours of motion, rendered with photorealistic textures and diverse clothing in a game engine. The network employs a transfer-learning strategy, initializing the encoder with weights from a pre-trained sparse keypoint detector (ViTPose) trained on real-world images, then fine-tuning on the synthetic data to ensure generalization.
2. 3D Triangulation and Calibration: The system supports two workflows: one using synchronized videos with physical checkerboard calibration, and an automated calibration mode. The latter treats the subject as a dynamic calibration object, using the dense anatomical landmarks detected across views to estimate camera extrinsics via bundle adjustment, eliminating the need for physical calibration targets.
3. Biomechanical Simulation: The triangulated 3D virtual markers are processed using OpenSim. The system scales a generic musculoskeletal model to the subject's anthropometry and computes inverse kinematics and inverse dynamics (joint moments).

A key feature is environment-specific fine-tuning. The framework can generate synthetic training data tailored to a specific laboratory's camera geometry and background, allowing the model to be fine-tuned for a specific site without new manual annotations.

Key Contributions

• Synthetic Data-Driven Training: By leveraging large-scale synthetic data, DeepGaitLab avoids the labor-intensive annotation of real multi-view datasets while achieving high accuracy and generalizability.
• Dense Landmark Representation: The shift from sparse joint centers to dense surface landmarks provides geometric constraints that resolve segment orientations more accurately, addressing the ambiguity inherent in sparse keypoint approaches.
• Flexible and Calibration-Free Operation: The framework functions effectively with as few as two cameras and includes an automated calibration feature that removes the barrier of time-consuming manual calibration.
• Robustness to Pathology: The system maintains accuracy across diverse populations, including those with mobility limitations, where other tools often degrade.

Results
The authors evaluated DeepGaitLab on 80 individuals across four groups: healthy adults, individuals recovering from anterior cruciate ligament reconstruction (ACLR), stroke survivors, and individuals with mild cognitive impairment (MCI).

• Accuracy vs. State-of-the-Art: In healthy adults, DeepGaitLab consistently outperformed OpenCap and Theia3D. In a two-camera setup, it showed significantly lower root mean square difference (RMSD) compared to marker-based references than OpenCap (4.2° vs. 5.9°).
• Scalability with Camera Count: Unlike OpenCap, whose accuracy plateaued when increasing cameras from two to ten, DeepGaitLab demonstrated monotonic accuracy improvement as camera density increased.
• Generalization to Pathology: DeepGaitLab maintained consistent performance in ACLR patients, showing no significant difference in accuracy between healthy and impaired groups. In contrast, OpenCap and Theia3D showed significantly higher errors in the ACLR group. DeepGaitLab successfully detected clinically relevant interlimb asymmetries in hip flexion/extension and adduction/abduction that other tools missed.
• Spatiotemporal Metrics: In patients with MCI and stroke, DeepGaitLab achieved step-length asymmetry agreement comparable to Theia3D (which used eight cameras) while using only two cameras.
• Inverse Dynamics: DeepGaitLab yielded lower relative RMSD in joint moment estimation compared to OpenCap in two-camera setups, particularly during non-sagittal movements.
• Feature Validation: The automated calibration feature achieved performance comparable to manual checkerboard calibration. Environment-specific fine-tuning provided marginal but statistically significant improvements in overground walking tasks, particularly for knee and ankle kinematics.

Significance
The paper positions DeepGaitLab as a practical, scalable platform that bridges the gap between research-grade biomechanics and clinical deployment. By demonstrating that research-quality motion analysis can be achieved with consumer-grade RGB cameras and synthetic supervision, the framework democratizes access to quantitative gait analysis. The authors suggest that while marker-based systems have long been the "gold standard," the ability of vision-based methods to approach or potentially surpass these limits (once new, higher-fidelity references like biplane radiography are adopted) could redefine clinical evaluation metrics. DeepGaitLab offers a transparent, open-source alternative to proprietary systems, enabling broader adoption in clinics and laboratories without the trade-offs of accuracy, scalability, or flexibility that have limited previous tools.