💻 computer science

Human Pose Estimation in Trampoline Gymnastics: Improving Performance Using a New Synthetic Dataset

This paper introduces a new synthetic dataset (STP) generated from motion capture data to fine-tune a ViTPose model, significantly improving 2D and 3D human pose estimation accuracy for the extreme poses and viewpoints characteristic of trampoline gymnastics.

Original authors: Léa Drolet-Roy, Victor Nogues, Sylvain Gaudet, Eve Charbonneau, Mickaël Begon, Lama Séoud

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Léa Drolet-Roy, Victor Nogues, Sylvain Gaudet, Eve Charbonneau, Mickaël Begon, Lama Séoud

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a computer to understand human movement. You'd probably start by showing it thousands of photos of people walking, running, or waving hello. The computer learns these "normal" poses very well. But what happens when you ask that same computer to analyze a trampoline gymnast?

Suddenly, the computer is confused. The gymnast is spinning upside down, twisting their body into impossible shapes, and moving so fast they are a blur. To the computer, it looks like a mess of limbs. This is the problem this paper tackles: How do we teach AI to understand extreme, acrobatic human poses?

Here is the story of how the researchers solved it, explained in simple terms.

1. The Problem: The Computer Gets Dizzy

Think of standard pose estimation models (the AI that tracks body parts) as a student who has only ever studied textbooks about people standing still. If you show them a picture of a gymnast doing a backflip, the student panics. They might think the gymnast's head is their knee, or they might miss an arm entirely because it's hidden behind a leg.

In trampoline gymnastics, athletes jump up to 8 meters high, spin rapidly, and twist. Cameras often lose track of them, or the markers used to track their movement fall off. It's like trying to film a race car with a shaky camera while the driver is doing donuts; the footage is messy and hard to read.

2. The Solution: Building a "Training Gym" for AI

The researchers realized they couldn't just wait for enough real photos of these crazy moves (because they are rare and hard to film perfectly). Instead, they decided to build a virtual training gym for the AI.

Here is how they built it:

Step 1: The Real Skeleton (The Raw Data): They filmed real elite gymnasts using high-tech motion capture cameras (the kind with little reflective dots on their suits). But, as expected, the dots fell off or got hidden during the flips. The data was "noisy" and broken.
Step 2: The Digital Repair Shop: They created a special tool to fix this broken data. Imagine a digital puppet (a 3D human model called SMPL). The tool takes the broken, messy data from the real gymnasts and "fits" the digital puppet to it. It smooths out the glitches, fills in the missing dots, and creates a perfect, realistic 3D animation of the gymnast's move, even if the original video was messy.
Step 3: The Virtual Studio (Synthetic Data): Once they had the perfect 3D animations, they put them into a video game engine (Blender). They dressed the digital gymnasts in different clothes, put them in different lighting conditions, and filmed them from 8 different camera angles simultaneously.
- The Analogy: Think of this like a video game level designer. They didn't just film one gymnast; they created thousands of "fake" gymnasts doing the same moves, but with different skin tones, clothes, and backgrounds. This gave the AI a massive library of practice examples.

3. The Training: From "Book Smarts" to "Street Smarts"

They took a powerful AI model (called ViTPose) that was already good at recognizing normal people. Then, they gave it a crash course using their new "Virtual Gym" dataset.

Before: The AI was like a librarian who knows every book in the library but has never seen a real fight.
After: The AI practiced on thousands of synthetic trampoline flips. It learned what an upside-down elbow looks like, how a twisted spine appears from below, and how to spot a limb even when it's spinning fast.

4. The Results: The AI Finally "Gets It"

When they tested the newly trained AI on real trampoline footage (not the fake stuff), the results were amazing:

2D Accuracy (The Flat Picture): The AI got much better at finding joints in a single photo. It stopped confusing left and right hands and could spot limbs even when the gymnast was upside down.
3D Accuracy (The 3D Reconstruction): This is the big win. To get a 3D pose, you usually need to combine views from multiple cameras. If the AI misses a joint in one camera, the 3D reconstruction falls apart. Because the new AI was so good at spotting joints in every angle, the final 3D model of the gymnast was much more accurate.
- The Metaphor: Imagine trying to build a 3D sculpture of a spinning dancer using only 2D shadows. If your shadow-catcher misses a hand in one angle, the sculpture has a missing arm. The new AI catches the shadow of the hand every time, so the final sculpture is complete and perfect.

Why This Matters

This isn't just about gymnastics. It proves that when real-world data is too dangerous, too rare, or too messy to collect, we can simulate it to teach AI.

By creating a "synthetic dataset" (fake but realistic data) based on real motion capture, they bridged the gap between "everyday movements" and "extreme acrobatics." They showed that with the right training data, AI can learn to understand the most chaotic, fast, and complex human movements we can imagine.

In short: They taught a computer to understand a gymnast's flip by first letting it practice on a million virtual flips in a video game, and then it became a master at analyzing the real thing.

1. Problem Statement

Human Pose Estimation (HPE) models, particularly state-of-the-art (SOTA) architectures like ViTPose, struggle significantly in trampoline gymnastics. This domain presents unique challenges that standard datasets (e.g., COCO, AMASS) fail to capture:

Extreme Poses: Athletes perform somersaults and twists in tucked, piked, and straight positions, resulting in atypical joint angles.
Self-Occlusion and Rapid Motion: High-speed rotations cause motion blur and frequent self-occlusion of body parts.
Uncommon Viewpoints: Cameras capture athletes from top, bottom, and side angles simultaneously, unlike the upright, frontal views dominant in training data.
Data Scarcity: Existing marker-based motion capture (MoCap) systems fail in this environment due to marker detachment and occlusion during high-acceleration jumps, making it difficult to generate high-quality ground truth data.

2. Methodology

The authors propose an end-to-end pipeline to generate a synthetic dataset and fine-tune a pose estimation model to bridge the performance gap.

A. Robust SMPL Fitting to Noisy MoCap Data

To overcome marker detachment in trampoline MoCap, the authors developed a custom fitting pipeline to map noisy marker sequences to the SMPL parametric human body model:

Pre-processing: Automatic filtering removes unreliable trajectories (e.g., non-moving markers, high residual errors, or those deviating from the dominant jump pattern).
Correspondence Estimation: A rough marker-to-vertex fitting is performed on sampled poses. The system accumulates frame projections to assign markers to the most supported vertices.
Optimization: A multi-stage optimization process (using LBFGS and Adam) applies temporal smoothing and Gaussian Mixture Model (GMM) regularization. This ensures the resulting motion is smooth and physically plausible, effectively ignoring detached markers while preserving the original motion dynamics.
Output: The TramPoseFit dataset, containing 10 sequences of elite athletes with reconstructed SMPL motions.

B. Synthetic Dataset Generation (SynTramPose / STP)

Using the reconstructed SMPL motions, the authors generated the SynTramPose (STP) dataset:

Rendering: Images are rendered in Blender using an 8-camera setup mimicking real-world acquisition.
Diversity: The pipeline introduces variability in HDRI backgrounds (handling low contrast and vibrant colors), skin textures, and clothing (generated by expanding SMPL submeshes).
Annotations: Automatic 2D joint annotations are generated in COCO format, including visibility flags ( $v=0, 1, 2$ ) based on occlusion and image boundaries.
Scale: The dataset contains 2,520 synthetic images.

C. Model Training and Evaluation

Base Model: ViTPose (Small and Base variants) pre-trained on COCO.
Fine-tuning Strategy: The models are fine-tuned on a combination of the synthetic STP dataset and the Leeds Sports Pose (LSP) dataset (real sports images with extreme poses) to mitigate the domain gap.
Evaluation:
- 2D: Tested on a private Multiview Real TramPose (MRT) dataset (292 images, 8 cameras, manually annotated).
- 3D: 3D poses are reconstructed via geometric triangulation (using Pose2Sim) from the estimated 2D keypoints across multiple views.

3. Key Contributions

Robust SMPL Fitting Method: A novel, GPU-accelerated pipeline to fit SMPL models to noisy, incomplete MoCap data specifically tailored for high-acceleration trampoline routines.
TramPoseFit & STP Datasets: A collection of 10 real-world trampoline motion sequences (converted to SMPL) and a new synthetic multi-view dataset (STP) featuring extreme acrobatic poses under-represented in existing collections like AMASS.
Open-Source Toolchain: An end-to-end tool for generating synthetic HPE data from noisy MoCap inputs.
SOTA Performance: A fine-tuned ViTPose model that achieves state-of-the-art results on challenging trampoline data, effectively bridging the gap between standard and extreme pose estimation.

4. Experimental Results

The fine-tuned model (ViTPose + LSP + STP) was evaluated against the baseline (COCO pre-trained ViTPose) on the MRT dataset.

2D Performance:
- The fine-tuned model achieved an Average Precision (AP) of 73.1 (Small) and 75.6 (Base) on the challenging MRT test set.
- This performance rivals ViTPose's results on the standard MS COCO validation set, despite the MRT data being significantly more difficult.
- It outperformed RePoGen (a specialized synthetic-data model) by a margin of ~1.4 AP.
3D Reconstruction Performance:
- MPJPE Reduction: The fine-tuned model reduced the Mean Per Joint Position Error (MPJPE) by 12.5 mm, representing a 19.6% improvement over the baseline.
- Triangulation Robustness:
  - Completeness: With 3 cameras, the fine-tuned model successfully triangulated 51% of joints, compared to only 31% for the baseline.
  - View Independence: Using only 3 cameras with the fine-tuned model (MPJPE = 54.7mm) yielded better accuracy than using 8 cameras with the baseline model (MPJPE = 56.3mm).
- Error Propagation: The study highlights that triangulation tends to propagate 2D errors rather than mitigate them; thus, high-quality 2D detection is critical for accurate 3D reconstruction.

5. Significance

This work demonstrates that synthetic data generation, when derived from real-world motion capture and processed through robust fitting pipelines, is a viable solution for domains where data collection is physically difficult or dangerous.

Practical Application: The improved 3D accuracy enables quantitative sports analytics, markerless motion capture for coaching, and automatic scoring systems for trampoline gymnastics.
Generalizability: The approach suggests that specialized synthetic datasets can effectively address the "long tail" of extreme poses and uncommon viewpoints that standard large-scale datasets miss.
Efficiency: The fine-tuning process is rapid (<1 hour on consumer hardware), suggesting a scalable workflow for adapting HPE models to specific, high-dynamic sports.