💻 computer science

SMART: SMPLest-X Mesh Adaptation and RAFT Tracking for Soccer Pose Estimation

The paper presents SMART, a soccer pose estimation system that combines a finetuned SMPLest-X model with RAFT-based camera tracking and temporal smoothing to achieve a 38.6% improvement over the baseline in the FIFA Skeletal Tracking Challenge 2026.

Original authors: Parthsarthi Rawat

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Parthsarthi Rawat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a soccer match on TV. The camera zooms in, pans across the field, and sometimes players block each other. Now, imagine you want to build a robot that can watch that same video and instantly draw a perfect 3D skeleton of every player, showing exactly where their nose, elbows, and toes are in the real world, not just on the flat screen.

That is the challenge this paper tackles. The authors created a system they call SMART (SMPLest-X Mesh Adaptation and RAFT Tracking). Think of SMART as a high-tech "digital puppet master" that turns flat video into a 3D movie.

Here is how they did it, broken down into simple steps:

1. The Brain: Learning to See Players in 3D

The team started with a powerful AI model called SMPLest-X. You can think of this model as a very smart artist who has seen millions of photos of people and knows how a human body is built. However, this artist was trained mostly on studio photos, not on messy, fast-moving soccer broadcasts where players are far away, partially hidden, or moving quickly.

To fix this, they gave the artist a "refresher course" specifically on soccer footage. They taught it to pay extra attention to the players' feet and how deep they are from the camera. They also taught it to handle the weird lighting and zooms of a TV broadcast. This step is like taking a painter who is great at portraits and training them specifically to paint action shots of athletes.

2. The Eyes: Tracking the Camera

The biggest trick in this video game is that the camera itself is moving. If the camera zooms or pans, the players look like they are moving even if they are standing still.

The old way of doing this was like trying to track a moving train by looking at a few specific pebbles on the track (sparse tracking). It often got confused.

The SMART team used a new method called RAFT. Imagine instead of looking at a few pebbles, you look at the entire grass field at once. RAFT watches how the texture of the grass moves from one frame to the next. It's like watching a crowd of people move in a stadium; even if you can't see every face, you can tell the whole crowd is shifting left or right. By tracking the grass, the system knows exactly how the camera is moving, so it can subtract that movement to find out where the players really are.

3. The Feet: Sticking to the Ground

In 3D space, it's easy for a computer to get confused about how high off the ground a player is. Is a player jumping? Or is the camera just tilted?

To solve this, SMART uses a "foot-plane anchoring" trick. It finds the lowest point of a player's shoe in the video and draws an invisible laser beam from the camera through that shoe until it hits the imaginary line of the soccer pitch. It then forces the player's skeleton to stand exactly on that spot. It's like giving every player a pair of magnetic boots that snap them firmly to the grass, preventing them from floating in the air or sinking into the ground.

4. The Smooth Operator: Cleaning Up the Jitter

When a computer guesses a player's position frame-by-frame, the result can look shaky, like a video game character glitching or teleporting.

The team added a "smoothing" step. Imagine watching a shaky hand-drawn animation. The first pass of smoothing fixes the big, impossible jumps (like a player suddenly teleporting across the field). The second pass gently smooths out the tiny shakes while making sure fast movements—like a quick kick—still look fast and natural. It's like using a steady hand to trace over a shaky sketch without losing the energy of the original drawing.

The Results: How Good Is It?

The team tested their system against the official "FIFA Baseline" (the standard they were trying to beat).

The Baseline Score: 1.053 (Lower is better).
SMART's Score: 0.593.

This is a 38.6% improvement. In plain English, their system is significantly more accurate at guessing where players are in 3D space than the previous best method.

What Are the Weaknesses?

The authors are honest about where the system struggles:

Diving: If a player is diving in the air (not touching the ground), the "magnetic boots" trick doesn't work, and the system gets confused about their height.
Hiding: If a player is completely blocked by another player, the system has to guess, and it's not perfect.
Speed: The system is heavy and requires a powerful computer to run, though the authors note they could make it faster if they didn't need such high accuracy.

In summary, SMART is a new way to turn flat soccer videos into accurate 3D models by combining a specialized AI artist, a camera tracker that watches the grass, and a set of rules that keep players stuck to the ground.

Technical Summary: SMART (SMPLest-X Mesh Adaptation and RAFT Tracking) for Soccer Pose Estimation

Problem Definition

The paper addresses the FIFA Skeletal Tracking Challenge 2026, which requires estimating the 3D world-space poses of soccer players from broadcast video. The input consists of video clips (50 fps, 1920×1080) with provided per-frame bounding boxes, camera intrinsics ( $K_t$ ), radial distortion coefficients ( $k_t$ ), and an initial camera pose ( $R_0, t_0$ ).

The objective is to predict 3D joint positions for $N$ tracked players across $T$ frames in a shared world coordinate system. Each player is represented by a FIFA-15 skeleton (15 joints, including Nose, shoulders, elbows, wrists, hips, knees, ankles, and big toes). Performance is evaluated using two metrics in meters:

Global MPJPE: Mean per-joint error in absolute world coordinates.
Local MPJPE: Mean per-joint error after subtracting the root (mid-hip).

The final competition score is defined as $L = \text{Global MPJPE} + 5 \times \text{Local MPJPE}$ , emphasizing the 5× higher leverage of local pose quality.

Methodology

The proposed SMART pipeline consists of four sequential stages:

1. 3D Body Mesh Recovery (SMPLest-X)

The system utilizes SMPLest-X, a regression-based model with a ViT-H backbone (687M parameters) built on the SMPL-X representation.

Input Processing: Player bounding boxes are expanded by 1.2× and resized to 512×384 pixels.
Inference: The model regresses SMPL-X parameters (shape $\beta$ , pose $\theta$ , global orientation) to produce a full body mesh (10,475 vertices). The 15 FIFA-15 joints are extracted via a fixed vertex lookup on the mesh.
Domain-Adaptive Fine-Tuning: To bridge the domain gap between pre-trained models and broadcast soccer (characterized by small player sizes and wide angles), the authors employ:
- Stratified Clip Split: 89 clips from the WorldPose dataset are split into 70 training and 19 validation clips with zero frame overlap and matched camera distributions.
- Multi-Task Loss: $L_{train} = \lambda_1 L_{3D} + \lambda_2 L_{2D} + \lambda_3 L_{depth}$ . $L_{3D}$ is weighted MPJPE (extremities weighted 3×), $L_{2D}$ is L2 loss on projected keypoints, and $L_{depth}$ is L1 loss on camera-space pelvis depth.
- Broadcast Augmentation: Random resized cropping, horizontal flipping with symmetric joint permutation, and color jitter.

2. Camera Pose Estimation (RAFT)

Instead of the baseline's Lucas-Kanade sparse tracker, the method uses RAFT-small for dense optical flow to propagate camera rotation ( $R_t$ ) and translation ( $t_t$ ) from the initial pose.

Flow Computation: Dense flow is computed between frames within the convex hull of 714 projected pitch landmarks.
Outlier Rejection: Flow vectors contaminated by player motion are removed using MAD outlier rejection (3 $\sigma$ threshold).
Pose Decomposition: A homography is fitted via RANSAC and decomposed into ( $R_t, t_t$ ). Updates with rotation changes $|\Delta R| > 60^\circ$ are rejected to prevent tracking failures.
Refinement: If field line intersections are detected, an EPnP solve corrects the absolute camera position on the first frame.

3. World-Space Skeleton Placement

The camera-space skeleton is lifted to the world coordinate system through three steps:

Rotation & Initial Placement: The root-relative skeleton is rotated to the world frame ( $\hat{S}_t = R_t S^{(cam)}_t$ ), and the pelvis translation is projected to world space ( $p_t = R_t p^{(cam)}_t + t_t$ ).
Foot-Plane Anchoring: The lowest foot pixel in the bounding box is back-projected to a camera ray and intersected with the known pitch-plane equation to find the ground contact point ( $f_t$ ). All joints are translated so the lower ankle aligns with $f_t$ .
Reprojection Refinement: A two-stage L-BFGS optimization aligns the skeleton centroid with the bounding box center and then aligns projected joints with 2D keypoints.

4. Temporal Smoothing

A two-pass smoothing strategy removes detection jitter:

Pass 1 (Global): Global root trajectories are cleaned using MAD outlier detection (flagging "teleporting" jumps), linear interpolation, and a Gaussian filter ( $\sigma=3$ frames).
Pass 2 (Local): Root-relative joint offsets are smoothed independently with a narrower Gaussian ( $\sigma=1.5$ frames) to remove single-frame jitter while preserving rapid limb movements (e.g., kicks).

Key Contributions and Ablation Results

The paper presents a systematic ablation study demonstrating the necessity of each component:

Fine-Tuning Strategy:
- Baseline (SAM-3D): Score 1.053.
- SMPLest-X Pretrained: Score 0.846 (shows domain gap).
- Naïve Fine-tuning: Score 0.820 (improves global but worsens local MPJPE due to viewpoint overfitting).
- Foot-Plane Anchoring: Reduces score to 0.714 by grounding skeletons to the pitch plane.
- Full Strategy (SMART): Achieves 0.647 on validation. The authors identify depth supervision ( $L_{depth}$ ) as the dominant factor; without it, local MPJPE stagnates at 0.067 m regardless of other augmentations.
Camera Tracker:
- ECC Homography: Performs poorly (0.107° error) due to inability to ignore moving players.
- Lucas-Kanade (Baseline): 0.043° error.
- RAFT-small + MAD: Achieves 0.041° error. The authors note that dense flow over player silhouettes introduces noise, which is effectively mitigated by MAD filtering. RAFT-large offered no further gain at double the compute cost.
Temporal Refiner:
- A learned sliding-window temporal network reduced local MPJPE on the training split but degraded competition performance (0.680 vs. 0.647) by over-smoothing fast actions not well-represented in the training distribution. Consequently, the final submission relies on Gaussian smoothing only.

Results

SMART demonstrates significant improvements over the FIFA baseline:

Validation Set: Score reduced from 1.053 (Baseline) to 0.647 (SMART), a 38.6% improvement.
- Global MPJPE: 0.602 $\to$ 0.370 m.
- Local MPJPE: 0.090 $\to$ 0.055 m.
Held-out Test Set: SMART achieves a score of 0.593.
- Global MPJPE: 0.324 m.
- Local MPJPE: 0.054 m.

Significance and Limitations

The paper claims that the primary bottleneck for global error is camera tracking, particularly in sequences with aggressive zoom that cause apparent pitch-plane deformation. Conversely, local error is largely camera-independent once the fine-tuned lifter and foot-plane anchoring are in place.

The authors modestly acknowledge specific limitations:

Diving Poses: Underrepresented in the training data (WorldPose), leading to struggles with extreme configurations and failures in foot-plane anchoring when players are airborne.
Occlusion: The per-frame nature of SMPLest-X makes it fragile under occlusion; a temporally-aware lifter is suggested as a future improvement.
Compute Cost: The 687M ViT-H backbone dominates inference cost, with the authors noting that a ViT-S backbone could yield a 4× speedup with modest accuracy loss.

The work establishes that combining domain-adaptive fine-tuning with robust dense optical flow tracking and geometric grounding (foot-plane anchoring) significantly outperforms standard baselines in broadcast soccer pose estimation.