Velocity and stroke rate reconstruction of canoe sprint team boats based on panned and zoomed video recordings

This paper presents an automated video-based framework that leverages YOLOv8, U-net calibration, and optical flow to accurately reconstruct canoe sprint team boats' velocity and stroke rate from panned and zoomed recordings, achieving high agreement with GPS data without requiring on-boat sensors.

The Gist

Imagine you are watching a high-speed canoe race on TV. The boats are moving incredibly fast, the water is splashing, and the camera is zooming and panning to follow the action. As a coach or a fan, you want to know two things: How fast are they going? and How hard are they paddling?

Usually, to get this data, you'd need to strap a GPS tracker and a motion sensor onto every single boat. But that's expensive, logistically a nightmare, and sometimes against the rules.

This paper introduces a "magic trick" that lets you get that same super-accurate data just by watching the video footage. It's like having a super-powered computer vision system that can look at a shaky, zoomed-in video and say, "Ah, that boat is exactly 4.2 meters per second, and they are paddling 110 times a minute."

Here is how they did it, broken down into simple concepts:

1. The Problem: The "Fish-Eye" Distortion

When a camera on the shore films a race, it doesn't look like a flat map. It looks like a distorted, angled view. If you just measure how many pixels a boat moves in the video, you get the wrong speed because the camera is zooming in and out and moving side-to-side.

The Analogy: Imagine watching a race from a Ferris wheel. As the wheel turns, the runners look like they are speeding up or slowing down just because of your angle, even if they are running at a constant speed. The researchers needed a way to "flatten" that Ferris wheel view into a perfect, flat map.

2. The Solution: The "Virtual Grid"

The race course has buoys (floating markers) spaced out perfectly every 25 meters. The researchers taught their computer to spot these buoys and the athletes.

• The Trick: Because the computer knows exactly where the buoys should be in real life, it can calculate the camera's angle and distortion for every single frame of the video. It essentially draws a "virtual grid" over the video, turning the messy footage into a precise 3D map.

3. The New Challenge: Team Boats (The "Tetris" Problem)

Previous versions of this tech only worked for single-person boats. But what about boats with 2 or 4 people?

• The Issue: In a 4-person boat, the athletes are sitting in a line. Sometimes the camera angle hides the person in the back, or they overlap. If the computer loses track of who is who, it can't figure out where the front of the boat is.
• The Fix: They built a special "AI Detective" (called a U-Net) that acts like a super-precise ruler. Instead of guessing where the boat tip is based on a rough average, this AI looks at a tiny patch of the video and says, "I see the very tip of the boat right here." It learns the specific shape of the boat tip for every single race.

4. Keeping the Team Together: The "Optical Flow" Rope

What if the computer misses a frame where an athlete is hidden by a splash?

• The Fix: They used a technique called "Optical Flow." Imagine the athletes are tied together with an invisible rope. If the computer sees the person in the front, it knows exactly where the person in the back must be, even if they are temporarily hidden. It tracks their movement like a dance partner, ensuring the order of the team never gets mixed up.

5. Counting the Strokes: The "Heartbeat" of the Race

Finally, they needed to count how many times the paddles hit the water (Stroke Rate).

• Method A (The Simple Way): They looked at the brightness changes inside the box surrounding the athlete. When the paddle moves, the pixels change brightness. It's like listening to a drumbeat by watching the drum skin vibrate.
• Method B (The Smart Way): They used a pose-estimation AI (ViTPose) to find the athlete's shoulder and wrist. It calculates the distance between them. As the athlete paddles, this distance stretches and shrinks rhythmically.
• The Result: The "Smart Way" (Method B) was much more accurate, almost as good as having a sensor on the athlete's wrist.

Why This Matters

This system is a game-changer for coaches and athletes because:

1. No Sensors Needed: You don't need to buy expensive gear or worry about regulations. Just film the race.
2. Instant Feedback: Coaches can see exactly how the race was paced (fast start? slow finish?) and how the team synchronized their paddling.
3. Universal: It works for single boats, double boats, and four-person boats, whether it's a 200m sprint or a 500m race.

In a nutshell: The researchers turned a standard video camera into a high-tech motion lab. They taught a computer to understand the geometry of a race, track a team of paddlers even when they hide from the camera, and count their strokes just by watching the pixels dance. It's like giving a coach X-ray vision, but using only a smartphone and some clever math.

Technical Summary

1. Problem Statement

In canoe sprint racing, pacing strategies defined by velocity and stroke rate profiles are critical for elite performance. While Global Positioning System (GPS) technology is the gold standard for data collection, it is often unavailable for all participants in high-stakes competitions due to logistical constraints or regulatory prohibitions.

Existing video-based analysis methods face significant challenges:

• Perspective Distortion: Shore-based cameras use panned and zoomed views, creating skewed perspectives that make manual reconstruction of absolute boat positions labor-intensive and inaccurate.
• Multi-Athlete Complexity: Previous automated methods worked well for single-athlete boats (K1/C1) but failed for team boats (K2, C2, K4) because multiple athletes overlap, making it difficult to determine which athlete represents the boat's position.
• Lack of Stroke Rate Data: Prior automated systems focused solely on velocity, lacking the ability to extract stroke rate from video.
• Static Offsets: Previous methods used a fixed empirical offset to estimate the boat tip from an athlete's position, which introduces systematic bias when boat classes or recording conditions change.

2. Methodology

The authors propose an extended, fully automated framework that reconstructs velocity and stroke rate profiles from standard broadcast or coach-recorded footage without on-boat sensors. The pipeline consists of four main components:

A. Scene Geometry and Homography Estimation

• Detection: A fine-tuned YOLOv8 model detects athletes and race buoys in every frame.
• Mapping: Leveraging the known, standardized geometry of the regatta course (buoy grid), the system estimates a homography matrix ($H_i$) for each frame. This maps 2D image coordinates to real-world 2D coordinates on the water surface.
• Propagation: Using Lucas-Kanade optical flow, buoy correspondences are tracked across frames to maintain a consistent coordinate system even when detections are intermittent.

B. Automatic Boat Tip Calibration (U-Net)

To address the limitations of static offsets in team boats, the authors introduce a learning-based calibration:

• Problem: In team boats, the distance between an athlete and the boat tip varies by seat position. A single global offset is insufficient.
• Solution: A U-Net model is trained to detect the exact location of the boat tip within a $150 \times 150$ pixel Region of Interest (ROI).
• Process:
  1. An initial boat position is estimated using a standard empirical offset on the mean athlete position.
  2. This position is reprojected to the image to define an ROI.
  3. The U-Net predicts the precise tip location.
  4. Per-athlete offsets ($o_{j,k}$) are calculated by comparing the detected tip to each individual athlete's position. These offsets are averaged over a 5-frame sequence for robustness.

C. Robust Multi-Athlete Tracking

Handling occlusions and missed detections in team boats is critical:

• Athlete Ordering: When all athletes are detected, their order is established. If detections fail, optical flow is used to track the motion vectors of previously detected athletes to the current frame, preserving their relative order.
• Interpolation: For frames where athletes are missing, the system uses optical flow interpolation (both causal and non-causal) to estimate positions based on motion vectors from adjacent frames, ensuring a continuous distance-time signal.

D. Stroke Rate Extraction

Two methods are proposed to generate a 1D motion signal from which stroke rate is derived:

1. ViTPose-based: Uses a pose estimation model to calculate the Euclidean distance between the shoulder and wrist joints of the front athlete.
2. Bounding Box (BBox) based: Analyzes the mean brightness of a small sub-region (20% of the bounding box size) in the lower-left corner of the athlete's box, capturing the motion of the paddle/athlete.

• Signal Processing: The motion signal is filtered (Savitzky-Golay), local maxima are detected (merged within tolerance), and intervals between peaks are converted to frequency (strokes per minute).

3. Key Contributions

1. Generalization to Team Boats: The framework extends automated analysis from single-person boats (K1/C1) to all major sprint disciplines, including team boats (K2, C2, K4).
2. Learned Boat Tip Calibration: Replaces error-prone static offsets with a U-Net based detection system that learns specific athlete-to-tip offsets, significantly improving localization accuracy.
3. Robust Tracking Logic: Introduces an optical flow-based strategy to maintain athlete ordering and interpolate missing data, ensuring continuity in multi-athlete scenarios.
4. Dual-Mode Stroke Rate Extraction: Provides a pipeline to extract stroke rates directly from video, offering a choice between high-accuracy pose estimation and lower-compute bounding box analysis.
5. Comprehensive Evaluation: Validates the system against GPS and gyroscopic ground truth across diverse distances (200m, 500m) and boat types.

4. Results

The system was evaluated on 54 GPS/Gyro tracks from elite competitions (German National Qualifiers, World Championships, World Cup).

• Velocity Reconstruction:

  • Achieved a Relative Root Mean Square Error (RRMSE) of 0.020 ± 0.011.
  • High correlation with GPS ground truth (Spearman $\rho$ = 0.956).
  • Performance was consistent across all boat types (K1, C1, K2, C2, K4), with no significant degradation for team boats.
  • Ablation Study: Non-causal optical flow interpolation significantly outperformed linear interpolation (RRMSE 0.020 vs. 0.090) and simple ordering.

• Stroke Rate Reconstruction:

  • ViTPose Method: Achieved an RMSE of 2.352 bpm and RRMSE of 0.022, with a correlation of $\rho$ = 0.932 against gyro data.
  • Bounding Box Method: Performed worse (RMSE 7.846 bpm, $\rho$ = 0.686) but was 2.8x faster computationally.
  • The ViTPose method showed minimal bias and high agreement with ground truth, though it is computationally heavier (1179s vs 411s per video).

• Calibration Stability:

  • The U-Net boat tip detector achieved 90.8% accuracy within a 5-pixel tolerance.
  • Calibrated offsets for the same boat across different races showed a mean relative difference of only 2.92%, proving high reproducibility.

5. Significance

This work represents a major advancement in sports analytics for water sports:

• Accessibility: It democratizes high-precision tactical analysis by removing the need for expensive, invasive on-boat sensors, allowing coaches to analyze any standard video footage.
• Tactical Insight: By providing accurate, high-frequency velocity and stroke rate profiles for all competitors in a race (not just those with sensors), it enables deep comparative analysis of pacing strategies.
• Scalability: The method is applicable to all canoe sprint disciplines and distances, bridging the gap between single-athlete and complex team dynamics.
• Future Potential: The geometric approach is generalizable to other water sports (e.g., rowing) and could be adapted for real-time applications with further optimization of homography estimation.

In conclusion, the paper successfully demonstrates that computer vision, when combined with geometric constraints and deep learning, can match the accuracy of GPS and gyroscopic sensors for canoe sprint analysis, offering a non-invasive, automated solution for elite performance optimization.