FlashCap: Millisecond-Accurate Human Motion Capture via Flashing LEDs and Event-Based Vision

This paper introduces FlashCap, a novel millisecond-accurate human motion capture system utilizing flashing LEDs and event-based vision, which includes the high-quality FlashMotion dataset and the ResPose baseline to enable precise motion timing and high-temporal-resolution human pose estimation.

The Gist

Imagine you are trying to film a hummingbird's wings. If you use a standard video camera, the wings just look like a blurry mess because the camera isn't fast enough to catch every tiny flap. You know the bird is moving, but you can't see exactly how or when.

This is the problem scientists face when trying to study fast human movements, like a boxer's punch or a sprinter's start. Standard cameras (30 or 60 frames per second) are too slow to capture the split-second details that decide a gold medal from a silver one.

Enter "FlashCap": The High-Speed Flashlight Solution

The researchers built a new system called FlashCap to solve this. Here is how it works, broken down into simple concepts:

1. The "Blinking Suit" (The Flashing LEDs)

Instead of trying to take thousands of photos per second (which is expensive and creates huge files), they put a special suit on the athletes. This suit has 17 tiny, flashing lights (LEDs) attached to key body parts like elbows, knees, and shoulders.

Think of these lights like fireflies that blink in a specific, unique rhythm. One firefly blinks "on" for a tiny bit, then "off" for a tiny bit. Another blinks at a slightly different speed. Because they blink so fast (thousands of times a second), they create a unique "signature" for every body part.

2. The "Event Camera" (The Super-Sensitive Eye)

To see these blinking fireflies, they didn't use a normal camera. They used an Event Camera.

• Normal Camera: Takes a full photo every 1/60th of a second. If the firefly blinks in between photos, the camera misses it.
• Event Camera: It doesn't take "photos." Instead, it's like a super-sensitive eye that only notices changes. The moment a pixel sees a light turn on or off, it shouts, "Hey! Something changed here at exactly this millisecond!"

This allows the system to track the blinking lights with millisecond precision (1,000 times a second) without needing massive storage or expensive lighting.

3. The "ResPose" Brain (Connecting the Dots)

Now, the system has two streams of data:

1. The "Anchor": A normal, slow video (RGB) that shows the person's general shape and color.
2. The "Residual": The super-fast data from the blinking lights that shows the tiny, rapid movements.

They created an AI called ResPose (Residual Pose). Imagine you are trying to draw a fast-moving car.

• First, you draw the car's general shape (the "Anchor" from the normal video).
• Then, you use the blinking lights to add the tiny, fast details (the "Residual") that the normal video missed.

By combining these two, ResPose can reconstruct the exact movement of the person's joints at a speed no other public system has ever achieved.

Why Does This Matter?

The paper introduces a new dataset called FlashMotion. Think of this as a "training gym" for AI, but instead of slow-motion exercises, it's filled with ultra-fast, millisecond-accurate movements.

• The Problem: Before this, if you wanted to know exactly when a runner's foot hit the ground, you might be off by 50 milliseconds. In a race, that's the difference between winning and losing.
• The Solution: FlashCap gets the timing down to single-digit milliseconds (like 4.8ms).
• The Result: They tested their AI (ResPose) against other methods. The old methods were like trying to guess the path of a bullet by looking at a blurry photo. ResPose was like having a high-speed radar. It reduced errors by about 40% and could predict movements with incredible accuracy.

The Big Picture

This isn't just for sports. This technology could help:

• Robots learn to catch a ball without dropping it.
• Doctors analyze a patient's gait to detect subtle neurological issues.
• Animators create hyper-realistic movie characters that move with true physics.

In short, FlashCap is like giving computers "super-vision" to see the invisible, split-second details of human motion that were previously impossible to capture.

Technical Summary

1. Problem Statement

Precise Motion Timing (PMT) is critical for analyzing swift human movements in competitive sports (e.g., fencing, speed climbing, luge), where millisecond differences determine outcomes. However, current Human Pose Estimation (HPE) research lacks the necessary infrastructure for PMT due to:

• Limited Temporal Resolution: Public datasets typically peak at 120 Hz, which is insufficient for ultra-fast motion analysis (e.g., swing trajectory reconstruction).
• Hardware Limitations: Traditional optical motion capture (MoCap) systems are limited by camera frame rates (30–60 Hz for standard RGB, up to ~330 Hz for specialized systems). High-speed RGB cameras (≥1000 Hz) exist but are prohibitively expensive, require extreme lighting, and generate massive bandwidth/storage burdens.
• Ground Truth (GT) Bottleneck: Existing methods for generating high-frequency GT labels suffer from hardware sampling ceilings or interpolation errors during rapid motion, failing to provide true millisecond-accurate ground truth.

2. Methodology

The authors propose FlashCap, a novel system combining flashing LEDs, event cameras, and a custom annotation pipeline to achieve 1000 Hz motion capture.

A. Hardware System

• MoCap Outfit: A suit equipped with 17 LEDs and 17 IMUs (Xsens). Each LED is mounted on a specific body segment (avoiding joints to reduce displacement artifacts) and controlled by a chip to emit green light at a configurable frequency (e.g., 4000 Hz).
• Multi-modal Capture Device:
  • Event Camera: (Prophesee 1280×720) captures asynchronous events triggered by LED flashes.
  • RGB Camera: (Hikrobot 1920×1200) captures standard frames at 20 FPS for structural context.
  • LiDAR & IMU: Used for verification and 3D reconstruction (LiDAR at 20 FPS).
• Synchronization: Rigorous temporal synchronization and spatial calibration (using a beam splitter) ensure alignment between event streams and RGB frames.

B. Data Annotation Pipeline

The core innovation is generating 1000 Hz 2D ground truth labels directly from the event stream without relying on high-speed RGB cameras. The pipeline involves:

1. Event Cluster Identification: Asynchronous events are binned into time windows (e.g., 1ms) and clustered using DBSCAN to identify LED locations.
2. Frequency Identification: Analyzing polarity changes in the event sequence to calculate the on-time ($t^p$), off-time ($t^n$), and flicker period ($T$) for each cluster.
3. Outlier Filtering: Removing noise-induced fluctuations and transient interference.
4. LED-Cluster Matching: A bipartite matching algorithm assigns event clusters to specific LEDs based on:
   • On-off time distance: Difference in $t^p$ and $t^n$.
   • Period distance: Difference in flicker periods.
   • Tracking: A tracking algorithm maintains cluster identity across frames to handle occlusions.

C. Proposed Model: ResPose

To address the gap between low-frame-rate inputs (RGB) and 1000 Hz ground truth, the authors propose ResPose, a hybrid architecture:

• Input: Low-frame-rate RGB frames (Anchor) + High-frequency Event streams (Residual).
• Architecture:
  • RGB Branch: Generates a static "anchor" pose ($P_{rgb}$) using a standard HPE model (e.g., ViTPose).
  • Event Branch: A SNN-CNN Hybrid Encoder extracts micro-movement dynamics. It uses Leaky Integrate-and-Fire (LIF) neurons for temporal spike integration and dynamic cropping centered on RGB anchors to filter background noise.
  • Fusion: A Multimodal Residual Transformer fuses the RGB anchors and event features. It employs Skeleton-aware Self-Attention to enforce kinematic consistency.
• Output: The final high-temporal-resolution pose is calculated as $P_i = P_{rgb} + P^\Delta_i$, where $P^\Delta_i$ represents the residual pose derived from events.

3. Key Contributions

1. FlashCap System: The first flashing LED-based MoCap system capable of capturing human motion at 1000 Hz with low bandwidth and cost, bypassing the limitations of high-speed RGB cameras.
2. FlashMotion Dataset: The first publicly available multi-modal dataset with millisecond-accuracy pose labels.
   • Specs: 240 sequences, 20 subjects, 4 modalities (RGB, Event, LiDAR, IMU).
   • Labels: 1000 Hz 2D labels (7.15M frames) and 60 Hz 3D SMPL labels.
   • Significance: Offers an order of magnitude higher temporal resolution than existing state-of-the-art datasets (max 120 Hz).
3. ResPose Baseline: A simple yet effective method that treats high-frequency events as residual signals to refine static RGB anchors, achieving millisecond-level tracking accuracy.
4. New Tasks: Defined and benchmarked two novel tasks: Precise Motion Timing (PMT) and High-Temporal-Resolution HPE.

4. Experimental Results

A. Dataset Quality Validation

• Qualitative: FlashMotion labels overlap closely with high-speed (100 Hz) RGB camera captures, even during rapid motions like hand-waving where standard interpolation fails.
• Quantitative: The annotation pipeline achieves 99.99% precision and 98.82% recall against manual annotations. Ablation studies confirm that removing frequency matching or outlier filtering significantly degrades performance.

B. Precise Motion Timing (PMT)

• Metric: Mean Error of Estimated Time (ms).
• Results:
  • Standard RGB methods (ViTPose): ~48–62 ms error.
  • Pure Event methods (Hybrid ANN-SNN): ~54–85 ms error.
  • ResPose: Achieved 4.8 ms to 7.2 ms error (e.g., 4.8 ms for punching).
• Insight: Existing methods fail to capture the exact moment a joint crosses a line due to low frame rates; ResPose successfully bridges this gap.

C. High-Temporal-Resolution HPE

• Metric: Mean Per Joint Position Error (MPJPE) and PCK.
• Results:
  • ResPose achieved the lowest MPJPE of 5.66, outperforming all baselines (next best was EvSharp2Blur at 8.78).
  • ResPose reduced pose estimation errors by ~40% compared to standard RGB interpolation.
  • Visualizations show ResPose generates smooth, high-fidelity trajectories that match ground truth dynamics, whereas interpolated methods produce jagged or inaccurate paths during fast motion.

5. Significance and Impact

• Paradigm Shift: FlashCap demonstrates that millisecond-accurate motion capture is feasible without expensive high-speed cameras, using low-cost event cameras and flashing LEDs.
• Research Enabler: The FlashMotion dataset provides the community with the first high-frequency ground truth necessary to develop and train algorithms for ultra-fast motion analysis.
• Application Potential: The technology has immediate applications in sports science (performance analysis, injury prevention), robotics (fine-grained motion imitation), and human-computer interaction.
• Methodological Advance: The ResPose architecture proves that fusing structural priors (RGB) with temporal residuals (Events) is a viable path toward sub-10ms tracking accuracy, setting a new benchmark for future HPE research.