💻 computer science

Motion Cues from Image-based Point Tracking for LiDAR Scene Flow Estimation

The paper introduces TrackCue, a self-supervised framework that leverages dense image-space point tracking and visual motion compensation to refine static-dynamic classification and provide more reliable supervision for LiDAR scene flow estimation, thereby overcoming the limitations of sparse geometric observations.

Original authors: Youngdong Jang, Gyeongrok Oh, Jong Wook Kim, Hyunju Ryu, Hyung-gun Chi, SeungHyeon Kim, Seungryong Kim, Jonghyun Choi, Sangpil Kim

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youngdong Jang, Gyeongrok Oh, Jong Wook Kim, Hyunju Ryu, Hyung-gun Chi, SeungHyeon Kim, Seungryong Kim, Jonghyun Choi, Sangpil Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Car to See Moving Objects

Imagine you are trying to teach a self-driving car to understand the world. The car has a LiDAR sensor (like a high-tech, 3D laser flashlight) that shoots out thousands of tiny laser beams to map the road. It also has cameras that see the world just like human eyes do.

The goal of this paper is to help the car figure out Scene Flow. Think of this as a "motion map." For every single dot the laser hits, the car needs to know: Is this dot moving? If so, how fast and in what direction?

The Problem: The "Sparse" Laser vs. The "Crowded" Eye

The authors point out a major headache with using only the laser (LiDAR):

The Laser is Spotty: Imagine trying to track a running dog by looking at a few scattered, floating dots in the air. Sometimes the dots are far apart, or the dog is hidden behind a bush (occlusion). The laser might miss the dog entirely or think a stationary wall is moving because a few dots disappeared.
The Camera is Dense: Now, imagine watching that same dog through a video camera. You see the whole dog, every frame, with smooth, continuous movement.

The Current Mistake: Existing self-supervised methods (systems that learn without human teachers) rely too much on the "spotty" laser data. They try to guess which dots are moving based on gaps in the laser beam. This leads to noisy labels—the system gets confused, thinking a parked car is moving or missing a moving pedestrian. When the system learns from these wrong clues, it gets bad at predicting motion.

The Solution: TrackCue (The "Video Detective")

The authors introduce a new framework called TrackCue. Instead of relying only on the spotty laser, they bring in the "video detective" (the camera) to help sort things out.

Here is how TrackCue works, step-by-step:

1. The Handoff (Projecting the Laser to the Camera)

First, the system takes the specific dots the laser thinks might be moving and projects them onto the camera's view. It's like saying, "Hey camera, look at these specific spots on your screen."

2. The Chase (Image-Based Point Tracking)

The system uses a powerful "point tracker" (a type of AI trained on millions of videos) to follow those spots across the video frames.

The Analogy: Imagine you put a sticker on a moving car and a sticker on a parked tree. The point tracker follows the stickers. Because the camera sees the whole picture, it can follow the sticker on the car even if the laser lost the dot for a second. It creates a smooth, continuous line of movement (a trajectory).

3. The "Ego-Motion" Filter (Separating You from the World)

Here is a tricky part: When the self-driving car moves forward, everything in the camera view appears to move backward, even the stationary trees.

The Analogy: Imagine you are on a train looking out the window. The trees seem to zoom backward. If you just looked at the video, you'd think the trees are flying.
The Fix: TrackCue calculates exactly how the car itself is moving (ego-motion). It draws a "rigid path" for what a stationary object should look like in the video. Then, it compares the actual path the tracker found against this rigid path.
- If the sticker follows the rigid path perfectly? It's a stationary object. (Ignore it).
- If the sticker swerves away from the rigid path? It's a moving object! (Keep it).

4. The Lift (Sending the Clues Back to the Laser)

Now the system has a clean list of "moving" and "stationary" points based on the video. But the car drives using the laser map. So, TrackCue takes these video clues and "lifts" them back up to the 3D laser world.

It finds the laser dot closest to the moving video sticker and says, "You are moving too."
This creates a refined map where the laser dots are correctly labeled as moving or static, even if the laser data was sparse or blocked.

The Result: A Cleaner, Smarter Map

By using the camera's smooth, dense vision to clean up the laser's spotty data, TrackCue acts like a filter that removes the "noise."

Before: The system was confused, often thinking walls were moving or missing pedestrians.
After: The system correctly identifies moving cars and people with much higher accuracy.

The paper shows that when they use these cleaner labels to train the self-driving AI, the AI becomes much better at predicting 3D motion. It's like giving a student a textbook with fewer typos; they learn the subject much faster and more accurately.

Summary

TrackCue is a method that uses video tracking to fix the mistakes made by laser sensors. It filters out the "fake motion" caused by the car driving itself, isolates the "real motion" of other objects, and sends those correct labels back to the laser system. This results in a self-driving car that understands the moving world much more reliably.

Technical Summary: TrackCue for LiDAR Scene Flow Estimation

Problem Statement

LiDAR scene flow estimation is critical for autonomous driving, providing 3D motion vectors for individual points to support tasks like motion prediction and planning. However, acquiring ground-truth 3D motion annotations is prohibitively expensive, leading to a reliance on self-supervised approaches. A primary challenge in feed-forward self-supervised methods is the severe class imbalance between static and dynamic points; most points in driving scenes are static. Without addressing this, models tend to bias toward predicting zero flow.

Existing self-supervised methods attempt to mitigate this by classifying points as static or dynamic to apply different supervision strategies. These methods typically rely on DUFOMap, a ray-casting-based dynamic-awareness mapping technique that identifies dynamic points by detecting void regions in LiDAR point clouds. However, DUFOMap is vulnerable to LiDAR data sparsity and occlusions. In sparse or occluded regions, incomplete ray evidence leads to incorrect void estimation, causing static points to be mislabeled as dynamic or moving points to be missed. These noisy pseudo-labels provide incorrect motion guidance, degrading the learning of scene flow.

Methodology: TrackCue

To address the limitations of LiDAR-only observations, the authors propose TrackCue, a tracking-guided framework that leverages dense visual motion cues from RGB images to refine static-dynamic classification. The framework operates through three core components:

1. Image-based Point Tracking

TrackCue repurposes point tracking to obtain dense trajectories anchored to LiDAR points.

Projection: LiDAR points initially labeled as dynamic by DUFOMap are projected onto the image plane using camera intrinsics and extrinsics.
Tracking: These projected pixels serve as queries for a pre-trained point tracker (e.g., CoTracker3). The tracker predicts long-term, temporally consistent trajectories across consecutive frames.
Benefit: This provides dense visual motion cues that remain continuous even when LiDAR observations are sparse or occluded, identifying dynamic points that are ambiguous in the 3D domain alone.

2. Visually Consistent Motion Compensation

A critical challenge in image-space tracking is distinguishing true object motion from apparent motion caused by the ego-vehicle's movement.

Rigid Trajectory Construction: For each tracked point, a "rigid trajectory" is computed by projecting the point through the world coordinate system using the vehicle's ego-poses (odometry). This represents where a static point should appear in the image if it were stationary.
Residual Analysis: The system calculates the trajectory residual between the tracker-predicted path and the rigid trajectory.
Selection: Points whose trajectories consistently deviate from the rigid path (exceeding a threshold over a temporal window) are identified as independently moving objects. This effectively filters out static points that appear to move due to ego-motion.

3. Visual Motion Cue Lifting

The refined motion cues from the 2D image plane are transferred back to the 3D LiDAR domain.

Association: The ego-compensated image trajectories are associated with LiDAR points in the same frame via nearest-neighbor search in the projected image space.
Label Refinement: If the distance between a tracked trajectory and a LiDAR point is within a specific lifting radius, the LiDAR point receives a dynamic cue.
Outcome: This process generates a refined set of static-dynamic labels ( $P_{TrackCue}$ ) that complement the original DUFOMap labels, recovering dynamic points missed by sparse LiDAR geometry.

Key Contributions

TrackCue Framework: A novel pipeline for self-supervised LiDAR scene flow that integrates image-based point tracking to identify dynamic points using dense visual motion cues, overcoming the sparsity limitations of LiDAR-only methods.
Visually Consistent Motion Compensation: A strategy that isolates true object motion from ego-induced apparent motion by comparing tracked image trajectories against rigid trajectories derived from ego-poses.
Visual Motion Cue Lifting: A mechanism that projects ego-compensated 2D trajectory evidence back to 3D LiDAR points, producing refined static-dynamic labels that improve the reliability of self-supervised supervision.

Experimental Results

The authors evaluated TrackCue on the Argoverse 2 dataset and tested its zero-shot generalization on Waymo, nuScenes, TruckScenes, and AEVAScenes. The method was integrated into three baseline self-supervised frameworks: SeFlow, SeFlow++, and TeFlow.

Label Quality: TrackCue significantly improved the quality of dynamic pseudo-labels. Compared to DUFOMap, it reduced false-positive dynamic labels by 35.6% and improved the F1 score for dynamic classification from 36.7% to 42.4% (in the SeFlow context).
Scene Flow Performance: The refined labels led to measurable gains in scene flow estimation metrics (Dynamic Bucket-Normalized EPE).
- On the Argoverse 2 leaderboard, TrackCue reduced the overall error for SeFlow++ by 4.06 points.
- Improvements were particularly pronounced in zero-shot settings on unseen datasets (e.g., a 13.09-point reduction on AEVAScenes), suggesting that high-fidelity pseudo-labels enhance cross-dataset generalization.
Qualitative Analysis: Visualizations demonstrated that TrackCue accurately estimates flow velocity for dynamic objects and prevents erroneous flow estimation in static background regions, a common failure mode in baselines that misclassify static points as dynamic.

Significance and Claims

The paper claims that TrackCue offers a promising direction for future research by integrating RGB images not merely to enhance feature representations, but to provide cross-modal supervision for 3D scene flow estimation. By leveraging the temporal consistency and density of video data, the method addresses the inherent sparsity and occlusion issues of LiDAR sensors.

The authors emphasize that their work focuses on refining the reliability of self-supervised dynamic labels, which is a prerequisite for effective scene flow learning. They note that while the method improves performance, it is not without limitations: the 3D-to-2D projection inevitably loses geometric information, potentially affecting small or distant objects. Future work is suggested to incorporate object masks from foundation models (like SAM) to provide spatial constraints and to explore richer cross-modal information exchange.

Ultimately, the paper posits that reducing the dependence on costly manual annotations while improving the fidelity of pseudo-labels through multi-modal cues is a critical step toward scalable and robust autonomous driving perception systems.