Technical Summary: FalconTrack

Problem Statement

Vision-based aerial tracking is essential for applications like transportation, environmental monitoring, and search and rescue, particularly in GPS-denied environments. However, developing reliable tracking systems requires large-scale, accurately labeled datasets. Existing approaches face two primary bottlenecks:

1. Manual Annotation Costs: Real-world datasets (e.g., ImageNet, BDD100K) rely on costly, time-consuming, and error-prone human annotation, which does not scale well to irregular, non-parametric objects.
2. Sim-to-Real Gaps: While simulators like CARLA and Gazebo offer automated labeling, their outputs often lack photorealism, leading to poor performance when models are deployed in the real world. Conversely, photorealistic generative models (e.g., diffusion-based) often fail to provide precise 6-DoF pose labels or guarantee target appearance fidelity.
3. Limited Generalization: Current aerial tracking methods often overfit to specific track geometries or single object types (e.g., static gates) and fail to generalize to irregular, dynamically moving targets or unseen backgrounds.

Methodology

The authors propose FalconTrack, a unified framework that integrates automated data generation with a physics-aware tracking controller to enable zero-shot sim-to-real transfer.

1. Automated Labeling Pipeline

To address the data scarcity and photorealism gap, the authors developed an automated pipeline using 3D Gaussian Splatting (GSplat) within the FalconGym 2.0 simulator:

• Data Collection: A short (approx. 2-minute) handheld video of a target object is captured.
• Reconstruction & Isolation: The video is used to reconstruct a 3D Gaussian Splatting model of the target. Using SAGA and the FalconGym 2.0 Edit API, the target's splats are isolated from the background.
• Compositing & Rendering: The isolated target GSplats are fused with diverse background GSplats under domain randomization (varying pose, scale, and lighting).
• Label Generation: Using known camera intrinsics/extrinsics and rendering transforms, the system automatically generates pixel-accurate segmentation masks and 6-DoF pose labels for each rendered frame.
• Efficiency: This process generates approximately 10,000 labeled images (RGB, mask, class, pose) in under 20 minutes without manual annotation.

2. Unified Perception Module

The perception module is a multi-head neural network designed for onboard deployment:

• Architecture: It utilizes a shared ResNet-18 encoder feeding three heads: classification, segmentation (mask), and 6-DoF pose regression.
• Gated Attention: The pose head uses gated attention conditioned on the predicted mask to suppress background clutter while preserving appearance cues necessary for orientation estimation.
• Staged Training: To ensure stability, the network is trained in three stages:
  1. Train the classification head (freeze others).
  2. Train classification and segmentation heads (freeze pose).
  3. Jointly train all heads.
• Reprojection Consistency: A key innovation is the inclusion of a reprojection loss. The predicted pose is used to re-render the target in FalconGym 2.0, and the result is compared to the input image using SSIM. This enforces geometric consistency and corrects pose errors, particularly in yaw.

3. Physics-Aware Tracking Controller

The tracking controller fuses perception outputs with target dynamics to maintain robust tracking:

• State Estimation: An Extended Kalman Filter (EKF) fuses raw pose estimates from the perception module with class-specific dynamics priors (e.g., Dubins car model for F1-tenth, rigid-body dynamics for quadrotors, double-integrator for gates). This smooths noise and allows the system to recover from brief out-of-view events.
• Visual Servoing: A Pose-Based Visual Servoing (PBVS) controller uses the EKF-estimated state and a desired relative offset (specific to the target class) to generate body-frame velocity and yaw rate commands.

Key Contributions

1. Automated Labeling Pipeline: A method to generate photorealistic, auto-labeled datasets for diverse, irregular objects without manual annotation, significantly reducing the sim-to-real gap.
2. Unified Multi-Head Perception: A perception architecture employing staged learning and reprojection consistency that achieves zero-shot transfer across three geometrically diverse objects (F1-tenth vehicle, quadrotor, gate) and unseen backgrounds.
3. Physics-Aware Tracking: A closed-loop tracking system that integrates class-conditioned dynamics priors into an EKF, enabling robust tracking on real hardware across multiple target types and trajectories.

Experimental Results

The framework was evaluated in FalconGym 2.0 simulations and on a real quadrotor (NVIDIA Jetson Orin) across two environments.

• Perception Performance:
  • The full FalconTrack model achieved 96–100% class accuracy in zero-shot sim-to-real transfer.
  • It outperformed baselines (PnP and Neural Pose Estimators) in Mean Translational Error (MTE) and Mean Angular Error (MAE).
  • Ablation studies confirmed that both staged training and reprojection loss significantly improved pose accuracy.
• Tracking Performance:
  • Success Rate: FalconTrack achieved 100% success in closed-loop tracking for F1-tenth and gate targets in both simulation and real-world hardware trials.
  • Robustness: In real-world F1-tenth tracking, FalconTrack maintained 100% success, whereas a mask-centered vision baseline dropped to 60% success during fast out-of-view scenarios.
  • Real-Time Operation: The onboard system runs at approximately 25 Hz, sufficient for real-time control.
• Operational Design Domain (ODD): The system demonstrates reliable performance for target distances between 2 and 6 body lengths.

Significance and Claims

The paper claims that FalconTrack successfully bridges the gap between photorealistic image synthesis and automated labeling for irregular objects. By combining an automated labeling pipeline with a physics-aware tracking controller, the system achieves robust, zero-shot sim-to-real transfer without requiring manual annotation or target-specific retraining. The authors emphasize that their approach generalizes across different object geometries and environments, overcoming the limitations of previous methods that were either restricted to static gates, required manual labeling, or failed to generalize to unseen targets.

The authors modestly note current limitations: the system assumes a single target per run, requires a short video for initial GSplat reconstruction (limiting open-world deployment to unknown objects), and has not yet been validated on very fast targets (>2 m/s) or in quadrotor-vs-quadrotor scenarios due to safety constraints.