FalconApp: Rapid iPhone Deployment of End-to-End Perception via Automatically Labeled Synthetic Data

1. Problem Statement

Reliable robotics perception (e.g., for manipulation, navigation, and autonomous driving) typically requires large-scale, manually annotated real-world datasets. This process is:

• Time-consuming and expensive: Manual annotation of masks and 6-DoF poses is slow.
• Sim-to-Real Gap: Traditional simulators (e.g., CARLA, Gazebo) often lack photorealism, leading to poor performance when models trained in simulation are deployed on real robots.
• Workflow Gaps: While recent advances like NeRF and 3D Gaussian Splatting (GSplat) offer photorealistic reconstruction, they do not inherently provide an end-to-end workflow that converts casual user videos into deployable perception modules on edge devices.

Goal: Create a system that allows a user to capture a short video of a rigid object on an iPhone and automatically generate a high-performance, object-specific perception module for mask detection and 6-DoF pose estimation within minutes, without manual labeling.

---

2. Methodology

FalconApp is an iPhone application with a frontend-backend pipeline consisting of four distinct stages:

A. Frontend: Data Capture (Recording Mode)

• Implementation: Built in Swift for iOS.
• Process: The user walks around a rigid object to record a short RGB video. The app captures multi-view coverage along with camera metadata.
• Transmission: Video and metadata are uploaded to the backend via a TCP client, avoiding manual file transfers.

B. Backend: Automated Synthetic Data Labeling

• 3D Reconstruction: The backend uses FalconGym to reconstruct the object as a 3D Gaussian Splatting (GSplat) representation from the video.
• Asset Editing: Using FalconGym 2.0's Edit API, the object splats are separated from the background to create an editable, isolated 3D asset.
• Domain Randomization: The object is composited with diverse photorealistic background GSplats. Images are rendered under randomized viewpoints, scales, and lighting conditions to minimize the sim-to-real gap.
• Auto-Labeling: Because the rendering engine controls the scene, every synthetic image is automatically paired with:
  • Pixel-accurate foreground masks.
  • Ground-truth relative 6-DoF pose.
  • Empty-scene images (to reduce false positives).
• Efficiency: Rendering and labeling take approximately 0.2 seconds per image (1280×720) on an RTX 4090 GPU.

C. Backend: Perception Module Training

• Architecture: A custom network (Figure 4) featuring:
  • An Encoder for feature extraction.
  • A Mask Prediction Head.
  • A Gated-Attention Pose Head: The predicted mask conditions the pose head, allowing the model to focus on object features while suppressing background clutter.
• Training Strategy:
  1. Stage 1: Optimize mask prediction.
  2. Stage 2: Jointly optimize mask and pose using synthetic labels.
  3. Loss Functions: Combines segmentation loss, pose regression loss, and a reprojection consistency term (re-rendering the object at the predicted pose and comparing via SSIM) to enforce geometric consistency.

D. Frontend: Deployment & Inference

• Packaging: The trained model is packaged and sent back to the iPhone.
• Inference: The app loads the model and performs live inference on incoming camera frames.
• Output: Visualizes the segmentation mask and the estimated 6-DoF pose relative to the camera.
• Latency: The system achieves real-time performance (~30 ms end-to-end latency).

---

3. Key Contributions

1. Rapid Mobile Deployment Pipeline: A complete system that transforms a 2-minute casual iPhone video into a functional, object-specific perception module in approximately 20 minutes.
2. Photorealistic Auto-Labeling Workflow: A novel approach using editable GSplats to generate infinite synthetic training data with perfect ground-truth labels (masks and poses), eliminating the need for manual annotation.
3. End-to-End Integration: Successfully bridges the gap between 3D reconstruction, synthetic data generation, model training, and on-device inference on consumer hardware (iPhone).

---

4. Experimental Results

The system was evaluated on five rigid objects with diverse geometry, scale, and symmetry (Car, Quadrotor, Gate, Plane, Lamp).

Efficiency Metrics

• Total Pipeline Time: ~20 minutes per object (including 1000 synthetic images generation and training).
• On-Device Latency: ~30 ms per frame (enabling ~30 fps inference).
• Reconstruction Quality: High photorealism (SSIM 0.71–0.88, PSNR 30.0–32.4) across all objects.

Accuracy Metrics (vs. PnP Baseline)

FalconApp was compared against a traditional Perspective-n-Point (PnP) baseline, which relies on manually selected feature correspondences.

Metric	FalconApp Performance	Comparison to PnP
Mask IoU	0.62 – 0.83	N/A (PnP does not output masks)
Pose Accuracy (Real World)	Superior on 4/5 objects	FalconApp significantly reduced Mean Translational Error (MTE) and Mean Angular Error (MAE) on the Car, Quadrotor, Gate, and Plane.
Failure Case	Lamp	The Lamp performed worse than PnP due to rotational symmetry, which creates ambiguity in orientation estimation.

• Real-World Transfer: The models trained on synthetic data generalized well to real-world environments with different backgrounds, demonstrating effective Sim-to-Real transfer.

---

5. Significance and Future Work

Significance:
FalconApp demonstrates that on-demand, object-specific perception is feasible without large-scale manual data collection. It democratizes robotics perception by allowing non-experts to deploy custom vision models on mobile devices in minutes. The use of GSplat-based auto-labeling effectively solves the "labeling bottleneck" and the "sim-to-real gap" simultaneously.

Limitations & Future Work:

• Rigid Objects Only: Currently assumes rigid objects; deformable objects are not supported.
• Symmetry Issues: Performance degrades on highly symmetric objects (e.g., the lamp) where orientation is ambiguous.
• Future Directions:
  • Improved handling of symmetry and ambiguity.
  • Stronger domain randomization techniques.
  • Fully onboard deployment (running the entire pipeline on the device).
  • Integration with downstream tasks like aerial navigation and manipulation.