E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy

1. Problem Statement

Training and deploying end-to-end learning policies for quadrotors (mapping raw sensor data directly to control commands) faces significant hurdles in moving from simulation to reality (Sim-to-Real). Key challenges include:

• Inefficiency and Inaccuracy: Existing simulators often lack high-performance rendering, differentiable physics, or accurate physical modeling.
• Fragmented Workflows: There is a lack of unified platforms that integrate differentiable physics learning, reinforcement learning (RL), structured reward design, and systematic validation.
• Sim-to-Real Gap: Real-world deployment is hindered by unmodeled sensor discrepancies, physical modeling inaccuracies, and control latency, often requiring extensive manual tuning or failing entirely without a zero-shot transfer pipeline.
• Safety Risks: Validating aggressive flight policies on physical hardware without intermediate steps risks severe damage to the vehicle and sensors.

2. Methodology: The E2E-Fly Framework

The authors propose E2E-Fly, a cohesive, full-stack system designed to bridge the gap between simulation and real-world deployment. The framework consists of four main stages:

A. Training Phase

• Simulator (VisFly): Utilizes VisFly, a high-performance differentiable simulator built on Habitat-sim. It supports multi-modal sensing (RGB, depth, segmentation) at high frame rates (up to 6000 FPS) and provides end-to-end differentiability through quadrotor dynamics.
• Algorithms: Supports both Reinforcement Learning (RL) (e.g., PPO) and Differentiable Physics Learning (e.g., Back-Propagation Through Time - BPTT). BPTT leverages analytical gradients from the differentiable dynamics model for higher sample efficiency.
• Reward Design: Introduces a structured, general-purpose reward design framework covering:
  • Dense Rewards: Progress shaping, action smoothness, orientation alignment, linear/angular velocity constraints, and collision avoidance.
  • Sparse Rewards: Goal achievement and crash penalties.
• Curriculum Learning: Implements a progressive training strategy (e.g., starting with obstacle-free navigation before introducing dense obstacles) to prevent local optima in complex tasks.

B. Validation Phase

Before real-world deployment, policies undergo a two-stage validation to ensure robustness:

1. Sim-to-Sim Transfer: Policies trained in VisFly are transferred to AirSim (Unreal Engine) to verify logic consistency across different physics engines and rendering pipelines.
2. Hardware-in-the-Loop (HIL): A physical quadrotor flies within a motion-capture system while receiving virtual visual inputs (RGB/Depth) generated by the simulator. This allows testing perception-aware policies in complex virtual environments without physical collision risks.

C. Deployment Phase

• Hardware Platforms: Two custom quadrotor designs are provided:
  • VIS-R: Onboard computing (Intel N100) for autonomous flight.
  • VIS-H: Offboard computing with wireless links for HIL and high-speed testing.
• Control Bridge: A custom betaflight-ctrl package manages the Finite State Machine (FSM) for initialization, ascent, and policy execution, converting high-level policy outputs into low-level CTBR (Collective Thrust, Body Rates) commands.
• Sim-to-Real Alignment: A four-stage methodology ensures zero-shot transfer:
  1. System Identification: Precise measurement of mass, inertia, and motor thrust curves.
  2. Latency Compensation: Modeling and compensating for motor response and communication delays (approx. 30-90ms).
  3. Domain Randomization: Randomizing initial states and environmental parameters (without randomizing intrinsic physics, which are already identified).
  4. Noise Modeling: Adding Gaussian/Redwood noise to depth maps and IMU data to mimic real sensor imperfections.

3. Key Contributions

1. Unified System: The first system to unify differentiable physical learning, structured reward design, multi-stage validation (Sim-to-Sim + HIL), and real-world deployment into a single pipeline for quadrotors.
2. Differentiable Physics Integration: Demonstrates the integration of analytical gradients (BPTT) with standard RL, showing that BPTT achieves significantly faster convergence and higher sample efficiency than PPO.
3. Structured Reward Guidelines: Provides a comprehensive manual for designing reward functions across diverse tasks (hovering, landing, tracking, racing, obstacle avoidance), reducing the need for heuristic tuning.
4. Zero-Shot Transfer: Successfully deploys policies trained in simulation to physical hardware without task-specific fine-tuning, achieving high-performance agile flight.
5. Open Hardware/Software: Releases two reference hardware platforms and the full software stack (including betaflight-ctrl and alignment tools) to lower the barrier for end-to-end autonomy research.

4. Experimental Results

• Algorithm Comparison: In four state-based tasks (Hovering, Landing, Tracking, Racing), BPTT converged in approximately 30% of the time required by PPO (e.g., ~1,200s vs. ~4,150s) and achieved higher final rewards, demonstrating superior sample efficiency.
• Task Performance: The framework successfully trained policies for six benchmark tasks:
  • State-based: Hovering, Landing, Trajectory Tracking, Racing.
  • Vision-based: Visual Landing (using semantic segmentation), Racing with Obstacles (using depth maps).
• Sim-to-Real Success:
  • Step-response alignment showed near-identical behavior between simulation and real hardware after alignment.
  • Zero-shot deployment was achieved: Policies trained in simulation flew successfully on physical VIS-R and VIS-H platforms in tasks like aggressive racing and obstacle avoidance without any real-world retraining.
  • Vision-based tasks (e.g., landing on random shapes, racing through cluttered environments) achieved 100% success rates in simulation and demonstrated robust transfer to reality.

5. Significance

• Accelerated Research: By providing a complete, reproducible pipeline, E2E-Fly significantly reduces the time and cost associated with developing end-to-end quadrotor controllers.
• Bridging the Gap: It addresses the critical "Sim-to-Real" gap not just through randomization, but through a systematic combination of precise system identification, latency modeling, and noise injection.
• Paradigm Shift: The work validates that differentiable physics learning (BPTT) is a highly efficient alternative or complement to traditional RL for robotics, provided accurate dynamics models are available.
• Community Impact: The release of the hardware designs and software stack enables researchers to focus on algorithm innovation rather than infrastructure development, fostering rapid iteration in agile autonomous flight.