Fly360: Omnidirectional Obstacle Avoidance within Drone View

This paper introduces Fly360, a lightweight two-stage perception-decision pipeline that enables drones with panoramic views to achieve stable, omnidirectional obstacle avoidance by converting RGB observations into depth maps and employing a fixed random-yaw training strategy, outperforming traditional forward-view baselines in both simulation and real-world scenarios.

The Gist

Imagine you are flying a drone, but instead of looking through a narrow window in front of you, you have 360-degree "X-ray vision" that lets you see everything around you—above, below, left, right, and behind—simultaneously.

That is the core idea behind Fly360, a new system designed to make drones safer and smarter at avoiding obstacles.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Tunnel Vision" Drone

Most drones today are like drivers looking only through their windshield. They have cameras pointing forward.

• The Issue: If a tree branch swoops in from the side, or a bird flies up from behind, the drone is blind to it. It might crash because it can't see the danger until it's too late.
• The Old Way: To fix this, engineers used to try to build a detailed 3D map of the world first (like drawing a blueprint of a room before walking in). This is slow, heavy, and often fails in chaotic, moving environments.

2. The Solution: The "Omnidirectional" Drone

Fly360 changes the game by giving the drone a panoramic view. Think of it like a person wearing a helmet with cameras all around their head. They don't just see what's in front; they see the whole world at once.

How Fly360 Works (The Two-Step Dance):

• Step 1: The "Depth Glasses" (Perception)
  The drone takes a flat, 360-degree photo and instantly turns it into a depth map.

  • Analogy: Imagine looking at a black-and-white photo where white means "far away" and black means "close." The drone does this instantly, turning a 2D picture into a 3D understanding of how far away every tree, wall, or person is. It doesn't need to build a perfect map; it just needs to know "how close is that thing?"

• Step 2: The "Reflexive Brain" (Decision)
  Once the drone knows where the obstacles are, a lightweight AI brain decides how to move.

  • Analogy: Instead of a slow, thinking professor calculating a route, Fly360 is like a reflexive athlete. It sees a ball coming from the left and instantly shifts its body to the right without overthinking. It outputs simple commands: "Move forward, left, and up."

3. The Secret Sauce: The "Fixed Random-Yaw" Training

This is the most clever part of the paper. Usually, when you teach a robot to avoid obstacles, you teach it to always face forward while moving forward. But in the real world, a drone might need to fly sideways or backward while keeping its camera pointed at a subject (like a movie camera following a runner).

• The Training Trick: The researchers trained the drone in a simulator with a weird rule: Every time the drone started a training run, they locked its "head" (yaw) in a random direction and kept it there.
  • Analogy: Imagine teaching a basketball player to dribble. Instead of letting them run forward, you tape their head to look at a random spot on the wall and tell them to dribble anyway.
  • The Result: The drone learned that obstacles are obstacles, no matter which way it is facing. It learned to react to the shape of the world around it, not just what is in front of its nose. This makes it incredibly robust.

4. The Results: Why It Matters

The team tested Fly360 in three tough scenarios:

1. Hovering: Staying still while obstacles (like people or other drones) zoom past from all sides.
2. Chasing: Following a moving target while dodging things.
3. Filming: Flying a specific path while keeping the camera pointed at a subject.

The Outcome:

• Old Drones (Forward-View): They crashed constantly because they couldn't see things coming from the side or back.
• Fly360: It danced through the obstacles like a ninja. In tests, it succeeded where others failed, crashing far less often and recovering quickly if it did bump into something.

Summary

Fly360 is like giving a drone a superpower: the ability to see the entire world at once and react instantly, regardless of which way it is facing. It stops relying on slow, complex maps and instead uses a "reflex" system trained to handle chaos from any angle. This means drones can finally fly safely in crowded cities, forests, and even while being chased by humans, without needing a human pilot to steer them.

Technical Summary

Here is a detailed technical summary of the paper "Fly360: Omnidirectional Obstacle Avoidance within Drone View".

1. Problem Statement

Current Unmanned Aerial Vehicle (UAV) obstacle avoidance methods predominantly rely on limited Field-of-View (FoV) sensors (e.g., forward-facing cameras). This creates a critical limitation: when a UAV's movement direction differs from its heading (e.g., strafing sideways while facing a subject for filming), forward-view sensors cannot perceive obstacles approaching from the sides or rear.

• The Gap: Existing systems struggle in scenarios requiring full-spatial awareness where motion and heading are decoupled.
• The Goal: To enable omnidirectional obstacle avoidance for panoramic drones, allowing them to navigate complex, dynamic environments safely regardless of their orientation relative to the movement vector.

2. Methodology: Fly360 Framework

Fly360 is a two-stage perception–decision pipeline designed to map 360° RGB inputs directly to body-frame velocity commands without explicit mapping or specialized setups.

A. System Architecture

1. Perception Stage (Panoramic Depth Estimation):

   • Input: 360° panoramic RGB images (equirectangular projection).
   • Process: A pre-trained panoramic depth model converts RGB inputs into dense depth maps.
   • Representation: The depth is represented as a compact 64×128 equirectangular map. This serves as a robust intermediate representation, reducing the domain gap between simulation and reality compared to raw RGB.
   • Processing: The depth map is processed using SphereConv layers to preserve global geometric continuity and mitigate projection distortions inherent in panoramic images.

2. Decision Stage (Policy Network):

   • Input: The downsampled panoramic depth map + an auxiliary observation vector ($o_t$).
   • Observation Vector Components:
     • $d_{goal}$: Relative direction to the target/goal.
     • $v_t$: Current body-frame velocity.
     • $q^{up}_t$: UAV's upward orientation (attitude).
     • $r$: Predefined safety radius.
   • Network Design: A lightweight Spherical Convolutional Recurrent network.
     • Extracts features via SphereConv and 2D convolutions.
     • Fuses visual features with the observation vector embedding.
     • Uses a GRU (Gated Recurrent Unit) to model temporal dependencies.
     • Output: 3D body-frame velocity commands ($v_x, v_y, v_z$).

B. Training Strategy: Fixed Random-Yaw

A core innovation is the Fixed Random-Yaw Training Strategy designed to achieve orientation-invariant control:

• Problem: In standard training, the UAV's yaw often aligns with its motion, failing to teach the policy how to avoid obstacles when the drone is facing a different direction than it is moving.
• Solution: At the start of each training episode, a random yaw angle is sampled and held fixed throughout the entire episode.
• Effect: This forces the policy to learn a mapping between omnidirectional geometry and collision-free motion that is independent of the UAV's heading. The policy learns to react to obstacles from any direction regardless of where the drone is facing.

C. Loss Function

The policy is optimized in a differentiable simulator using a combined objective:
$$\mathcal{L} = \lambda_{trk}\mathcal{L}_{trk} + \lambda_{safe}\mathcal{L}_{safe} + \lambda_{smooth}\mathcal{L}_{smooth}$$

• Velocity Tracking: Minimizes error between executed velocity and goal-directed target velocity.
• Safety: Penalizes proximity to obstacles and collisions (using softplus and approach velocity terms).
• Smoothness: Regularizes acceleration and jerk to ensure dynamically feasible motion.

3. Key Contributions

1. Problem Formulation: Defined a new, under-explored setting for omnidirectional obstacle avoidance where motion and heading are explicitly decoupled, requiring full-view perception.
2. Fly360 Framework: Proposed a two-stage pipeline integrating panoramic depth estimation with a lightweight, spherical-convolutional policy network.
3. Training Innovation: Introduced the Fixed Random-Yaw strategy, enabling the policy to learn orientation-invariant behaviors without needing exhaustive scenario coverage.
4. Benchmark: Established a benchmark with three representative tasks:
   • Hovering Maintenance: Maintaining position/orientation while avoiding approaching obstacles.
   • Dynamic Target Following: Tracking a moving object while avoiding dynamic obstacles.
   • Fixed-Trajectory Filming: Following a path while keeping the camera fixed on a target.

4. Experimental Results

The authors evaluated Fly360 in high-fidelity simulations (Park, Forest, Urban Street, Factory) and real-world flights.

Quantitative Performance

• Success Rate (SR) & Collision Time (CT): Fly360 significantly outperformed forward-view and multi-view baselines.
  • Example (Hovering in Park): Fly360 achieved 7/10 success with 0.54s collision time. Forward-view baselines (e.g., Zhang et al., 2025) achieved 0/10 success with collision times exceeding 3–15s.
  • Dynamic Target Following: Fly360 achieved 10/10 success with 0s collision time in forest scenes, whereas multi-view baselines often failed completely.
• Robustness: The system remained stable even with added Gaussian noise to depth maps (up to 20% relative noise) and across varying obstacle speeds and densities.
• Efficiency: Despite processing 360° data, Fly360 runs at ~44.6 FPS on an RTX 3090, comparable to forward-view baselines, due to the lightweight network and low-resolution depth input.

Real-World Validation

• Deployed on a custom quadrotor with dual fisheye cameras.
• Successfully demonstrated omnidirectional avoidance in confined hovering and chasing scenarios where a human pursued the drone.
• Showed effective sim-to-real transfer, maintaining stability against dynamic threats and partial occlusions.

5. Significance and Impact

• Beyond Forward-View: Fly360 addresses the fundamental blind-spot limitation of current UAV navigation, enabling safe operation in scenarios where the drone must move laterally or backward while facing a subject (critical for cinematography and inspection).
• Scalability: By using a unified panoramic representation rather than stitching multiple camera views, the method avoids the discontinuities and calibration issues common in multi-view systems.
• Practicality: The framework does not require explicit 3D mapping or complex localization, making it suitable for agile, real-time flight in unstructured environments.
• Future Direction: This work paves the way for "Spatial Intelligence" in drones, allowing them to perceive and navigate 3D space holistically rather than just along a forward vector.