Technical Summary: RecoverFly

Problem Statement

Unmanned Aerial Vehicle Vision-Language Navigation (UAV-VLN) requires agents to translate visual observations and natural language instructions into reliable flight actions within complex, large-scale 3D environments. While recent end-to-end Vision-Language-Action (VLA) policies (e.g., AerialVLA) have reduced reliance on modular perception and planning by directly mapping observations to controls, they face significant limitations in closed-loop execution:

1. Lack of Corrective Supervision: Behavior cloning (BC) trains on expert demonstrations but fails to learn from the agent's own errors during inference. Local errors (e.g., drift, occlusion) compound over long trajectories, leading to collisions, premature landings, or target misses.
2. Inefficient Sample Utilization in RL: Direct application of Reinforcement Learning (RL) to autoregressive VLA policies is hindered by sparse feedback, where failure events may result from a sequence of actions rather than a single step. Furthermore, standard online sampling often discards informative failure cases quickly, and scene imbalances cause optimization to focus on frequent environments while neglecting rare (long-tail) scenarios.
3. Policy Distribution Shift: Unconstrained RL updates can degrade previously acquired navigation capabilities, particularly when adapting to new maps or objects.

Existing methods often rely on hierarchical planners or external modules to mitigate these issues, but they do not comprehensively address the challenges of sparse closed-loop feedback, long-tail scene adaptation, and policy drift within a unified end-to-end framework.

Methodology: RecoverFly

RecoverFly is a failure-aware RL post-training framework designed to optimize end-to-end UAV-VLA policies. It integrates four core components to enable stable, corrective learning:

1. Token-Level RL for Grammar-Constrained Actions

Unlike standard continuous-action PPO, RecoverFly adapts RL to the token-level output of autoregressive VLA models.

• Action Representation: The policy generates a sequence of action tokens constrained by a specific grammar (three numerical control tokens for forward progression, vertical movement, and yaw, followed by an optional LAND token).
• Optimization: The framework applies the clipped surrogate objective separately to each valid action token within a sequence, assigning a shared advantage estimate to all tokens encoding the same action. This enables fine-grained updates in the native autoregressive space while preserving the learning signal for the entire action step.

2. Dynamic Failure Replay

To address the rapid dilution of difficult failure cases in online sampling, RecoverFly introduces a dynamic failure replay mechanism.

• Mechanism: Instead of storing old trajectories, the system maintains a pool of unresolved task initializations (identified by scene and failure type).
• Process: During training, the agent samples a mix of fresh tasks and unresolved tasks from the pool. After each rollout, failed fresh instances are added to the pool, successful replays are marked as solved, and persistently unsolvable cases are dropped.
• Benefit: This ensures that the policy repeatedly attempts to resolve high-value failure cases under the current policy, converting sparse failure feedback into reusable corrective experience without violating on-policy constraints.

3. Two-Stage Long-Tail Scene Curriculum

To mitigate scene-level sampling imbalance, RecoverFly employs a curriculum learning strategy:

• Stage I: Samples scenes according to their empirical frequencies in the training dataset to establish baseline navigation capabilities.
• Stage II: Transitions to an equal quota sampling strategy, ensuring rare scenes are visited as frequently as common ones.
• Goal: This progressively shifts the training distribution toward long-tailed scenes, improving generalization to unseen maps and difficult routes without overfitting to frequent scenarios.

4. Reference-Policy KL Regularization

To prevent catastrophic forgetting and excessive policy drift, RecoverFly incorporates a stage-wise Kullback-Leibler (KL) regularization term.

• Anchoring: Stage I anchors updates to the initial pre-trained VLA policy. Stage II anchors updates to the policy at the end of Stage I.
• Effect: This constrains the policy from deviating too far from acquired capabilities, balancing the trade-off between adapting to new scenarios and retaining robust navigation skills.

Key Contributions

1. Integrated RL Framework for End-to-End UAV-VLA: The authors propose RecoverFly, a unified post-training framework that adapts token-level RL to grammar-constrained UAV commands, enabling stable closed-loop optimization of autoregressive actions.
2. Failure-Aware Replay Learning: A novel mechanism that retains and revisits unresolved navigation cases, improving the utilization of sparse failure feedback and strengthening corrective learning without applying updates to stale trajectories.
3. Long-Tail Adaptation with Stability: The introduction of a two-stage scene curriculum combined with reference-policy KL regularization, which strengthens learning from underrepresented scenes while limiting policy distribution shift and preserving prior capabilities.

Experimental Results

The framework was evaluated on the TravelUAV benchmark, comparing RecoverFly against heuristic controls, task-specific UAV-VLN methods, generalist navigation models, and the AerialVLA baseline.

• Performance Gains: RecoverFly achieved the best performance across all three evaluation splits (Seen, Unseen-Map, and Unseen-Object).
  • Compared to the AerialVLA initialization, RecoverFly improved the Success Rate (SR) by 3.12 to 8.37 percentage points across different splits.
  • On the "Hard" subset of Seen environments, the SR improvement reached 10.08 percentage points.
  • On Unseen-Map "Hard" trajectories, the Navigation Error (NE) decreased by 19.23 meters.
• Efficiency: These improvements were achieved with a total rollout budget of approximately 30% of the training-set size, demonstrating high sample efficiency.
• Ablation Studies:
  • Failure Replay: Isolating this component showed it significantly boosts performance on Seen and Unseen-Object splits but can negatively impact Unseen-Map performance without the curriculum, highlighting the need for balanced scene exposure.
  • KL Regularization: This component was crucial for recovering performance on Unseen-Map splits, preventing excessive drift from the initial policy.
  • Curriculum: The two-stage curriculum outperformed both uniform sampling and original distribution sampling, confirming the complementary roles of establishing baselines and then balancing rare scene exposure.

Significance and Claims

The paper claims that RecoverFly establishes a practical post-training method that jointly addresses sparse closed-loop feedback, long-tail scene adaptation, and policy drift. By integrating fine-grained action optimization, failure-oriented learning, and stable policy updating, RecoverFly demonstrates that RL post-training can effectively transform end-to-end VLA policies from behavior-cloned agents into robust, self-correcting navigators. The work suggests that revisiting unresolved tasks and shifting training distributions toward rare scenes are critical for improving generalization in aerial navigation, providing a viable path for advancing UAV-VLN capabilities without relying on external perception modules or hierarchical planners.