💻 computer science

CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

CosFly-VLA is a spatially aware Vision-Language-Action model that enhances UAV target tracking in complex, occluded urban environments by integrating depth-aware pretraining, curriculum-based fine-tuning, chain-of-thought reasoning, and closed-loop reinforcement learning to significantly reduce displacement errors and improve success rates compared to existing policies.

Original authors: Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen, Xiangyue Wang, Tianle Zeng, Shuai Yuan, Binbo Li, Hanzhong Guo, Ji Pei, Da Zhang, Kangli Wang

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen, Xiangyue Wang, Tianle Zeng, Shuai Yuan, Binbo Li, Hanzhong Guo, Ji Pei, Da Zhang, Kangli Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the pilot of a tiny, super-smart drone, zooming through a busy city. Your job is to follow a specific person walking down the street. Usually, this is easy: you see them, you fly toward them, and you keep them in your camera lens. But cities are tricky. Suddenly, a tall building, a dense tree, or a crowd of people blocks your view. The person disappears from your screen. A normal drone might panic, guess wrong, and crash into a wall because it lost its "mental map" of where the person went. This is the world of Unmanned Aerial Vehicle (UAV) tracking, a field of robotics where machines learn to chase moving targets. The big challenge isn't just seeing the target; it's knowing what to do when you can't see them anymore. To solve this, scientists use Vision-Language-Action (VLA) models. Think of these as a robot's brain that can "see" an image, "read" a command like "follow the runner," and "act" by moving its motors, all at the same time.

Enter CosFly-VLA, a new, super-smart drone pilot designed by researchers to handle these "blind spots." Instead of just staring at the screen and waiting for the target to pop back into view, CosFly-VLA acts like a detective with a strong imagination. When the target vanishes behind a building, the drone doesn't freeze. It uses its knowledge of the city's layout and the person's last known speed to guess where they probably are. It thinks, "Okay, they were walking right, and that building is on the left, so they must be coming out on the other side." It then flies a calculated path to meet them there. The researchers trained this drone using a special "recipe" that included lots of practice with long periods of blindness, teaching it to trust its spatial reasoning over its eyes when the eyes fail. They tested it in a high-tech video game world called CARLA, where the drone had to chase pedestrians through complex city maps. The results suggest that this new approach is much better at keeping the target in sight and avoiding crashes compared to older, more basic drone brains, especially when the target is hidden for a long time.

The Detective Drone: How CosFly-VLA Works

The core idea behind this paper is that chasing a moving target in a city is like playing a game of "hide-and-seek" where the seeker has to keep moving even when they can't see the hider. The researchers, led by a team from Autel Robotics and several universities, realized that most existing drone systems are great at "seeing and following" but terrible at "guessing and recovering" when the view is blocked.

The Problem: The "Blind Spot" Panic
Imagine you are playing a video game where you have to follow a character. Suddenly, a wall covers the screen. If your character just stops or flies randomly, you lose. In the real world, if a drone loses sight of a person, it might fly in the wrong direction, crash into a tree, or just give up. The paper argues that the old way of training drones—showing them thousands of pictures of people walking—doesn't teach them how to handle the moments when the person is not visible.

The Solution: A Brain That Thinks in 3D
CosFly-VLA is a "Vision-Language-Action" model. Let's break that down with a simple analogy:

Vision: It has eyes (cameras) that see the world.
Language: It can read instructions like "Follow the person in the red shirt."
Action: It controls the drone's motors to move up, down, left, right, and turn.

But what makes CosFly-VLA special is its Spatial Awareness. When the target is hidden, the drone doesn't just guess; it builds a "mental hypothesis." It asks itself: "Where was the person last? Where are the buildings? If they kept walking straight, where would they be now?" It then flies to that spot, ready to grab the target the moment they reappear.

The Training Recipe: From Student to Master
The researchers didn't just teach the drone to fly; they gave it a very specific, four-step training course to make it a master tracker:

The "City Map" Boot Camp (Spatially Grounded Pretraining): Before learning to chase, the drone studied thousands of aerial photos and 3D puzzles. It learned about distances, heights, and how buildings block views. This is like teaching a student to read a map before sending them on a scavenger hunt.
The "Easy to Hard" School (Curriculum Learning): The drone started with easy tracks where the person was always visible. Then, the teachers (the researchers) slowly made it harder, introducing long periods where the person was hidden behind buildings. This forced the drone to practice its "guessing" skills.
The "Think Aloud" Class (Chain-of-Thought): This is the coolest part. The drone was taught to "think out loud" before it moved. When the target disappeared, it would generate a text explanation like: "The target is behind the building. They were walking right, so I should fly right and wait." This reasoning step helped the drone understand why it was making a move, not just what to do.
The "Live Fire" Drill (Reinforcement Learning): Finally, the drone flew in a simulated city (the CARLA game engine) and learned by doing. If it crashed, it got a "bad score." If it successfully found the person after a long hide, it got a "good score." Over time, it learned to fly smoother and safer.

What the Numbers Say
The researchers tested their new drone against older, standard models. They used two types of tests:

Open-Loop: The drone looked at a video clip and predicted where the person would be, without actually flying.
Closed-Loop: The drone actually flew in the simulator, making real-time decisions.

The results were promising. In the "Open-Loop" tests, CosFly-VLA made fewer mistakes in predicting the target's path compared to the standard models. Specifically, it reduced the average error in predicting the target's position by about 34% on familiar maps and 35% on new, unseen maps.

In the real "Closed-Loop" flying tests, the improvement was even more dramatic for success rates. On familiar maps, the new drone succeeded in tracking the target 29.8% more often than the standard model (jumping from 57% success to 74%). On completely new maps, it also improved, though by a smaller margin (2.5%). Most importantly, the new drone crashed less often and kept a safer distance from the target.

The Catch: It's Still a Simulation
While the results are exciting, the paper is very clear about its limits. All of these tests happened inside a computer simulation (the CARLA game). The drone has never flown in the real world with real wind, real rain, or real unpredictable people. The researchers admit that real-world physics might be messier than the game. They also note that the drone was trained on a specific set of city maps and pedestrian behaviors, so it might struggle with very different environments (like a dense forest or a crowded indoor mall) that it hasn't seen before.

The Big Takeaway
CosFly-VLA suggests that for drones to be truly useful in complex cities, they need to stop just "reacting" to what they see and start "reasoning" about what they don't see. By combining a strong understanding of 3D space with a step-by-step reasoning process, these drones can stay on track even when the target disappears. It's a step toward making autonomous drones that are as clever as a human pilot who knows the neighborhood, rather than just a camera that follows a dot.

Technical Summary: CosFly-VLA

Problem Formulation

Dynamic target tracking for Unmanned Aerial Vehicles (UAVs) in complex urban environments faces a critical failure mode: sustained occlusion. While existing Vision-Language-Action (VLA) policies excel at tracking visible targets, their performance degrades significantly when targets are blocked by buildings, vegetation, or crowds. In these scenarios, conventional detection-reidentification-control pipelines lose the stable identity evidence required for recovery. The paper reframes this challenge not merely as "seeing and following," but as a closed-loop recovery problem. The agent must maintain a spatial hypothesis about an unseen target, predict its likely re-emergence region using spatial priors, and execute corrective actions to suppress drift and re-acquire the target without direct visual observation.

Methodology: CosFly-VLA

The authors propose CosFly-VLA, a spatially aware VLA model designed to jointly ground targets, estimate visibility, and generate continuous flight actions.

1. Model Architecture

Backbone: The model utilizes a frozen Qwen3.5 vision-language backbone enhanced with Low-Rank Adaptation (LoRA) adapters.
Meta-Query Interface: To avoid brittle JSON decoding, the architecture introduces boundary-tagged meta-query spans (<boi_act>, <boi_box>, <boi_vis>) that route task-specific hidden states to decoupled heads.
Three Structured Outputs:
1. Action Head: A 12-layer Diffusion Transformer (DiT) using Flow Matching to predict an 8-step, 4-DoF (dx, dy, dz, dyaw) waypoint-delta action chunk. This models a continuous trajectory distribution rather than discrete tokens.
2. Bounding Box Head: An MLP that predicts the current target location (0–1000 grid) for visual grounding.
3. Visibility Head: An MLP estimating the probability of target visibility, crucial for determining when to rely on spatial reasoning over appearance.
Chain-of-Thought (CoT): The model is trained to emit a reasoning trace (<thought>) before the structured answer, explicitly reasoning about occlusion and recovery strategies.

2. Training Recipe

The training follows a four-stage progressive curriculum designed to inject aerial spatial priors and closed-loop robustness:

Spatially Grounded Continued Pretraining (CPT): A 500k-sample mixture (70% internal UAV trajectories, 30% external aerial spatial-VL datasets like AirSpatial and Open3DVQA-v2) adapts the backbone to UAV-view depth, distance, and 3-D spatial reasoning.
Curriculum-based Supervised Fine-Tuning (SFT): A three-stage process:
- Multi-head warm-up: Initializes heads with a frozen backbone.
- Natural curriculum: Trains on standard tracking distributions.
- Hard/Long-occlusion curriculum: Explicitly includes windows with 4–5 invisible frames (previously filtered out) to teach recovery behavior.
Chain-of-Thought Training: Uses teacher-generated reasoning traces for hard/long-occlusion cases to teach recovery-oriented logic before structured answers.
Closed-Loop Reinforcement Learning (RL): The action head is fine-tuned using Expert-Guided Flow-Policy Optimization (EG-FPO) in a CARLA simulator. The reward function combines stand-off distance maintenance, grounding quality (IoU), collision avoidance, and task success. The backbone remains frozen; only the action expert is updated.

Key Contributions

Reformulation of Tracking: The paper shifts the focus from visible-frame imitation to occlusion-robust closed-loop recovery, defining reproducible Easy/Medium/Hard protocols based on geometric visibility.
Spatially Aware VLA Architecture: Introduces a meta-query interface that decouples action, grounding, and visibility prediction, enabling the model to maintain state estimates under partial observability.
Progressive Training Strategy: Demonstrates the efficacy of combining Spatially Grounded CPT, curriculum learning with long-occlusion data, CoT reasoning, and on-policy RL to bridge the gap between offline supervision and closed-loop deployment.
Evaluation Framework: Establishes a rigorous evaluation using occlusion-graded open-loop windows and closed-loop CARLA rollouts on both seen and unseen maps.

Experimental Results

The evaluation compares CosFly-VLA-0.8B against general VLMs (e.g., Qwen3.5, Opus) and general VLA baselines (OpenVLA, $\pi_0$ , $\pi_{0.5}$ ).

Open-Loop Performance:
- Relative to OpenVLA, CosFly-VLA-0.8B (SFT+CoT) reduces Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test.
- Improvements are most pronounced in "Hard" occlusion slices, with ADE reductions of 46.7% (seen) and 39.2% (unseen).
- The model achieves superior bounding box grounding (IoU ~0.65) and visibility estimation (F1 ~0.94).
Closed-Loop Performance:
- The RL-tuned variant improves Success Rate (SR) by 29.8% (+17 percentage points) on seen-test and 2.5% (+2 points) on unseen-test compared to OpenVLA.
- It reduces rollout ADE by 10.0% (seen) and 8.7% (unseen) and significantly lowers stand-off distance error.
- Qualitative analysis shows the RL policy successfully recovers from drift and avoids collisions in dense urban corridors where baselines fail.
Ablation Studies:
- Spatially Grounded CPT is identified as the primary driver for cross-map generalization (unseen-test gains).
- CoT supervision provides the largest marginal gain on "Hard" seen-test cases, validating the importance of explicit reasoning for occlusion recovery.

Significance and Claims

The paper claims that CosFly-VLA demonstrates a significant step forward in moving UAV tracking from visible-frame imitation toward spatially grounded, action-closed-loop control. The authors emphasize that robust tracking under occlusion requires:

Explicit spatial reasoning about unseen targets rather than relying solely on appearance.
Data construction that preserves long-occlusion samples rather than filtering them as noise.
Closed-loop optimization to adapt to the observation distribution induced by the policy's own actions, preventing error accumulation.

The work establishes that a 0.8B parameter model, when trained with a spatially grounded curriculum and closed-loop RL, can outperform larger general-purpose VLA baselines in complex, occluded aerial tracking scenarios. The authors note that while the results are promising in simulation, future work is required to address real-world deployment challenges such as sensor noise, communication latency, and diverse target behaviors.