💻 computer science

SkySim: A ROS2-based Simulation Environment for Natural Language Control of Drone Swarms using Large Language Models

This paper introduces SkySim, a ROS2-based simulation framework that enables natural language control of UAV swarms by integrating Large Language Models for high-level planning with an Artificial Potential Field safety filter to ensure real-time, collision-free execution for non-expert users.

Original authors: Aditya Shibu, Marah Saleh, Mohamed Al-Musleh, Nidhal Abdulaziz

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aditya Shibu, Marah Saleh, Mohamed Al-Musleh, Nidhal Abdulaziz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a flock of tiny, smart drones. Usually, telling them what to do is like trying to conduct an orchestra where every musician needs a complex sheet of music written by a professional composer. If you want them to form a circle or a triangle, you have to write precise code for every single drone.

This paper introduces SkySim, a new way to talk to these drone swarms using plain English, like you're talking to a friend.

Here is how it works, broken down into simple parts:

1. The "Brain" and the "Reflex"

The system uses two main parts working together, kind of like a human brain and your body's reflexes.

The Brain (The LLM): This is a powerful Artificial Intelligence (specifically Google's Gemini 3.5 Pro). You type a command like, "Form a Christmas tree," or "Make a circle." The Brain understands the idea and figures out where every drone needs to go to make that shape. It's great at understanding language and geometry.
The Reflex (The Safety Filter): The problem is that the Brain is smart but "airheaded." It doesn't really understand physics. It might tell two drones to fly into the exact same spot at the same time, which would cause a crash.
- This is where the Reflex comes in. It's a fast, automatic safety system (running 20 times a second) that watches the drones. If the Brain tells two drones to get too close, the Reflex gently pushes them apart, just like your hand instinctively pulling away from a hot stove before you even think about it.

2. How They Talk to Each Other

The system is built on a framework called ROS2 (think of it as the nervous system connecting everything).

Step 1: You give a voice command.
Step 2: The Brain looks at where the drones are right now and calculates the target spots for the new shape.
Step 3: The Brain sends these target spots to the Reflex.
Step 4: The Reflex checks: "Is this safe? Are they too close? Are they hitting the ground?" If everything is good, it sends the movement command. If not, it makes tiny, instant adjustments to keep them safe.

3. The Test Drive

The researchers tested this in a computer simulation (a virtual world) with different numbers of drones: 3, 10, and even 30 at once.

The Shapes: They asked the system to make triangles, circles, grids, cubes, and even a 3D Christmas tree.
The Result: The Brain got the shapes right 100% of the time. It successfully figured out the math to arrange the drones into these shapes.
The Safety: They tried to trick the system by telling all the drones to crash into the center of the room. The Reflex kicked in immediately, pushing the drones apart so they never actually hit each other. Even when the drones were packed tightly together, the system kept them safe.

4. Speed and Limits

Speed: The safety system is super fast (reacting in milliseconds). However, the "Brain" takes a little time to think (about 34 to 50 seconds, depending on how many drones there are). This is because the Brain is in the "cloud" (on a remote server), and internet traffic can sometimes cause delays.
The Catch: Because the Brain is in the cloud, if the internet is slow, the drones might wait a bit before moving to a new shape. Also, the system currently works best in a clean, virtual world. In the real world, wind or messy sensors might make it harder for the Reflex to be perfect.

The Bottom Line

SkySim proves that you don't need to be a robotics expert to control a swarm of drones. You can just ask them to do something in English, and a "safety net" underneath will make sure they do it without crashing. It bridges the gap between human ideas and robot safety, allowing non-experts to create complex drone shows or tasks easily.

The researchers plan to take this from the computer simulation to real physical drones next, and they hope to make the "Brain" faster by putting it directly on the drones instead of relying on the internet.

Technical Summary: SkySim – A ROS2-based Simulation Environment for Natural Language Control of Drone Swarms using Large Language Models

Problem Statement
Unmanned Aerial Vehicle (UAV) swarms offer significant potential in logistics, agriculture, and surveillance; however, their deployment is hindered by the requirement for expert-level knowledge to ensure safety and physical feasibility. Traditional control methods rely on static, pre-programmed flows that lack adaptability in dynamic environments. While Large Language Models (LLMs) present a promising avenue for enabling natural language control, they inherently lack real-world physical grounding. Consequently, LLMs often generate reference trajectories that are unsafe, failing to account for collision risks, kinematic limits, or environmental constraints. There is a critical need for a framework that leverages the cognitive reasoning of LLMs while enforcing strict, real-time safety constraints to bridge the gap between high-level human intent and low-level robotic execution.

Methodology
The authors introduce SkySim, a simulation framework built on ROS2 (Jazzy) and Gazebo (Harmonic) that decouples high-level cognitive planning from low-level safety enforcement. The system architecture operates on a "sense-plan-act" loop with the following components:

State Sensing: The system subscribes to the /odom topic in Gazebo, emulating high-precision motion capture systems (e.g., OptiTrack) to obtain noiseless, real-time 3D positional telemetry of all agents.
LLM-Based Planning (High-Level): Using Gemini 3.5 Pro, the system translates natural language commands (e.g., "Form a circle") into spatial waypoints. The LLM is constrained by strict system instructions regarding swarm size, coordinate systems, and ground-collision limits ( $Z \ge 0.5$ m). The model receives a dynamic prompt containing system instructions, current drone states, and the user command, outputting a serialized Python list of target coordinates.
Safety Filter (Low-Level): To ensure physical feasibility, a deterministic Artificial Potential Field (APF) controller operates at 20 Hz, independent of the LLM's inference latency. This layer applies:
- Attractive Forces: Proportional control driving agents toward LLM-generated waypoints.
- Repulsive Forces: Collision avoidance triggered when inter-agent distance falls below a safety radius ( $r_{min} = 0.8$ m), generating repulsive vectors to prevent collisions.
- Kinematic Constraints: Velocity saturation ( $v_{max} = 0.5$ m/s) and geo-fencing to ensure commands remain within the operational volume and motor limits.

Key Contributions

Decoupled Architecture: The paper proposes a modular framework that isolates the probabilistic nature of LLM planning from the deterministic requirements of real-time safety, allowing non-experts to define complex behaviors without managing low-level constraints.
Zero-Shot Spatial Reasoning: Unlike prior work focused on temporally coordinated choreography, SkySim demonstrates the ability to translate abstract natural language commands into valid, arbitrary static geometries (e.g., "Christmas Tree," "Grid," "Sphere") without prior training on specific shapes.
Lightweight Safety Mechanism: The implementation of a reactive APF controller is highlighted as a computationally efficient alternative to heavy learning-based safety layers (e.g., DRL) or non-linear solvers (e.g., MPC), ensuring immediate reflexive collision avoidance with negligible overhead.

Experimental Results
The framework was evaluated using simulated Crazyflie 2.1 Brushless drones in swarm sizes of N = 3, 10, and 30.

Spatial Reasoning Accuracy: The LLM achieved a 100% success rate in generating valid target coordinates for various geometric primitives across all tested swarm sizes.
Collision Avoidance: In stress tests involving static hazards (converging to a single point) and dynamic hazards (simultaneous position swapping), the APF filter successfully prevented collisions. While inter-agent distances briefly dipped below the 0.8 m APF activation threshold, they remained strictly above the physical collision limit (~55 mm).
Scalability and Latency: The system demonstrated sub-linear scalability; increasing the swarm size from 3 to 30 agents resulted in only a ~45% increase in mean planning latency (from 34s to 50s). Crucially, the variable planning latency (seconds) was shown to be fully decoupled from the control loop latency (milliseconds), ensuring swarm stability even during LLM inference delays.
Trajectory Feasibility: Velocity profiles confirmed that the generated trajectories were smooth and remained within kinematic limits ( $v_{max} = 0.5$ m/s), with no high-frequency oscillations.

Significance and Limitations
The paper claims that SkySim successfully bridges the gap between AI cognition and robotic safety, empowering non-experts to iteratively refine swarm behaviors for dynamic environments. The decoupled approach is presented as a viable pathway for deploying UAV swarms in sectors like agriculture and disaster response where complex, on-demand behaviors are required without extensive programming.

However, the authors explicitly acknowledge current limitations:

Latency Variance: Reliance on cloud-based APIs (Gemini) introduces significant latency variance, making the system unsuitable for time-critical missions requiring millisecond-level reactions to new commands.
Simulation Constraints: The current validation relies on noiseless ground-truth data from Gazebo; real-world deployment will face challenges from sensor noise, packet loss, and environmental disturbances.
Local Minima: The deterministic APF controller is susceptible to local minima in highly cluttered environments, potentially trapping agents.
Context Window Limits: The system is currently capped at N=30 due to LLM context window constraints and hardware resources for real-time physics simulation.

Future Work
The authors outline plans to transition the stack to physical hardware (Crazyflie with OptiTrack), integrate Model Predictive Control (MPC) to address local minima, incorporate multi-modal inputs, and shift to locally deployed Small Language Models (SLMs) to eliminate cloud-based latency bottlenecks.

1. The "Brain" and the "Reflex"

2. How They Talk to Each Other

3. The Test Drive

4. Speed and Limits

The Bottom Line

More like this